2024-04-19T00:30:42.361 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-19T00:30:42.361 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} --archive /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199 --name yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/orig.config.yaml 2024-04-19T00:30:42.400 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T00:30:42.486 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199 branch: wip-yuri5-testing-2024-04-17-1400 description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: yweinste@redhat.com first_in_suite: false job_id: '7663199' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri5-testing-2024-04-17-1400 ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri5-testing-2024-04-17-1400 sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mon.d - mon.e - osd.0 - osd.1 - osd.2 - - mon.f - mon.g - mon.h - mon.i - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1937 sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri5-testing-2024-04-17-1400 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 targets: smithi043.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI95gXZ4zVyone2S8SHXEndb9bMXVCh5+kGX01O7ftejcUi77MTgcyz8sgacIRkn980TDKv2qMlWU1Ir/Pd/70Q= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxAMGsllWyXdlTRNdwf2anP9RKvXywVq3dsQ2q18q6otGTFjE4YwKJ+uTIh+eiuKsRZzSga5mia9b0v3TEltdg= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-18_22:49:05 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-19T00:30:42.486 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa; will attempt to use it 2024-04-19T00:30:42.487 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa/tasks 2024-04-19T00:30:42.487 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-19T00:30:42.488 INFO:teuthology.task.internal:Checking packages... 2024-04-19T00:30:42.513 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8b4ab94f9c30da63487fa2f241ce12f483378860' 2024-04-19T00:30:42.513 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-19T00:30:42.513 INFO:teuthology.packaging:ref: None 2024-04-19T00:30:42.513 INFO:teuthology.packaging:tag: None 2024-04-19T00:30:42.513 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-04-17-1400 2024-04-19T00:30:42.513 INFO:teuthology.packaging:sha1: 8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:30:42.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri5-testing-2024-04-17-1400 2024-04-19T00:30:42.731 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:30:42.733 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-19T00:30:42.739 INFO:teuthology.task.internal:no buildpackages task found 2024-04-19T00:30:42.739 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-19T00:30:42.767 INFO:teuthology.task.internal:Saving configuration 2024-04-19T00:30:42.783 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-19T00:30:42.818 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-19T00:30:42.842 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi043.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 00:22:05.072557', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH8qfZDgvkCXc8GivxMOxeWBMxdLKK384+4HS3BDFTs6'} 2024-04-19T00:30:42.865 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-19 00:22:05.073841', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKrbPbFggOnSQcsQD2ZIky/oRT5JDOP43BG5OLAEUTYD'} 2024-04-19T00:30:42.865 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-19T00:30:42.871 INFO:teuthology.task.internal:roles: ubuntu@smithi043.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'] 2024-04-19T00:30:42.871 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-19T00:30:42.871 INFO:teuthology.run_tasks:Running task console_log... 2024-04-19T00:30:42.955 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdb73b183a0>, signals=[15]) 2024-04-19T00:30:42.955 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-19T00:30:42.962 INFO:teuthology.task.internal:Opening connections... 2024-04-19T00:30:42.962 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T00:30:42.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:30:43.048 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T00:30:43.049 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:30:43.131 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-19T00:30:43.138 DEBUG:teuthology.orchestra.run.smithi043:> uname -m 2024-04-19T00:30:43.143 INFO:teuthology.orchestra.run.smithi043.stdout:x86_64 2024-04-19T00:30:43.143 DEBUG:teuthology.orchestra.run.smithi043:> cat /etc/os-release 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:NAME="Ubuntu" 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_ID="22.04" 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:VERSION_CODENAME=jammy 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:ID=ubuntu 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:ID_LIKE=debian 2024-04-19T00:30:43.191 INFO:teuthology.orchestra.run.smithi043.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T00:30:43.192 INFO:teuthology.orchestra.run.smithi043.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T00:30:43.192 INFO:teuthology.orchestra.run.smithi043.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T00:30:43.192 INFO:teuthology.orchestra.run.smithi043.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T00:30:43.192 INFO:teuthology.orchestra.run.smithi043.stdout:UBUNTU_CODENAME=jammy 2024-04-19T00:30:43.192 INFO:teuthology.lock.ops:Updating smithi043.front.sepia.ceph.com on lock server 2024-04-19T00:30:43.211 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-04-19T00:30:43.216 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-04-19T00:30:43.216 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-04-19T00:30:43.260 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-19T00:30:43.260 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="Ubuntu" 2024-04-19T00:30:43.260 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="22.04" 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_CODENAME=jammy 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:ID=ubuntu 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE=debian 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-19T00:30:43.261 INFO:teuthology.orchestra.run.smithi161.stdout:UBUNTU_CODENAME=jammy 2024-04-19T00:30:43.261 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-04-19T00:30:43.294 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-19T00:30:43.301 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-19T00:30:43.325 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-19T00:30:43.326 DEBUG:teuthology.orchestra.run.smithi043:> test '!' -e /home/ubuntu/cephtest 2024-04-19T00:30:43.328 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-04-19T00:30:43.335 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-19T00:30:43.341 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-19T00:30:43.341 DEBUG:teuthology.orchestra.run.smithi043:> test -z $(ls -A /var/lib/ceph) 2024-04-19T00:30:43.380 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-04-19T00:30:43.464 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-19T00:30:43.516 INFO:teuthology.run_tasks:Running task kernel... 2024-04-19T00:30:43.530 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-19T00:30:43.530 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.f': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-19T00:30:43.530 DEBUG:teuthology.orchestra.run.smithi043:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T00:30:43.531 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-19T00:30:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:30:43.537 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-04-19T00:30:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:30:43.539 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-04-19T00:30:43.583 INFO:teuthology.orchestra.run.smithi043.stdout:5.15.0-56-generic 2024-04-19T00:30:43.583 INFO:teuthology.task.kernel:Running kernel on smithi043: 5.15.0-56-generic 2024-04-19T00:30:43.584 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get clean 2024-04-19T00:30:43.585 INFO:teuthology.orchestra.run.smithi161.stdout:5.15.0-56-generic 2024-04-19T00:30:43.585 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.15.0-56-generic 2024-04-19T00:30:43.585 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get clean 2024-04-19T00:30:43.719 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get update 2024-04-19T00:30:43.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get update 2024-04-19T00:30:43.866 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:30:43.881 INFO:teuthology.orchestra.run.smithi161.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T00:30:43.905 INFO:teuthology.orchestra.run.smithi043.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:30:43.960 INFO:teuthology.orchestra.run.smithi161.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T00:30:43.969 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:30:43.998 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:30:44.080 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-19T00:30:44.105 INFO:teuthology.orchestra.run.smithi161.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T00:30:44.143 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T00:30:44.219 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T00:30:44.225 INFO:teuthology.orchestra.run.smithi161.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T00:30:44.228 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T00:30:44.229 INFO:teuthology.orchestra.run.smithi161.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T00:30:44.229 INFO:teuthology.orchestra.run.smithi161.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T00:30:44.233 INFO:teuthology.orchestra.run.smithi043.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T00:30:44.254 INFO:teuthology.orchestra.run.smithi161.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T00:30:44.258 INFO:teuthology.orchestra.run.smithi161.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:30:44.258 INFO:teuthology.orchestra.run.smithi161.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T00:30:44.270 INFO:teuthology.orchestra.run.smithi161.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T00:30:44.278 INFO:teuthology.orchestra.run.smithi161.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T00:30:44.287 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T00:30:44.293 INFO:teuthology.orchestra.run.smithi043.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T00:30:44.293 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T00:30:44.294 INFO:teuthology.orchestra.run.smithi161.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T00:30:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T00:30:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T00:30:44.295 INFO:teuthology.orchestra.run.smithi161.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T00:30:44.303 INFO:teuthology.orchestra.run.smithi161.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T00:30:44.303 INFO:teuthology.orchestra.run.smithi161.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T00:30:44.304 INFO:teuthology.orchestra.run.smithi161.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T00:30:44.305 INFO:teuthology.orchestra.run.smithi161.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T00:30:44.305 INFO:teuthology.orchestra.run.smithi161.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T00:30:44.322 INFO:teuthology.orchestra.run.smithi043.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T00:30:44.323 INFO:teuthology.orchestra.run.smithi043.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T00:30:44.328 INFO:teuthology.orchestra.run.smithi043.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:30:44.328 INFO:teuthology.orchestra.run.smithi043.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T00:30:44.329 INFO:teuthology.orchestra.run.smithi161.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T00:30:44.329 INFO:teuthology.orchestra.run.smithi161.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T00:30:44.329 INFO:teuthology.orchestra.run.smithi161.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T00:30:44.329 INFO:teuthology.orchestra.run.smithi161.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T00:30:44.335 INFO:teuthology.orchestra.run.smithi043.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T00:30:44.343 INFO:teuthology.orchestra.run.smithi043.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T00:30:44.344 INFO:teuthology.orchestra.run.smithi043.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T00:30:44.345 INFO:teuthology.orchestra.run.smithi043.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T00:30:44.345 INFO:teuthology.orchestra.run.smithi043.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T00:30:44.354 INFO:teuthology.orchestra.run.smithi043.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T00:30:44.354 INFO:teuthology.orchestra.run.smithi043.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T00:30:44.401 INFO:teuthology.orchestra.run.smithi161.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-19T00:30:44.442 INFO:teuthology.orchestra.run.smithi043.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-19T00:30:44.579 INFO:teuthology.orchestra.run.smithi043.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [616 kB] 2024-04-19T00:30:44.778 INFO:teuthology.orchestra.run.smithi043.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,597 kB] 2024-04-19T00:30:44.928 INFO:teuthology.orchestra.run.smithi043.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [302 kB] 2024-04-19T00:30:44.943 INFO:teuthology.orchestra.run.smithi043.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-19T00:30:44.944 INFO:teuthology.orchestra.run.smithi043.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-19T00:30:44.945 INFO:teuthology.orchestra.run.smithi043.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,806 kB] 2024-04-19T00:30:45.006 INFO:teuthology.orchestra.run.smithi043.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [305 kB] 2024-04-19T00:30:45.013 INFO:teuthology.orchestra.run.smithi043.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:30:45.038 INFO:teuthology.orchestra.run.smithi161.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,351 kB] 2024-04-19T00:30:45.177 INFO:teuthology.orchestra.run.smithi043.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-19T00:30:45.754 INFO:teuthology.orchestra.run.smithi043.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-19T00:30:45.865 INFO:teuthology.orchestra.run.smithi043.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-19T00:30:45.884 INFO:teuthology.orchestra.run.smithi043.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-19T00:30:45.886 INFO:teuthology.orchestra.run.smithi043.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-19T00:30:45.886 INFO:teuthology.orchestra.run.smithi043.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-19T00:30:45.890 INFO:teuthology.orchestra.run.smithi043.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-19T00:30:45.890 INFO:teuthology.orchestra.run.smithi043.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-19T00:30:45.891 INFO:teuthology.orchestra.run.smithi043.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-19T00:30:45.894 INFO:teuthology.orchestra.run.smithi043.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-19T00:30:45.899 INFO:teuthology.orchestra.run.smithi043.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-19T00:30:46.064 INFO:teuthology.orchestra.run.smithi043.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-19T00:30:46.147 INFO:teuthology.orchestra.run.smithi043.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-19T00:30:46.169 INFO:teuthology.orchestra.run.smithi161.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [238 kB] 2024-04-19T00:30:46.228 INFO:teuthology.orchestra.run.smithi043.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-19T00:30:46.274 INFO:teuthology.orchestra.run.smithi161.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-19T00:30:46.279 INFO:teuthology.orchestra.run.smithi161.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-19T00:30:46.298 INFO:teuthology.orchestra.run.smithi161.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,680 kB] 2024-04-19T00:30:46.309 INFO:teuthology.orchestra.run.smithi043.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-19T00:30:46.337 INFO:teuthology.orchestra.run.smithi043.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-19T00:30:48.366 INFO:teuthology.orchestra.run.smithi161.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [281 kB] 2024-04-19T00:30:48.811 INFO:teuthology.orchestra.run.smithi161.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-19T00:30:48.811 INFO:teuthology.orchestra.run.smithi161.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-19T00:30:49.705 INFO:teuthology.orchestra.run.smithi161.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-19T00:30:50.790 INFO:teuthology.orchestra.run.smithi161.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [162 kB] 2024-04-19T00:30:51.003 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 13.0 MB in 3s (5,158 kB/s) 2024-04-19T00:30:51.201 INFO:teuthology.orchestra.run.smithi161.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-19T00:30:51.207 INFO:teuthology.orchestra.run.smithi161.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-19T00:30:51.223 INFO:teuthology.orchestra.run.smithi161.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-19T00:30:51.223 INFO:teuthology.orchestra.run.smithi161.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-19T00:30:51.227 INFO:teuthology.orchestra.run.smithi161.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-19T00:30:52.535 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2024-04-19T00:30:52.554 DEBUG:teuthology.orchestra.run.smithi043:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:30:52.621 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2024-04-19T00:30:52.811 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2024-04-19T00:30:52.812 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2024-04-19T00:30:52.973 INFO:teuthology.orchestra.run.smithi043.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:30:52.973 INFO:teuthology.orchestra.run.smithi043.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:30:52.973 INFO:teuthology.orchestra.run.smithi043.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:30:52.974 INFO:teuthology.orchestra.run.smithi043.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:30:52.974 INFO:teuthology.orchestra.run.smithi043.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T00:30:52.974 INFO:teuthology.orchestra.run.smithi043.stdout: python2.7-minimal 2024-04-19T00:30:52.975 INFO:teuthology.orchestra.run.smithi043.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:30:52.975 INFO:teuthology.orchestra.run.smithi043.stdout:The following additional packages will be installed: 2024-04-19T00:30:52.976 INFO:teuthology.orchestra.run.smithi043.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:30:52.976 INFO:teuthology.orchestra.run.smithi043.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T00:30:52.976 INFO:teuthology.orchestra.run.smithi043.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T00:30:52.977 INFO:teuthology.orchestra.run.smithi043.stdout:Suggested packages: 2024-04-19T00:30:52.977 INFO:teuthology.orchestra.run.smithi043.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T00:30:52.977 INFO:teuthology.orchestra.run.smithi043.stdout:Recommended packages: 2024-04-19T00:30:52.977 INFO:teuthology.orchestra.run.smithi043.stdout: thermald 2024-04-19T00:30:53.024 INFO:teuthology.orchestra.run.smithi043.stdout:The following NEW packages will be installed: 2024-04-19T00:30:53.024 INFO:teuthology.orchestra.run.smithi043.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:30:53.024 INFO:teuthology.orchestra.run.smithi043.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T00:30:53.024 INFO:teuthology.orchestra.run.smithi043.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T00:30:53.025 INFO:teuthology.orchestra.run.smithi043.stdout:The following packages will be upgraded: 2024-04-19T00:30:53.026 INFO:teuthology.orchestra.run.smithi043.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T00:30:53.207 INFO:teuthology.orchestra.run.smithi043.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:30:53.207 INFO:teuthology.orchestra.run.smithi043.stdout:Need to get 113 MB of archives. 2024-04-19T00:30:53.207 INFO:teuthology.orchestra.run.smithi043.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T00:30:53.207 INFO:teuthology.orchestra.run.smithi043.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T00:30:55.458 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 13.0 MB in 7s (1,748 kB/s) 2024-04-19T00:30:56.837 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2024-04-19T00:30:56.855 DEBUG:teuthology.orchestra.run.smithi161:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:30:56.916 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2024-04-19T00:30:57.116 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2024-04-19T00:30:57.117 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2024-04-19T00:30:57.294 INFO:teuthology.orchestra.run.smithi161.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:30:57.294 INFO:teuthology.orchestra.run.smithi161.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:30:57.294 INFO:teuthology.orchestra.run.smithi161.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:30:57.294 INFO:teuthology.orchestra.run.smithi161.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:30:57.294 INFO:teuthology.orchestra.run.smithi161.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-19T00:30:57.295 INFO:teuthology.orchestra.run.smithi161.stdout: python2.7-minimal 2024-04-19T00:30:57.295 INFO:teuthology.orchestra.run.smithi161.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:30:57.295 INFO:teuthology.orchestra.run.smithi161.stdout:The following additional packages will be installed: 2024-04-19T00:30:57.295 INFO:teuthology.orchestra.run.smithi161.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:30:57.295 INFO:teuthology.orchestra.run.smithi161.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-19T00:30:57.296 INFO:teuthology.orchestra.run.smithi161.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-19T00:30:57.297 INFO:teuthology.orchestra.run.smithi161.stdout:Suggested packages: 2024-04-19T00:30:57.297 INFO:teuthology.orchestra.run.smithi161.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-19T00:30:57.297 INFO:teuthology.orchestra.run.smithi161.stdout:Recommended packages: 2024-04-19T00:30:57.297 INFO:teuthology.orchestra.run.smithi161.stdout: thermald 2024-04-19T00:30:57.344 INFO:teuthology.orchestra.run.smithi161.stdout:The following NEW packages will be installed: 2024-04-19T00:30:57.345 INFO:teuthology.orchestra.run.smithi161.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-19T00:30:57.345 INFO:teuthology.orchestra.run.smithi161.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-19T00:30:57.345 INFO:teuthology.orchestra.run.smithi161.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-19T00:30:57.346 INFO:teuthology.orchestra.run.smithi161.stdout:The following packages will be upgraded: 2024-04-19T00:30:57.347 INFO:teuthology.orchestra.run.smithi161.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-19T00:30:57.526 INFO:teuthology.orchestra.run.smithi161.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:30:57.526 INFO:teuthology.orchestra.run.smithi161.stdout:Need to get 113 MB of archives. 2024-04-19T00:30:57.527 INFO:teuthology.orchestra.run.smithi161.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-19T00:30:57.527 INFO:teuthology.orchestra.run.smithi161.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-19T00:30:59.175 INFO:teuthology.orchestra.run.smithi161.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T00:30:59.525 INFO:teuthology.orchestra.run.smithi161.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T00:31:01.754 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T00:31:01.754 INFO:teuthology.orchestra.run.smithi161.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T00:31:01.755 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T00:31:02.116 INFO:teuthology.orchestra.run.smithi161.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T00:31:02.197 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T00:31:02.580 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 113 MB in 5s (23.4 MB/s) 2024-04-19T00:31:02.731 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T00:31:04.769 INFO:teuthology.orchestra.run.smithi161.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T00:31:04.773 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:04.886 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:06.870 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T00:31:06.891 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:06.981 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:07.431 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T00:31:07.452 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:07.489 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:13.805 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:31:13.913 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:31:14.297 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:31:14.442 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:31:14.736 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T00:31:14.758 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T00:31:14.794 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:31:20.821 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T00:31:20.857 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:20.887 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:23.546 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:31:23.672 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:31:24.005 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:31:24.126 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:24.252 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T00:31:24.381 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:25.717 INFO:teuthology.orchestra.run.smithi161.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:31:25.718 INFO:teuthology.orchestra.run.smithi161.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T00:31:25.718 INFO:teuthology.orchestra.run.smithi161.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:31:25.718 INFO:teuthology.orchestra.run.smithi161.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T00:31:25.882 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:27.293 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:28.677 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T00:31:28.803 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T00:31:28.929 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:28.994 INFO:teuthology.orchestra.run.smithi161.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T00:31:28.994 INFO:teuthology.orchestra.run.smithi161.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:31:35.481 INFO:teuthology.orchestra.run.smithi043.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-19T00:31:36.476 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-19T00:31:38.776 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-19T00:31:38.858 INFO:teuthology.orchestra.run.smithi043.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-19T00:31:38.940 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-19T00:31:39.370 INFO:teuthology.orchestra.run.smithi043.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-19T00:31:39.527 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-19T00:31:39.962 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 113 MB in 46s (2,437 kB/s) 2024-04-19T00:31:40.154 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-19T00:31:42.251 INFO:teuthology.orchestra.run.smithi043.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-19T00:31:42.255 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:42.413 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:42.725 INFO:teuthology.orchestra.run.smithi161.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T00:31:42.726 INFO:teuthology.orchestra.run.smithi161.stdout:Sourcing file `/etc/default/grub' 2024-04-19T00:31:42.741 INFO:teuthology.orchestra.run.smithi161.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:31:42.766 INFO:teuthology.orchestra.run.smithi161.stdout:Generating grub configuration file ... 2024-04-19T00:31:43.112 INFO:teuthology.orchestra.run.smithi161.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:31:43.126 INFO:teuthology.orchestra.run.smithi161.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:31:43.440 INFO:teuthology.orchestra.run.smithi161.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:31:43.446 INFO:teuthology.orchestra.run.smithi161.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:31:43.517 INFO:teuthology.orchestra.run.smithi161.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:31:43.523 INFO:teuthology.orchestra.run.smithi161.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:31:43.686 INFO:teuthology.orchestra.run.smithi161.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:31:43.686 INFO:teuthology.orchestra.run.smithi161.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:31:43.687 INFO:teuthology.orchestra.run.smithi161.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:31:43.736 INFO:teuthology.orchestra.run.smithi161.stdout:done 2024-04-19T00:31:44.183 DEBUG:teuthology.orchestra.run.smithi161:> dpkg -s linux-image-generic 2024-04-19T00:31:44.209 INFO:teuthology.orchestra.run.smithi161.stdout:Package: linux-image-generic 2024-04-19T00:31:44.209 INFO:teuthology.orchestra.run.smithi161.stdout:Status: install ok installed 2024-04-19T00:31:44.209 INFO:teuthology.orchestra.run.smithi161.stdout:Priority: optional 2024-04-19T00:31:44.209 INFO:teuthology.orchestra.run.smithi161.stdout:Section: kernel 2024-04-19T00:31:44.209 INFO:teuthology.orchestra.run.smithi161.stdout:Installed-Size: 21 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: amd64 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Source: linux-meta 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Version: 5.15.0.105.102 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Recommends: thermald 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout:Description: Generic Linux kernel image 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:31:44.210 INFO:teuthology.orchestra.run.smithi161.stdout: available. 2024-04-19T00:31:44.210 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T00:31:44.211 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T00:31:44.211 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T00:31:44.211 INFO:teuthology.task.kernel:Installing distro kernel on mon.f... 2024-04-19T00:31:44.211 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T00:31:44.211 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get clean 2024-04-19T00:31:44.357 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get update 2024-04-19T00:31:44.474 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:31:44.474 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:31:44.487 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:31:44.508 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:31:44.661 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-19T00:31:44.682 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:44.769 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:45.265 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-19T00:31:45.285 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:31:45.456 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:31:45.808 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2024-04-19T00:31:45.828 DEBUG:teuthology.orchestra.run.smithi161:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:31:45.879 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2024-04-19T00:31:46.058 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2024-04-19T00:31:46.058 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2024-04-19T00:31:46.214 INFO:teuthology.orchestra.run.smithi161.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T00:31:46.214 INFO:teuthology.orchestra.run.smithi161.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:31:46.214 INFO:teuthology.orchestra.run.smithi161.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:31:46.214 INFO:teuthology.orchestra.run.smithi161.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:31:46.214 INFO:teuthology.orchestra.run.smithi161.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:31:46.215 INFO:teuthology.orchestra.run.smithi161.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:31:46.215 INFO:teuthology.orchestra.run.smithi161.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:31:46.215 INFO:teuthology.orchestra.run.smithi161.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:31:46.252 INFO:teuthology.orchestra.run.smithi161.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:31:46.254 DEBUG:teuthology.orchestra.run.smithi161:> dpkg -s linux-image-generic 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Package: linux-image-generic 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Status: install ok installed 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Priority: optional 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Section: kernel 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Installed-Size: 21 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: amd64 2024-04-19T00:31:46.311 INFO:teuthology.orchestra.run.smithi161.stdout:Source: linux-meta 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout:Version: 5.15.0.105.102 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout:Recommends: thermald 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout:Description: Generic Linux kernel image 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:31:46.312 INFO:teuthology.orchestra.run.smithi161.stdout: available. 2024-04-19T00:31:46.312 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-19T00:31:46.356 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.QfGM7eMsKc 2024-04-19T00:31:46.356 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QfGM7eMsKc 2024-04-19T00:31:46.411 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.QfGM7eMsKc 2024-04-19T00:31:46.538 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.QfGM7eMsKc is 10KB 2024-04-19T00:31:46.552 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.QfGM7eMsKc 2024-04-19T00:31:46.559 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T00:31:46.612 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:31:46.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T00:31:46.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T00:31:46.684 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T00:31:46.684 DEBUG:teuthology.orchestra.run.smithi161:> sudo update-grub 2024-04-19T00:31:47.846 INFO:teuthology.orchestra.run.smithi161.stderr:Sourcing file `/etc/default/grub' 2024-04-19T00:31:47.847 INFO:teuthology.orchestra.run.smithi161.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:31:47.847 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2024-04-19T00:31:48.051 INFO:teuthology.orchestra.run.smithi161.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:31:48.057 INFO:teuthology.orchestra.run.smithi161.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:31:48.305 INFO:teuthology.orchestra.run.smithi161.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:31:48.311 INFO:teuthology.orchestra.run.smithi161.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:31:48.354 INFO:teuthology.orchestra.run.smithi161.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:31:48.360 INFO:teuthology.orchestra.run.smithi161.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:31:48.482 INFO:teuthology.orchestra.run.smithi161.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:31:48.482 INFO:teuthology.orchestra.run.smithi161.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:31:48.482 INFO:teuthology.orchestra.run.smithi161.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:31:48.495 INFO:teuthology.orchestra.run.smithi161.stderr:done 2024-04-19T00:31:48.508 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2024-04-19T00:31:52.617 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:31:52.741 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:31:53.116 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:31:53.245 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:31:53.590 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-19T00:31:53.612 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-19T00:31:53.648 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:32:00.074 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-19T00:32:00.092 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-19T00:32:00.132 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:32:03.163 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-19T00:32:03.302 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-19T00:32:03.686 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-19T00:32:03.832 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:32:03.958 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-19T00:32:04.111 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:32:05.454 INFO:teuthology.orchestra.run.smithi043.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:32:05.454 INFO:teuthology.orchestra.run.smithi043.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-19T00:32:05.454 INFO:teuthology.orchestra.run.smithi043.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:32:05.455 INFO:teuthology.orchestra.run.smithi043.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-19T00:32:05.621 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:32:07.116 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:32:08.549 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-19T00:32:08.700 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-19T00:32:08.826 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-19T00:32:08.925 INFO:teuthology.orchestra.run.smithi043.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-19T00:32:08.925 INFO:teuthology.orchestra.run.smithi043.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:32:18.511 INFO:teuthology.task.kernel:Checking client mon.f for new kernel version... 2024-04-19T00:32:18.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:32:18.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:32:23.666 INFO:teuthology.orchestra.run.smithi043.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-19T00:32:23.667 INFO:teuthology.orchestra.run.smithi043.stdout:Sourcing file `/etc/default/grub' 2024-04-19T00:32:23.695 INFO:teuthology.orchestra.run.smithi043.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:32:23.751 INFO:teuthology.orchestra.run.smithi043.stdout:Generating grub configuration file ... 2024-04-19T00:32:24.156 INFO:teuthology.orchestra.run.smithi043.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:32:24.183 INFO:teuthology.orchestra.run.smithi043.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:32:24.553 INFO:teuthology.orchestra.run.smithi043.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:32:24.559 INFO:teuthology.orchestra.run.smithi043.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:32:24.626 INFO:teuthology.orchestra.run.smithi043.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:32:24.632 INFO:teuthology.orchestra.run.smithi043.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:32:24.835 INFO:teuthology.orchestra.run.smithi043.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:32:24.835 INFO:teuthology.orchestra.run.smithi043.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:32:24.835 INFO:teuthology.orchestra.run.smithi043.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:32:24.869 INFO:teuthology.orchestra.run.smithi043.stdout:done 2024-04-19T00:32:25.433 DEBUG:teuthology.orchestra.run.smithi043:> dpkg -s linux-image-generic 2024-04-19T00:32:25.463 INFO:teuthology.orchestra.run.smithi043.stdout:Package: linux-image-generic 2024-04-19T00:32:25.463 INFO:teuthology.orchestra.run.smithi043.stdout:Status: install ok installed 2024-04-19T00:32:25.463 INFO:teuthology.orchestra.run.smithi043.stdout:Priority: optional 2024-04-19T00:32:25.463 INFO:teuthology.orchestra.run.smithi043.stdout:Section: kernel 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Installed-Size: 21 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: amd64 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Source: linux-meta 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Version: 5.15.0.105.102 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Recommends: thermald 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout:Description: Generic Linux kernel image 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:32:25.464 INFO:teuthology.orchestra.run.smithi043.stdout: available. 2024-04-19T00:32:25.464 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-19T00:32:25.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-19T00:32:25.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-19T00:32:25.465 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-19T00:32:25.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi043.front.sepia.ceph.com, path=None, version=distro) 2024-04-19T00:32:25.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get clean 2024-04-19T00:32:25.648 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get update 2024-04-19T00:32:25.790 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:32:25.800 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:32:25.822 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:32:25.903 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-19T00:32:26.351 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 110 kB in 1s (208 kB/s) 2024-04-19T00:32:27.320 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2024-04-19T00:32:27.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-19T00:32:27.404 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2024-04-19T00:32:27.609 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2024-04-19T00:32:27.610 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2024-04-19T00:32:27.782 INFO:teuthology.orchestra.run.smithi043.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-19T00:32:27.782 INFO:teuthology.orchestra.run.smithi043.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:32:27.782 INFO:teuthology.orchestra.run.smithi043.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:32:27.783 INFO:teuthology.orchestra.run.smithi043.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:32:27.783 INFO:teuthology.orchestra.run.smithi043.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:32:27.783 INFO:teuthology.orchestra.run.smithi043.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:32:27.783 INFO:teuthology.orchestra.run.smithi043.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:32:27.783 INFO:teuthology.orchestra.run.smithi043.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:32:27.823 INFO:teuthology.orchestra.run.smithi043.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-19T00:32:27.825 DEBUG:teuthology.orchestra.run.smithi043:> dpkg -s linux-image-generic 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Package: linux-image-generic 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Status: install ok installed 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Priority: optional 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Section: kernel 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Installed-Size: 21 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Maintainer: Ubuntu Kernel Team 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Architecture: amd64 2024-04-19T00:32:27.841 INFO:teuthology.orchestra.run.smithi043.stdout:Source: linux-meta 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout:Version: 5.15.0.105.102 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout:Recommends: thermald 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout:Description: Generic Linux kernel image 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout: This package will always depend on the latest generic kernel image 2024-04-19T00:32:27.842 INFO:teuthology.orchestra.run.smithi043.stdout: available. 2024-04-19T00:32:27.842 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-19T00:32:27.887 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.jT63UnUCCC 2024-04-19T00:32:27.888 DEBUG:teuthology.orchestra.run.smithi043:> sudo cp /boot/grub/grub.cfg /tmp/tmp.jT63UnUCCC 2024-04-19T00:32:27.942 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.jT63UnUCCC 2024-04-19T00:32:28.057 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.jT63UnUCCC is 10KB 2024-04-19T00:32:28.070 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.jT63UnUCCC 2024-04-19T00:32:28.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-19T00:32:28.131 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:32:28.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-19T00:32:28.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-19T00:32:28.217 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-19T00:32:28.217 DEBUG:teuthology.orchestra.run.smithi043:> sudo update-grub 2024-04-19T00:32:29.646 INFO:teuthology.orchestra.run.smithi043.stderr:Sourcing file `/etc/default/grub' 2024-04-19T00:32:29.647 INFO:teuthology.orchestra.run.smithi043.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-19T00:32:29.657 INFO:teuthology.orchestra.run.smithi043.stderr:Generating grub configuration file ... 2024-04-19T00:32:29.909 INFO:teuthology.orchestra.run.smithi043.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-19T00:32:29.916 INFO:teuthology.orchestra.run.smithi043.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-19T00:32:30.212 INFO:teuthology.orchestra.run.smithi043.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-19T00:32:30.217 INFO:teuthology.orchestra.run.smithi043.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-19T00:32:30.266 INFO:teuthology.orchestra.run.smithi043.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-19T00:32:30.271 INFO:teuthology.orchestra.run.smithi043.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-19T00:32:30.397 INFO:teuthology.orchestra.run.smithi043.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-19T00:32:30.397 INFO:teuthology.orchestra.run.smithi043.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-19T00:32:30.397 INFO:teuthology.orchestra.run.smithi043.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-19T00:32:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:done 2024-04-19T00:32:30.410 DEBUG:teuthology.orchestra.run.smithi043:> sudo shutdown -r now 2024-04-19T00:32:36.898 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-04-19T00:32:45.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:32:45.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:32:48.961 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-04-19T00:33:00.466 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-19T00:33:00.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-19T00:33:00.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:33:00.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:33:00.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:33:18.882 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.43 2024-04-19T00:33:27.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-19T00:33:27.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:34:00.999 DEBUG:teuthology.orchestra.remote:timed out 2024-04-19T00:34:16.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:34:16.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:34:16.256 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-04-19T00:34:16.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:34:16.763 INFO:teuthology.task.kernel:Checking kernel version of mon.f, want "5.15.0-105-generic"... 2024-04-19T00:34:16.763 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-04-19T00:34:16.810 INFO:teuthology.orchestra.run.smithi161.stdout:5.15.0-105-generic 2024-04-19T00:34:16.810 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T00:34:16.811 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T00:34:16.811 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T00:34:17.811 INFO:teuthology.task.kernel:Enabling kdb on mon.f... 2024-04-19T00:34:17.812 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T00:34:17.889 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-04-19T00:34:17.923 DEBUG:teuthology.parallel:result is None 2024-04-19T00:34:27.896 DEBUG:teuthology.orchestra.remote:timed out 2024-04-19T00:34:39.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-19T00:34:39.898 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:34:40.164 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-04-19T00:34:40.740 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-19T00:34:40.740 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-19T00:34:40.741 DEBUG:teuthology.orchestra.run.smithi043:> uname -r 2024-04-19T00:34:40.791 INFO:teuthology.orchestra.run.smithi043.stdout:5.15.0-105-generic 2024-04-19T00:34:40.791 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-19T00:34:40.791 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-19T00:34:40.791 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-19T00:34:41.792 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-19T00:34:41.792 DEBUG:teuthology.orchestra.run.smithi043:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-19T00:34:41.865 INFO:teuthology.orchestra.run.smithi043.stdout:ttyS1 2024-04-19T00:34:41.899 DEBUG:teuthology.parallel:result is None 2024-04-19T00:34:41.899 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-19T00:34:41.908 INFO:teuthology.task.internal:Creating test directory... 2024-04-19T00:34:41.908 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T00:34:41.911 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-19T00:34:41.918 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-19T00:34:41.936 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-19T00:34:41.970 INFO:teuthology.task.internal:Creating archive directory... 2024-04-19T00:34:41.971 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T00:34:41.974 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-19T00:34:42.046 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-19T00:34:42.054 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-19T00:34:42.055 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T00:34:42.057 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-19T00:34:42.079 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:34:42.087 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:34:42.089 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:34:42.098 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-19T00:34:42.100 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-19T00:34:42.107 INFO:teuthology.task.internal:Configuring sudo... 2024-04-19T00:34:42.108 DEBUG:teuthology.orchestra.run.smithi043:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T00:34:42.136 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-19T00:34:42.153 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-19T00:34:42.163 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-19T00:34:42.163 DEBUG:teuthology.orchestra.run.smithi043:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T00:34:42.191 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-19T00:34:42.201 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T00:34:42.242 DEBUG:teuthology.orchestra.run.smithi043:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T00:34:42.291 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:34:42.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T00:34:42.361 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-19T00:34:42.367 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-19T00:34:42.415 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:34:42.415 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-19T00:34:42.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo service rsyslog restart 2024-04-19T00:34:42.483 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-04-19T00:34:42.556 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-19T00:34:42.571 INFO:teuthology.task.internal:Starting timer... 2024-04-19T00:34:42.571 INFO:teuthology.run_tasks:Running task pcp... 2024-04-19T00:34:42.608 INFO:teuthology.run_tasks:Running task selinux... 2024-04-19T00:34:42.642 DEBUG:teuthology.task.selinux:Excluding smithi043: OS 'ubuntu' does not support SELinux 2024-04-19T00:34:42.642 DEBUG:teuthology.task.selinux:Excluding smithi161: OS 'ubuntu' does not support SELinux 2024-04-19T00:34:42.642 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-19T00:34:42.642 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-19T00:34:42.643 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-19T00:34:42.643 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-19T00:34:42.695 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-19T00:34:42.713 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-19T00:34:42.714 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi043.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-19T00:41:53.488 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi043.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-04-19T00:41:53.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-19T00:41:53.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:41:53.580 DEBUG:teuthology.orchestra.run.smithi043:> true 2024-04-19T00:41:53.658 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi043.front.sepia.ceph.com' 2024-04-19T00:41:53.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:41:53.659 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-19T00:41:53.741 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-04-19T00:41:53.818 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-04-19T00:41:53.818 INFO:teuthology.run_tasks:Running task clock... 2024-04-19T00:41:53.829 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-19T00:41:53.830 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T00:41:53.830 DEBUG:teuthology.orchestra.run.smithi043:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T00:41:53.832 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-19T00:41:53.833 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T00:41:53.863 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T00:41:53.863 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Command line: ntpd -gq 2024-04-19T00:41:53.863 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: ---------------------------------------------------- 2024-04-19T00:41:53.863 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: corporation. Support and training for ntp-4 are 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: available at https://www.nwtime.org/support 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: ---------------------------------------------------- 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: proto: precision = 0.048 usec (-24) 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: basedate set to 2022-02-04 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: gps base set to 2022-02-06 (week 2196) 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T00:41:53.864 INFO:teuthology.orchestra.run.smithi043.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T00:41:53.865 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T00:41:53.865 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T00:41:53.865 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T00:41:53.866 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T00:41:53.866 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listen normally on 3 ens1f0 172.21.15.43:123 2024-04-19T00:41:53.866 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listen normally on 4 lo [::1]:123 2024-04-19T00:41:53.866 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1358%4]:123 2024-04-19T00:41:53.866 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:41:53 ntpd[17126]: Listening on routing socket on fd #22 for interface updates 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Command line: ntpd -gq 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: ---------------------------------------------------- 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: ntp-4 is maintained by Network Time Foundation, 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: corporation. Support and training for ntp-4 are 2024-04-19T00:41:53.887 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: available at https://www.nwtime.org/support 2024-04-19T00:41:53.888 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: ---------------------------------------------------- 2024-04-19T00:41:53.888 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: proto: precision = 0.055 usec (-24) 2024-04-19T00:41:53.888 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: basedate set to 2022-02-04 2024-04-19T00:41:53.888 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: gps base set to 2022-02-06 (week 2196) 2024-04-19T00:41:53.888 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T00:41:53.889 INFO:teuthology.orchestra.run.smithi161.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-19T00:41:53.889 INFO:teuthology.orchestra.run.smithi161.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: restrict ::: KOD does nothing without LIMITED. 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listen and drop on 0 v6wildcard [::]:123 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listen normally on 2 lo 127.0.0.1:123 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listen normally on 3 enp3s0f1 172.21.15.161:123 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listen normally on 4 lo [::1]:123 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7251%5]:123 2024-04-19T00:41:53.890 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:41:53 ntpd[17265]: Listening on routing socket on fd #22 for interface updates 2024-04-19T00:41:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.867 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.869 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:54.870 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.871 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:54.872 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:54 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:54.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:54 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:54 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.890 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:54 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:54.890 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:54 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:54.890 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:54 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:54.890 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:54 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:56.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:56.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:56.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:56.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:56 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:56.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:56.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:56 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:57.887 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:57 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:57.887 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:57 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:57.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:57 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:57.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:57 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:57.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:57 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:57.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:57 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:58.866 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:41:58 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:58.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:58.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:58 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:41:59.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:59 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:59.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:59 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:59.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:59 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:41:59.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:59 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:41:59.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:59 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:41:59.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:41:59 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:42:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stdout:19 Apr 00:42:00 ntpd[17126]: ntpd: time slew +0.000627 s 2024-04-19T00:42:00.864 INFO:teuthology.orchestra.run.smithi043.stdout:ntpd: time slew +0.000627s 2024-04-19T00:42:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T00:42:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:19 Apr 00:42:00 ntpd[17126]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/rawstats.20240419: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-19T00:42:00.888 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/peerstats.20240419: Permission denied 2024-04-19T00:42:00.889 INFO:teuthology.orchestra.run.smithi161.stdout:19 Apr 00:42:00 ntpd[17265]: ntpd: time slew -0.003581 s 2024-04-19T00:42:00.889 INFO:teuthology.orchestra.run.smithi161.stdout:ntpd: time slew -0.003581s 2024-04-19T00:42:00.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-19T00:42:00.889 INFO:teuthology.orchestra.run.smithi161.stderr:19 Apr 00:42:00 ntpd[17265]: can't open /var/log/ntpstats/loopstats.20240419: Permission denied 2024-04-19T00:42:00.917 INFO:teuthology.orchestra.run.smithi043.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T00:42:00.917 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================== 2024-04-19T00:42:00.917 INFO:teuthology.orchestra.run.smithi043.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.917 INFO:teuthology.orchestra.run.smithi043.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.917 INFO:teuthology.orchestra.run.smithi043.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.917 INFO:teuthology.orchestra.run.smithi043.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.942 INFO:teuthology.orchestra.run.smithi161.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T00:42:00.942 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================== 2024-04-19T00:42:00.942 INFO:teuthology.orchestra.run.smithi161.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.942 INFO:teuthology.orchestra.run.smithi161.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.942 INFO:teuthology.orchestra.run.smithi161.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.942 INFO:teuthology.orchestra.run.smithi161.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-19T00:42:00.943 INFO:teuthology.run_tasks:Running task install... 2024-04-19T00:42:00.962 DEBUG:teuthology.task.install:project ceph 2024-04-19T00:42:00.962 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}} 2024-04-19T00:42:00.963 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'} 2024-04-19T00:42:00.963 INFO:teuthology.task.install:Using flavor: default 2024-04-19T00:42:00.968 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-19T00:42:00.968 INFO:teuthology.task.install:extra packages: [] 2024-04-19T00:42:00.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-key list | grep Ceph 2024-04-19T00:42:00.969 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-key list | grep Ceph 2024-04-19T00:42:01.032 INFO:teuthology.orchestra.run.smithi043.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T00:42:01.037 INFO:teuthology.orchestra.run.smithi161.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-19T00:42:01.062 INFO:teuthology.orchestra.run.smithi043.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T00:42:01.062 INFO:teuthology.orchestra.run.smithi043.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T00:42:01.062 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T00:42:01.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:42:01.067 INFO:teuthology.orchestra.run.smithi161.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-19T00:42:01.068 INFO:teuthology.orchestra.run.smithi161.stdout:uid [ unknown] Ceph.com (release key) 2024-04-19T00:42:01.068 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-19T00:42:01.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:42:01.207 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T00:42:01.207 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:42:01.283 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default/ 2024-04-19T00:42:01.283 INFO:teuthology.task.install.deb:Package version is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:42:01.319 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:42:01.319 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T00:42:01.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo apt-get update 2024-04-19T00:42:01.450 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:42:01.450 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-19T00:42:01.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo apt-get update 2024-04-19T00:42:01.508 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:42:01.509 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:42:01.521 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:42:01.541 INFO:teuthology.orchestra.run.smithi043.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:42:01.557 INFO:teuthology.orchestra.run.smithi043.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T00:42:01.578 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T00:42:01.601 INFO:teuthology.orchestra.run.smithi043.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T00:42:01.623 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T00:42:01.639 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-19T00:42:01.639 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-19T00:42:01.653 INFO:teuthology.orchestra.run.smithi043.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T00:42:01.653 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-19T00:42:01.674 INFO:teuthology.orchestra.run.smithi161.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-19T00:42:01.689 INFO:teuthology.orchestra.run.smithi161.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy InRelease 2024-04-19T00:42:01.710 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-19T00:42:01.732 INFO:teuthology.orchestra.run.smithi161.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-19T00:42:01.753 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-19T00:42:01.780 INFO:teuthology.orchestra.run.smithi161.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,408 B] 2024-04-19T00:42:01.881 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 29.2 kB in 0s (80.9 kB/s) 2024-04-19T00:42:01.993 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 29.2 kB in 0s (83.5 kB/s) 2024-04-19T00:42:02.883 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2024-04-19T00:42:02.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:42:02.915 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2024-04-19T00:42:02.931 DEBUG:teuthology.orchestra.run.smithi161:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3154-g8b4ab94f-1jammy cephadm=19.0.0-3154-g8b4ab94f-1jammy ceph-mds=19.0.0-3154-g8b4ab94f-1jammy ceph-mgr=19.0.0-3154-g8b4ab94f-1jammy ceph-common=19.0.0-3154-g8b4ab94f-1jammy ceph-fuse=19.0.0-3154-g8b4ab94f-1jammy ceph-test=19.0.0-3154-g8b4ab94f-1jammy ceph-volume=19.0.0-3154-g8b4ab94f-1jammy radosgw=19.0.0-3154-g8b4ab94f-1jammy python3-rados=19.0.0-3154-g8b4ab94f-1jammy python3-rgw=19.0.0-3154-g8b4ab94f-1jammy python3-cephfs=19.0.0-3154-g8b4ab94f-1jammy python3-rbd=19.0.0-3154-g8b4ab94f-1jammy libcephfs2=19.0.0-3154-g8b4ab94f-1jammy libcephfs-dev=19.0.0-3154-g8b4ab94f-1jammy librados2=19.0.0-3154-g8b4ab94f-1jammy librbd1=19.0.0-3154-g8b4ab94f-1jammy rbd-fuse=19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:42:02.960 INFO:teuthology.orchestra.run.smithi043.stdout:Reading package lists... 2024-04-19T00:42:02.983 INFO:teuthology.orchestra.run.smithi161.stdout:Reading package lists... 2024-04-19T00:42:03.161 INFO:teuthology.orchestra.run.smithi161.stdout:Building dependency tree... 2024-04-19T00:42:03.162 INFO:teuthology.orchestra.run.smithi161.stdout:Reading state information... 2024-04-19T00:42:03.163 INFO:teuthology.orchestra.run.smithi043.stdout:Building dependency tree... 2024-04-19T00:42:03.164 INFO:teuthology.orchestra.run.smithi043.stdout:Reading state information... 2024-04-19T00:42:03.334 INFO:teuthology.orchestra.run.smithi161.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:42:03.334 INFO:teuthology.orchestra.run.smithi161.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:42:03.334 INFO:teuthology.orchestra.run.smithi161.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:42:03.334 INFO:teuthology.orchestra.run.smithi161.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:42:03.335 INFO:teuthology.orchestra.run.smithi161.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:42:03.336 INFO:teuthology.orchestra.run.smithi161.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:42:03.336 INFO:teuthology.orchestra.run.smithi161.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:42:03.336 INFO:teuthology.orchestra.run.smithi161.stdout:The following additional packages will be installed: 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T00:42:03.337 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T00:42:03.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T00:42:03.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T00:42:03.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T00:42:03.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T00:42:03.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T00:42:03.338 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T00:42:03.339 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet zip 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout:Suggested packages: 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T00:42:03.340 INFO:teuthology.orchestra.run.smithi161.stdout:Recommended packages: 2024-04-19T00:42:03.341 INFO:teuthology.orchestra.run.smithi161.stdout: btrfs-tools 2024-04-19T00:42:03.369 INFO:teuthology.orchestra.run.smithi043.stdout:The following packages were automatically installed and are no longer required: 2024-04-19T00:42:03.369 INFO:teuthology.orchestra.run.smithi043.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-19T00:42:03.370 INFO:teuthology.orchestra.run.smithi043.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-19T00:42:03.370 INFO:teuthology.orchestra.run.smithi043.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-19T00:42:03.370 INFO:teuthology.orchestra.run.smithi043.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-19T00:42:03.371 INFO:teuthology.orchestra.run.smithi043.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-19T00:42:03.371 INFO:teuthology.orchestra.run.smithi043.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-19T00:42:03.371 INFO:teuthology.orchestra.run.smithi043.stdout:The following additional packages will be installed: 2024-04-19T00:42:03.371 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-19T00:42:03.372 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-19T00:42:03.372 INFO:teuthology.orchestra.run.smithi043.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-19T00:42:03.372 INFO:teuthology.orchestra.run.smithi043.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-19T00:42:03.372 INFO:teuthology.orchestra.run.smithi043.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T00:42:03.374 INFO:teuthology.orchestra.run.smithi043.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout: xmlstarlet zip 2024-04-19T00:42:03.375 INFO:teuthology.orchestra.run.smithi043.stdout:Suggested packages: 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout:Recommended packages: 2024-04-19T00:42:03.376 INFO:teuthology.orchestra.run.smithi043.stdout: btrfs-tools 2024-04-19T00:42:03.438 INFO:teuthology.orchestra.run.smithi161.stdout:The following NEW packages will be installed: 2024-04-19T00:42:03.439 INFO:teuthology.orchestra.run.smithi161.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T00:42:03.439 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T00:42:03.439 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T00:42:03.439 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T00:42:03.439 INFO:teuthology.orchestra.run.smithi161.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T00:42:03.439 INFO:teuthology.orchestra.run.smithi161.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T00:42:03.440 INFO:teuthology.orchestra.run.smithi161.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T00:42:03.440 INFO:teuthology.orchestra.run.smithi161.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T00:42:03.440 INFO:teuthology.orchestra.run.smithi161.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T00:42:03.440 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T00:42:03.440 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T00:42:03.440 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T00:42:03.441 INFO:teuthology.orchestra.run.smithi161.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout:The following NEW packages will be installed: 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-19T00:42:03.483 INFO:teuthology.orchestra.run.smithi043.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-19T00:42:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-19T00:42:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-19T00:42:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-19T00:42:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-19T00:42:03.485 INFO:teuthology.orchestra.run.smithi043.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-19T00:42:03.486 INFO:teuthology.orchestra.run.smithi043.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-19T00:42:03.487 INFO:teuthology.orchestra.run.smithi043.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-19T00:42:03.551 INFO:teuthology.orchestra.run.smithi043.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T00:42:03.551 INFO:teuthology.orchestra.run.smithi043.stdout:Need to get 181 MB of archives. 2024-04-19T00:42:03.551 INFO:teuthology.orchestra.run.smithi043.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T00:42:03.551 INFO:teuthology.orchestra.run.smithi043.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T00:42:03.581 INFO:teuthology.orchestra.run.smithi161.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-19T00:42:03.581 INFO:teuthology.orchestra.run.smithi161.stdout:Need to get 181 MB of archives. 2024-04-19T00:42:03.581 INFO:teuthology.orchestra.run.smithi161.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-19T00:42:03.581 INFO:teuthology.orchestra.run.smithi161.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T00:42:03.630 INFO:teuthology.orchestra.run.smithi161.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-19T00:42:03.662 INFO:teuthology.orchestra.run.smithi043.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T00:42:03.665 INFO:teuthology.orchestra.run.smithi043.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T00:42:03.686 INFO:teuthology.orchestra.run.smithi043.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T00:42:03.702 INFO:teuthology.orchestra.run.smithi043.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,555 kB] 2024-04-19T00:42:03.756 INFO:teuthology.orchestra.run.smithi043.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T00:42:03.760 INFO:teuthology.orchestra.run.smithi043.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T00:42:03.773 INFO:teuthology.orchestra.run.smithi043.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T00:42:03.778 INFO:teuthology.orchestra.run.smithi043.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T00:42:03.778 INFO:teuthology.orchestra.run.smithi043.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T00:42:03.779 INFO:teuthology.orchestra.run.smithi043.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T00:42:03.788 INFO:teuthology.orchestra.run.smithi043.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T00:42:03.790 INFO:teuthology.orchestra.run.smithi043.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T00:42:03.793 INFO:teuthology.orchestra.run.smithi043.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T00:42:03.794 INFO:teuthology.orchestra.run.smithi043.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T00:42:03.802 INFO:teuthology.orchestra.run.smithi043.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T00:42:03.804 INFO:teuthology.orchestra.run.smithi043.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T00:42:03.806 INFO:teuthology.orchestra.run.smithi043.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T00:42:03.823 INFO:teuthology.orchestra.run.smithi043.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T00:42:03.823 INFO:teuthology.orchestra.run.smithi043.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T00:42:03.824 INFO:teuthology.orchestra.run.smithi043.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T00:42:03.825 INFO:teuthology.orchestra.run.smithi043.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T00:42:03.825 INFO:teuthology.orchestra.run.smithi043.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T00:42:03.826 INFO:teuthology.orchestra.run.smithi043.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T00:42:03.826 INFO:teuthology.orchestra.run.smithi043.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T00:42:03.828 INFO:teuthology.orchestra.run.smithi161.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T00:42:03.843 INFO:teuthology.orchestra.run.smithi043.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T00:42:03.844 INFO:teuthology.orchestra.run.smithi043.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T00:42:03.844 INFO:teuthology.orchestra.run.smithi043.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T00:42:03.864 INFO:teuthology.orchestra.run.smithi043.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T00:42:03.865 INFO:teuthology.orchestra.run.smithi043.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T00:42:03.865 INFO:teuthology.orchestra.run.smithi043.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T00:42:03.866 INFO:teuthology.orchestra.run.smithi043.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T00:42:03.867 INFO:teuthology.orchestra.run.smithi043.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T00:42:03.867 INFO:teuthology.orchestra.run.smithi043.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T00:42:03.868 INFO:teuthology.orchestra.run.smithi043.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T00:42:03.885 INFO:teuthology.orchestra.run.smithi043.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T00:42:03.886 INFO:teuthology.orchestra.run.smithi043.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T00:42:03.886 INFO:teuthology.orchestra.run.smithi043.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T00:42:03.906 INFO:teuthology.orchestra.run.smithi043.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T00:42:03.907 INFO:teuthology.orchestra.run.smithi043.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T00:42:03.907 INFO:teuthology.orchestra.run.smithi043.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T00:42:03.908 INFO:teuthology.orchestra.run.smithi043.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T00:42:03.909 INFO:teuthology.orchestra.run.smithi043.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T00:42:03.911 INFO:teuthology.orchestra.run.smithi043.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T00:42:03.912 INFO:teuthology.orchestra.run.smithi043.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T00:42:03.927 INFO:teuthology.orchestra.run.smithi043.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T00:42:03.928 INFO:teuthology.orchestra.run.smithi043.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T00:42:03.930 INFO:teuthology.orchestra.run.smithi043.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T00:42:03.948 INFO:teuthology.orchestra.run.smithi043.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T00:42:03.949 INFO:teuthology.orchestra.run.smithi043.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T00:42:03.951 INFO:teuthology.orchestra.run.smithi043.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T00:42:03.974 INFO:teuthology.orchestra.run.smithi161.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T00:42:04.003 INFO:teuthology.orchestra.run.smithi161.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T00:42:04.007 INFO:teuthology.orchestra.run.smithi043.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T00:42:04.008 INFO:teuthology.orchestra.run.smithi043.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T00:42:04.010 INFO:teuthology.orchestra.run.smithi043.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T00:42:04.010 INFO:teuthology.orchestra.run.smithi043.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T00:42:04.011 INFO:teuthology.orchestra.run.smithi043.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T00:42:04.022 INFO:teuthology.orchestra.run.smithi161.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T00:42:04.025 INFO:teuthology.orchestra.run.smithi161.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T00:42:04.030 INFO:teuthology.orchestra.run.smithi043.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T00:42:04.032 INFO:teuthology.orchestra.run.smithi043.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T00:42:04.032 INFO:teuthology.orchestra.run.smithi043.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T00:42:04.032 INFO:teuthology.orchestra.run.smithi161.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T00:42:04.035 INFO:teuthology.orchestra.run.smithi161.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T00:42:04.047 INFO:teuthology.orchestra.run.smithi043.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T00:42:04.047 INFO:teuthology.orchestra.run.smithi043.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T00:42:04.050 INFO:teuthology.orchestra.run.smithi161.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T00:42:04.052 INFO:teuthology.orchestra.run.smithi043.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T00:42:04.053 INFO:teuthology.orchestra.run.smithi043.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T00:42:04.053 INFO:teuthology.orchestra.run.smithi043.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T00:42:04.053 INFO:teuthology.orchestra.run.smithi043.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T00:42:04.073 INFO:teuthology.orchestra.run.smithi043.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T00:42:04.073 INFO:teuthology.orchestra.run.smithi043.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T00:42:04.077 INFO:teuthology.orchestra.run.smithi043.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T00:42:04.078 INFO:teuthology.orchestra.run.smithi043.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T00:42:04.079 INFO:teuthology.orchestra.run.smithi043.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T00:42:04.080 INFO:teuthology.orchestra.run.smithi043.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T00:42:04.084 INFO:teuthology.orchestra.run.smithi161.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-19T00:42:04.094 INFO:teuthology.orchestra.run.smithi043.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T00:42:04.096 INFO:teuthology.orchestra.run.smithi043.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T00:42:04.097 INFO:teuthology.orchestra.run.smithi043.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T00:42:04.098 INFO:teuthology.orchestra.run.smithi043.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T00:42:04.098 INFO:teuthology.orchestra.run.smithi161.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-19T00:42:04.115 INFO:teuthology.orchestra.run.smithi043.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T00:42:04.115 INFO:teuthology.orchestra.run.smithi043.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T00:42:04.118 INFO:teuthology.orchestra.run.smithi043.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T00:42:04.119 INFO:teuthology.orchestra.run.smithi043.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T00:42:04.119 INFO:teuthology.orchestra.run.smithi043.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T00:42:04.119 INFO:teuthology.orchestra.run.smithi043.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T00:42:04.136 INFO:teuthology.orchestra.run.smithi043.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T00:42:04.137 INFO:teuthology.orchestra.run.smithi043.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T00:42:04.144 INFO:teuthology.orchestra.run.smithi043.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T00:42:04.144 INFO:teuthology.orchestra.run.smithi043.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T00:42:04.157 INFO:teuthology.orchestra.run.smithi043.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T00:42:04.159 INFO:teuthology.orchestra.run.smithi043.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T00:42:04.160 INFO:teuthology.orchestra.run.smithi043.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T00:42:04.181 INFO:teuthology.orchestra.run.smithi161.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-19T00:42:04.228 INFO:teuthology.orchestra.run.smithi043.stdout:Get:89 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3154-g8b4ab94f-1jammy [3,224 kB] 2024-04-19T00:42:04.364 INFO:teuthology.orchestra.run.smithi161.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T00:42:04.367 INFO:teuthology.orchestra.run.smithi161.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T00:42:04.384 INFO:teuthology.orchestra.run.smithi161.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T00:42:04.421 INFO:teuthology.orchestra.run.smithi161.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-19T00:42:04.432 INFO:teuthology.orchestra.run.smithi161.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-19T00:42:04.463 INFO:teuthology.orchestra.run.smithi161.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-19T00:42:04.472 INFO:teuthology.orchestra.run.smithi161.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-19T00:42:04.473 INFO:teuthology.orchestra.run.smithi161.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-19T00:42:04.474 INFO:teuthology.orchestra.run.smithi161.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-19T00:42:04.493 INFO:teuthology.orchestra.run.smithi161.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-19T00:42:04.562 INFO:teuthology.orchestra.run.smithi043.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3154-g8b4ab94f-1jammy [712 kB] 2024-04-19T00:42:04.640 INFO:teuthology.orchestra.run.smithi043.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3154-g8b4ab94f-1jammy [356 kB] 2024-04-19T00:42:04.661 INFO:teuthology.orchestra.run.smithi161.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-19T00:42:04.680 INFO:teuthology.orchestra.run.smithi043.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3154-g8b4ab94f-1jammy [32.8 kB] 2024-04-19T00:42:04.687 INFO:teuthology.orchestra.run.smithi043.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3154-g8b4ab94f-1jammy [180 kB] 2024-04-19T00:42:04.706 INFO:teuthology.orchestra.run.smithi043.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3154-g8b4ab94f-1jammy [66.9 kB] 2024-04-19T00:42:04.712 INFO:teuthology.orchestra.run.smithi043.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3154-g8b4ab94f-1jammy [330 kB] 2024-04-19T00:42:04.748 INFO:teuthology.orchestra.run.smithi043.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3154-g8b4ab94f-1jammy [6,894 kB] 2024-04-19T00:42:05.068 INFO:teuthology.orchestra.run.smithi161.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-19T00:42:05.102 INFO:teuthology.orchestra.run.smithi161.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-19T00:42:05.104 INFO:teuthology.orchestra.run.smithi161.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-19T00:42:05.184 INFO:teuthology.orchestra.run.smithi161.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-19T00:42:05.240 INFO:teuthology.orchestra.run.smithi161.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-19T00:42:05.262 INFO:teuthology.orchestra.run.smithi161.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-19T00:42:05.268 INFO:teuthology.orchestra.run.smithi161.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-19T00:42:05.270 INFO:teuthology.orchestra.run.smithi161.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-19T00:42:05.281 INFO:teuthology.orchestra.run.smithi161.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-19T00:42:05.282 INFO:teuthology.orchestra.run.smithi161.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-19T00:42:05.448 INFO:teuthology.orchestra.run.smithi161.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-19T00:42:05.483 INFO:teuthology.orchestra.run.smithi043.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3154-g8b4ab94f-1jammy [112 kB] 2024-04-19T00:42:05.497 INFO:teuthology.orchestra.run.smithi043.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3154-g8b4ab94f-1jammy [466 kB] 2024-04-19T00:42:05.531 INFO:teuthology.orchestra.run.smithi161.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-19T00:42:05.559 INFO:teuthology.orchestra.run.smithi161.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-19T00:42:05.561 INFO:teuthology.orchestra.run.smithi043.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3154-g8b4ab94f-1jammy [25.8 MB] 2024-04-19T00:42:05.576 INFO:teuthology.orchestra.run.smithi161.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-19T00:42:05.670 INFO:teuthology.orchestra.run.smithi161.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-19T00:42:05.708 INFO:teuthology.orchestra.run.smithi161.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T00:42:05.832 INFO:teuthology.orchestra.run.smithi161.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T00:42:05.839 INFO:teuthology.orchestra.run.smithi161.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-19T00:42:05.840 INFO:teuthology.orchestra.run.smithi161.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T00:42:05.846 INFO:teuthology.orchestra.run.smithi161.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T00:42:05.854 INFO:teuthology.orchestra.run.smithi161.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-19T00:42:05.861 INFO:teuthology.orchestra.run.smithi161.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-19T00:42:05.866 INFO:teuthology.orchestra.run.smithi161.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-19T00:42:05.888 INFO:teuthology.orchestra.run.smithi161.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-19T00:42:05.892 INFO:teuthology.orchestra.run.smithi161.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-19T00:42:05.912 INFO:teuthology.orchestra.run.smithi161.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T00:42:06.058 INFO:teuthology.orchestra.run.smithi161.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-19T00:42:06.184 INFO:teuthology.orchestra.run.smithi161.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T00:42:06.374 INFO:teuthology.orchestra.run.smithi161.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-19T00:42:06.417 INFO:teuthology.orchestra.run.smithi161.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-19T00:42:06.426 INFO:teuthology.orchestra.run.smithi161.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-19T00:42:06.843 INFO:teuthology.orchestra.run.smithi161.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-19T00:42:06.858 INFO:teuthology.orchestra.run.smithi161.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-19T00:42:06.887 INFO:teuthology.orchestra.run.smithi161.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-19T00:42:06.898 INFO:teuthology.orchestra.run.smithi161.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-19T00:42:06.928 INFO:teuthology.orchestra.run.smithi161.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-19T00:42:06.977 INFO:teuthology.orchestra.run.smithi161.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-19T00:42:06.994 INFO:teuthology.orchestra.run.smithi161.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-19T00:42:07.191 INFO:teuthology.orchestra.run.smithi161.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-19T00:42:07.254 INFO:teuthology.orchestra.run.smithi161.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T00:42:07.254 INFO:teuthology.orchestra.run.smithi161.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T00:42:07.271 INFO:teuthology.orchestra.run.smithi161.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T00:42:07.335 INFO:teuthology.orchestra.run.smithi161.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T00:42:07.370 INFO:teuthology.orchestra.run.smithi161.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T00:42:07.376 INFO:teuthology.orchestra.run.smithi161.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T00:42:07.454 INFO:teuthology.orchestra.run.smithi161.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-19T00:42:07.485 INFO:teuthology.orchestra.run.smithi161.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T00:42:07.633 INFO:teuthology.orchestra.run.smithi161.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-19T00:42:07.647 INFO:teuthology.orchestra.run.smithi161.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-19T00:42:07.666 INFO:teuthology.orchestra.run.smithi161.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-19T00:42:07.731 INFO:teuthology.orchestra.run.smithi161.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-19T00:42:07.883 INFO:teuthology.orchestra.run.smithi161.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T00:42:07.884 INFO:teuthology.orchestra.run.smithi161.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T00:42:08.020 INFO:teuthology.orchestra.run.smithi161.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-19T00:42:08.022 INFO:teuthology.orchestra.run.smithi161.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-19T00:42:08.025 INFO:teuthology.orchestra.run.smithi161.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-19T00:42:08.025 INFO:teuthology.orchestra.run.smithi161.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-19T00:42:08.030 INFO:teuthology.orchestra.run.smithi161.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-19T00:42:08.250 INFO:teuthology.orchestra.run.smithi161.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-19T00:42:08.688 INFO:teuthology.orchestra.run.smithi161.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-19T00:42:08.694 INFO:teuthology.orchestra.run.smithi161.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-19T00:42:08.742 INFO:teuthology.orchestra.run.smithi043.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3154-g8b4ab94f-1jammy [5,095 kB] 2024-04-19T00:42:08.985 INFO:teuthology.orchestra.run.smithi161.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-19T00:42:08.985 INFO:teuthology.orchestra.run.smithi161.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-19T00:42:08.987 INFO:teuthology.orchestra.run.smithi161.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-19T00:42:08.989 INFO:teuthology.orchestra.run.smithi161.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-19T00:42:08.990 INFO:teuthology.orchestra.run.smithi161.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-19T00:42:08.992 INFO:teuthology.orchestra.run.smithi161.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-19T00:42:08.993 INFO:teuthology.orchestra.run.smithi161.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-19T00:42:08.995 INFO:teuthology.orchestra.run.smithi161.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-19T00:42:09.155 INFO:teuthology.orchestra.run.smithi043.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3154-g8b4ab94f-1jammy [244 kB] 2024-04-19T00:42:09.177 INFO:teuthology.orchestra.run.smithi161.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-19T00:42:09.179 INFO:teuthology.orchestra.run.smithi043.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T00:42:09.189 INFO:teuthology.orchestra.run.smithi043.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3154-g8b4ab94f-1jammy [1,526 kB] 2024-04-19T00:42:09.320 INFO:teuthology.orchestra.run.smithi043.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3154-g8b4ab94f-1jammy [6,260 kB] 2024-04-19T00:42:09.584 INFO:teuthology.orchestra.run.smithi161.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-19T00:42:09.631 INFO:teuthology.orchestra.run.smithi161.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-19T00:42:09.666 INFO:teuthology.orchestra.run.smithi161.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-19T00:42:09.763 INFO:teuthology.orchestra.run.smithi161.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-19T00:42:09.804 INFO:teuthology.orchestra.run.smithi161.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-19T00:42:09.815 INFO:teuthology.orchestra.run.smithi161.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-19T00:42:09.820 INFO:teuthology.orchestra.run.smithi161.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-19T00:42:09.868 INFO:teuthology.orchestra.run.smithi161.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-19T00:42:09.872 INFO:teuthology.orchestra.run.smithi161.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-19T00:42:09.892 INFO:teuthology.orchestra.run.smithi161.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-19T00:42:09.894 INFO:teuthology.orchestra.run.smithi043.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3154-g8b4ab94f-1jammy [22.8 MB] 2024-04-19T00:42:10.056 INFO:teuthology.orchestra.run.smithi161.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-19T00:42:10.330 INFO:teuthology.orchestra.run.smithi161.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-19T00:42:10.347 INFO:teuthology.orchestra.run.smithi161.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-19T00:42:10.518 INFO:teuthology.orchestra.run.smithi161.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-19T00:42:10.584 INFO:teuthology.orchestra.run.smithi161.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-19T00:42:10.668 INFO:teuthology.orchestra.run.smithi161.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T00:42:10.671 INFO:teuthology.orchestra.run.smithi161.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T00:42:10.672 INFO:teuthology.orchestra.run.smithi161.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T00:42:10.704 INFO:teuthology.orchestra.run.smithi161.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-19T00:42:10.707 INFO:teuthology.orchestra.run.smithi161.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-19T00:42:10.708 INFO:teuthology.orchestra.run.smithi161.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-19T00:42:10.728 INFO:teuthology.orchestra.run.smithi161.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-19T00:42:10.732 INFO:teuthology.orchestra.run.smithi161.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-19T00:42:11.203 INFO:teuthology.orchestra.run.smithi161.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T00:42:11.605 INFO:teuthology.orchestra.run.smithi043.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3154-g8b4ab94f-1jammy [14.1 kB] 2024-04-19T00:42:11.606 INFO:teuthology.orchestra.run.smithi043.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [870 kB] 2024-04-19T00:42:11.623 INFO:teuthology.orchestra.run.smithi043.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3154-g8b4ab94f-1jammy [2,437 kB] 2024-04-19T00:42:11.689 INFO:teuthology.orchestra.run.smithi043.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3154-g8b4ab94f-1jammy [791 kB] 2024-04-19T00:42:11.722 INFO:teuthology.orchestra.run.smithi043.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3154-g8b4ab94f-1jammy [148 kB] 2024-04-19T00:42:11.726 INFO:teuthology.orchestra.run.smithi043.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3154-g8b4ab94f-1jammy [2,297 kB] 2024-04-19T00:42:11.825 INFO:teuthology.orchestra.run.smithi043.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3154-g8b4ab94f-1jammy [8,625 kB] 2024-04-19T00:42:11.835 INFO:teuthology.orchestra.run.smithi161.stdout:Fetched 181 MB in 8s (23.2 MB/s) 2024-04-19T00:42:11.993 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T00:42:12.093 INFO:teuthology.orchestra.run.smithi161.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T00:42:12.098 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T00:42:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3154-g8b4ab94f-1jammy [14.2 kB] 2024-04-19T00:42:12.160 INFO:teuthology.orchestra.run.smithi043.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3154-g8b4ab94f-1jammy [51.6 MB] 2024-04-19T00:42:12.191 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T00:42:12.495 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T00:42:12.521 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T00:42:12.561 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T00:42:12.873 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T00:42:12.899 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T00:42:12.939 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T00:42:13.376 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T00:42:13.403 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T00:42:13.451 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:42:13.829 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T00:42:13.855 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T00:42:13.887 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:42:14.232 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T00:42:14.258 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T00:42:14.298 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:42:14.455 INFO:teuthology.orchestra.run.smithi043.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3154-g8b4ab94f-1jammy [124 kB] 2024-04-19T00:42:14.457 INFO:teuthology.orchestra.run.smithi043.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3154-g8b4ab94f-1jammy [38.9 kB] 2024-04-19T00:42:14.458 INFO:teuthology.orchestra.run.smithi043.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3154-g8b4ab94f-1jammy [13.6 MB] 2024-04-19T00:42:14.651 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T00:42:14.678 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T00:42:14.718 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T00:42:14.955 INFO:teuthology.orchestra.run.smithi043.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri5-testing-2024-04-17-1400/8b4ab94f9c30da63487fa2f241ce12f483378860/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3154-g8b4ab94f-1jammy [91.4 kB] 2024-04-19T00:42:15.012 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package librados2. 2024-04-19T00:42:15.038 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:15.070 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:15.546 INFO:teuthology.orchestra.run.smithi043.stdout:Fetched 181 MB in 11s (15.8 MB/s) 2024-04-19T00:42:15.665 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package librbd1. 2024-04-19T00:42:15.680 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-19T00:42:15.691 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:15.731 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:15.781 INFO:teuthology.orchestra.run.smithi043.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-19T00:42:15.787 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-19T00:42:15.854 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T00:42:16.124 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-19T00:42:16.150 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-19T00:42:16.167 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T00:42:16.182 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T00:42:16.194 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:16.234 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:16.518 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-19T00:42:16.519 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-rados. 2024-04-19T00:42:16.535 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:16.545 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-19T00:42:16.578 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:16.585 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T00:42:16.855 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T00:42:16.870 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:16.905 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:17.022 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-19T00:42:17.049 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T00:42:17.097 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:42:17.108 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T00:42:17.123 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:17.183 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:17.492 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-19T00:42:17.502 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T00:42:17.519 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T00:42:17.529 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:17.558 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:42:17.569 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:17.896 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T00:42:17.923 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T00:42:17.928 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-19T00:42:17.955 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-19T00:42:17.966 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T00:42:17.995 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:42:18.241 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T00:42:18.264 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T00:42:18.299 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T00:42:18.306 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-19T00:42:18.333 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-19T00:42:18.374 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T00:42:18.543 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T00:42:18.570 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:18.609 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:18.659 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package librados2. 2024-04-19T00:42:18.686 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../007-librados2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:18.725 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:18.954 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T00:42:18.981 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T00:42:19.020 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T00:42:19.178 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package librbd1. 2024-04-19T00:42:19.205 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../008-librbd1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:19.245 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:19.298 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T00:42:19.314 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T00:42:19.356 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T00:42:19.642 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T00:42:19.669 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T00:42:19.698 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libcephfs2. 2024-04-19T00:42:19.700 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T00:42:19.724 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:19.764 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:20.012 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package lua5.1. 2024-04-19T00:42:20.025 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-rados. 2024-04-19T00:42:20.038 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T00:42:20.041 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:20.075 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:20.078 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T00:42:20.378 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-19T00:42:20.405 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:20.406 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package lua-any. 2024-04-19T00:42:20.422 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T00:42:20.444 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:20.465 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T00:42:20.697 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-cephfs. 2024-04-19T00:42:20.709 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package zip. 2024-04-19T00:42:20.713 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:20.724 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T00:42:20.755 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:20.775 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T00:42:21.037 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package unzip. 2024-04-19T00:42:21.063 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T00:42:21.066 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-19T00:42:21.093 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:21.095 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T00:42:21.141 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:21.436 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-19T00:42:21.463 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-19T00:42:21.474 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package luarocks. 2024-04-19T00:42:21.494 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T00:42:21.500 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T00:42:21.540 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T00:42:21.814 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-prettytable. 2024-04-19T00:42:21.841 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-19T00:42:21.880 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-19T00:42:21.914 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package librgw2. 2024-04-19T00:42:21.941 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:21.989 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:22.133 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-rbd. 2024-04-19T00:42:22.149 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:22.200 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:22.525 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T00:42:22.528 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-19T00:42:22.541 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:22.555 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-19T00:42:22.575 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:22.594 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T00:42:22.839 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-19T00:42:22.865 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-19T00:42:22.869 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T00:42:22.896 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T00:42:22.896 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T00:42:22.936 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T00:42:23.175 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-19T00:42:23.190 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-19T00:42:23.241 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T00:42:23.247 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T00:42:23.274 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:23.314 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:23.578 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package lua5.1. 2024-04-19T00:42:23.605 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-19T00:42:23.644 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-19T00:42:23.750 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-common. 2024-04-19T00:42:23.777 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:23.816 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:23.947 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package lua-any. 2024-04-19T00:42:23.963 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-19T00:42:23.997 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-19T00:42:24.225 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package zip. 2024-04-19T00:42:24.252 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-19T00:42:24.292 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking zip (3.0-12build2) ... 2024-04-19T00:42:24.553 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package unzip. 2024-04-19T00:42:24.569 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-19T00:42:24.603 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-19T00:42:24.990 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package luarocks. 2024-04-19T00:42:25.160 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-19T00:42:25.160 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-19T00:42:25.161 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-base. 2024-04-19T00:42:25.161 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:25.161 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:25.466 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package librgw2. 2024-04-19T00:42:25.493 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../025-librgw2_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:25.531 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:25.884 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T00:42:25.911 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T00:42:25.951 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T00:42:26.067 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-rgw. 2024-04-19T00:42:26.084 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:26.117 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:26.253 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T00:42:26.280 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T00:42:26.320 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T00:42:26.429 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-19T00:42:26.455 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-19T00:42:26.495 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T00:42:26.622 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T00:42:26.649 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T00:42:26.697 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T00:42:26.798 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libradosstriper1. 2024-04-19T00:42:26.825 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:26.864 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:26.975 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T00:42:27.002 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T00:42:27.042 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T00:42:27.260 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-common. 2024-04-19T00:42:27.287 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:27.311 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T00:42:27.318 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:27.338 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T00:42:27.378 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T00:42:27.680 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T00:42:27.707 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T00:42:27.747 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T00:42:28.083 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-portend. 2024-04-19T00:42:28.110 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T00:42:28.150 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T00:42:28.411 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T00:42:28.426 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T00:42:28.460 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T00:42:28.665 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-base. 2024-04-19T00:42:28.692 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:28.741 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:28.747 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T00:42:28.773 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T00:42:28.813 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T00:42:29.142 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T00:42:29.168 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T00:42:29.210 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T00:42:29.319 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-19T00:42:29.347 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-19T00:42:29.386 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-19T00:42:29.469 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T00:42:29.485 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T00:42:29.519 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T00:42:29.613 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-cheroot. 2024-04-19T00:42:29.629 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-19T00:42:29.672 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T00:42:29.822 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T00:42:29.848 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T00:42:29.888 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T00:42:29.966 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-19T00:42:29.994 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-19T00:42:30.033 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-19T00:42:30.191 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-mako. 2024-04-19T00:42:30.218 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T00:42:30.258 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T00:42:30.336 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-19T00:42:30.363 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-19T00:42:30.403 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-19T00:42:30.586 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T00:42:30.612 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T00:42:30.652 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T00:42:30.680 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-19T00:42:30.697 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-19T00:42:30.730 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-19T00:42:30.913 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T00:42:30.940 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T00:42:30.958 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-tempora. 2024-04-19T00:42:30.972 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T00:42:30.985 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-19T00:42:31.025 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-19T00:42:31.258 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-webob. 2024-04-19T00:42:31.285 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T00:42:31.311 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-portend. 2024-04-19T00:42:31.324 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T00:42:31.338 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-19T00:42:31.378 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-19T00:42:31.689 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-19T00:42:31.694 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T00:42:31.716 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-19T00:42:31.720 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T00:42:31.755 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-19T00:42:31.789 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T00:42:32.017 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-19T00:42:32.032 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-19T00:42:32.066 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-19T00:42:32.097 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T00:42:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T00:42:32.163 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T00:42:32.412 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-natsort. 2024-04-19T00:42:32.439 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-19T00:42:32.458 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-paste. 2024-04-19T00:42:32.478 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-19T00:42:32.486 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T00:42:32.524 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T00:42:32.765 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-logutils. 2024-04-19T00:42:32.792 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-19T00:42:32.831 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-19T00:42:32.835 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T00:42:32.851 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T00:42:32.893 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T00:42:33.126 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-19T00:42:33.154 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-19T00:42:33.193 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-19T00:42:33.196 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T00:42:33.222 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T00:42:33.263 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T00:42:33.496 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-mako. 2024-04-19T00:42:33.523 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-19T00:42:33.532 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T00:42:33.559 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T00:42:33.562 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T00:42:33.591 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T00:42:33.876 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T00:42:33.890 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-19T00:42:33.892 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T00:42:33.918 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-19T00:42:33.926 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T00:42:33.957 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-19T00:42:34.219 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-19T00:42:34.246 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-19T00:42:34.285 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-19T00:42:34.289 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T00:42:34.316 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T00:42:34.356 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T00:42:34.529 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-webob. 2024-04-19T00:42:34.545 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-19T00:42:34.587 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-19T00:42:34.636 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-idna. 2024-04-19T00:42:34.651 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T00:42:34.686 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T00:42:34.932 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-waitress. 2024-04-19T00:42:34.960 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-19T00:42:34.963 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T00:42:34.979 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T00:42:35.002 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T00:42:35.013 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T00:42:35.310 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-tempita. 2024-04-19T00:42:35.316 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-requests. 2024-04-19T00:42:35.338 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-19T00:42:35.343 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T00:42:35.375 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T00:42:35.376 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T00:42:35.663 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-paste. 2024-04-19T00:42:35.678 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T00:42:35.690 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-19T00:42:35.704 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T00:42:35.729 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T00:42:35.744 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T00:42:36.033 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-19T00:42:36.049 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-19T00:42:36.091 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T00:42:36.181 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T00:42:36.196 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:36.239 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:36.411 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-19T00:42:36.437 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-19T00:42:36.477 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-19T00:42:36.628 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T00:42:36.654 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:36.694 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:37.688 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-webtest. 2024-04-19T00:42:37.706 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T00:42:37.716 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-19T00:42:37.732 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T00:42:37.772 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T00:42:37.830 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-19T00:42:38.058 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T00:42:38.085 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T00:42:38.100 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pecan. 2024-04-19T00:42:38.125 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T00:42:38.127 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-19T00:42:38.166 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T00:42:38.469 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T00:42:38.496 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:38.513 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-certifi. 2024-04-19T00:42:38.536 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:38.540 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-19T00:42:38.580 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-19T00:42:38.899 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-idna. 2024-04-19T00:42:38.914 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T00:42:38.926 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-19T00:42:38.941 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:38.966 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-19T00:42:38.972 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:39.286 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-urllib3. 2024-04-19T00:42:39.313 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-19T00:42:39.352 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T00:42:39.534 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T00:42:39.560 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:39.609 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:39.655 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-requests. 2024-04-19T00:42:39.683 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-19T00:42:39.722 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T00:42:39.991 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-19T00:42:40.018 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-19T00:42:40.049 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T00:42:40.328 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-19T00:42:40.344 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:40.378 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:40.645 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph. 2024-04-19T00:42:40.672 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:40.712 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:40.774 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-19T00:42:40.801 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:40.841 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:41.031 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T00:42:41.058 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:41.097 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:41.145 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-19T00:42:41.161 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-19T00:42:41.195 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-19T00:42:41.426 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T00:42:41.453 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:41.465 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-openssl. 2024-04-19T00:42:41.492 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-19T00:42:41.492 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:41.531 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-19T00:42:41.868 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mgr. 2024-04-19T00:42:41.878 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package cephadm. 2024-04-19T00:42:41.895 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:41.906 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:41.934 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:41.945 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:42.281 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T00:42:42.308 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T00:42:42.347 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T00:42:42.371 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mon. 2024-04-19T00:42:42.398 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:42.437 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:42.667 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T00:42:42.682 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T00:42:42.725 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T00:42:42.941 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-osd. 2024-04-19T00:42:42.968 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:43.007 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:43.697 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-babel. 2024-04-19T00:42:43.724 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T00:42:43.755 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T00:42:44.024 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T00:42:44.040 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T00:42:44.044 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph. 2024-04-19T00:42:44.060 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../066-ceph_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:44.074 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T00:42:44.102 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:44.386 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T00:42:44.413 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:44.421 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-fuse. 2024-04-19T00:42:44.448 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:44.452 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:44.488 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:44.714 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T00:42:44.740 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T00:42:44.772 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T00:42:44.791 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mds. 2024-04-19T00:42:44.807 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:44.874 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:45.083 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-routes. 2024-04-19T00:42:45.110 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T00:42:45.149 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T00:42:45.261 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package cephadm. 2024-04-19T00:42:45.288 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../069-cephadm_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:45.327 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:45.486 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T00:42:45.513 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:45.553 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:45.647 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-19T00:42:45.674 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-19T00:42:45.713 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-19T00:42:46.024 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-19T00:42:46.051 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T00:42:46.091 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T00:42:47.049 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T00:42:47.065 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T00:42:47.107 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T00:42:47.123 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-babel. 2024-04-19T00:42:47.149 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-19T00:42:47.190 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T00:42:47.507 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-jinja2. 2024-04-19T00:42:47.534 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-19T00:42:47.573 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T00:42:47.586 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T00:42:47.613 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T00:42:47.653 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T00:42:47.952 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-19T00:42:47.979 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:48.018 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:48.031 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T00:42:48.058 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T00:42:48.097 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T00:42:48.347 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-19T00:42:48.367 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T00:42:48.374 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-19T00:42:48.394 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T00:42:48.413 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-19T00:42:48.434 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T00:42:48.741 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-routes. 2024-04-19T00:42:48.768 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-19T00:42:48.807 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T00:42:49.125 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T00:42:49.152 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-19T00:42:49.153 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:49.180 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:49.219 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:49.278 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:50.249 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T00:42:50.275 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T00:42:50.314 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T00:42:50.416 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-19T00:42:50.432 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-19T00:42:50.466 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T00:42:50.617 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T00:42:50.644 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T00:42:50.684 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T00:42:50.961 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-joblib. 2024-04-19T00:42:50.988 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-19T00:42:51.021 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T00:42:51.027 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T00:42:51.051 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T00:42:51.087 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T00:42:51.339 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-19T00:42:51.367 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-19T00:42:51.398 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-19T00:42:51.406 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T00:42:51.434 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T00:42:51.474 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T00:42:51.701 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-sklearn. 2024-04-19T00:42:51.728 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-19T00:42:51.752 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T00:42:51.767 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T00:42:51.780 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T00:42:51.810 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T00:42:52.097 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T00:42:52.124 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T00:42:52.163 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T00:42:52.474 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T00:42:52.502 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T00:42:52.541 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T00:42:52.567 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-19T00:42:52.596 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:52.719 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:52.853 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T00:42:52.880 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T00:42:52.919 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T00:42:53.633 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T00:42:53.649 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:53.691 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:53.815 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-cachetools. 2024-04-19T00:42:53.842 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-19T00:42:53.881 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-19T00:42:54.002 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T00:42:54.030 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T00:42:54.069 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T00:42:54.184 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-19T00:42:54.212 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-19T00:42:54.250 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-19T00:42:54.372 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T00:42:54.399 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T00:42:54.430 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T00:42:54.579 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-19T00:42:54.607 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-19T00:42:54.646 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T00:42:54.649 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package jq. 2024-04-19T00:42:54.676 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T00:42:54.716 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T00:42:54.949 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-rsa. 2024-04-19T00:42:54.976 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-19T00:42:54.977 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package socat. 2024-04-19T00:42:55.004 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T00:42:55.015 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-19T00:42:55.043 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T00:42:55.319 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-google-auth. 2024-04-19T00:42:55.346 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-19T00:42:55.355 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T00:42:55.371 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T00:42:55.385 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-19T00:42:55.413 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T00:42:55.705 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-19T00:42:55.733 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-19T00:42:55.742 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-test. 2024-04-19T00:42:55.758 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:55.772 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T00:42:55.792 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:56.050 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-websocket. 2024-04-19T00:42:56.077 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-19T00:42:56.108 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-19T00:42:56.420 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-19T00:42:56.521 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-19T00:42:56.522 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T00:42:57.150 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-19T00:42:57.167 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:57.225 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:57.478 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-19T00:42:57.505 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-19T00:42:57.535 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T00:42:57.881 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-19T00:42:57.909 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T00:42:57.947 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T00:42:58.055 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T00:42:58.082 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:42:58.121 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:58.317 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package jq. 2024-04-19T00:42:58.344 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-19T00:42:58.383 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-19T00:42:58.550 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T00:42:58.578 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:58.617 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:58.627 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package socat. 2024-04-19T00:42:58.656 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-19T00:42:58.827 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-19T00:42:58.903 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T00:42:58.931 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T00:42:58.961 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T00:42:59.097 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package xmlstarlet. 2024-04-19T00:42:59.114 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-19T00:42:59.147 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-19T00:42:59.241 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T00:42:59.268 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T00:42:59.307 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T00:42:59.476 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-test. 2024-04-19T00:42:59.493 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:42:59.526 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:42:59.695 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T00:42:59.722 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T00:42:59.761 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T00:43:00.192 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package pkg-config. 2024-04-19T00:43:00.219 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T00:43:00.258 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T00:43:00.511 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T00:43:00.539 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T00:43:00.577 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T00:43:00.997 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T00:43:01.025 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T00:43:01.064 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T00:43:01.367 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T00:43:01.394 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T00:43:01.433 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T00:43:01.704 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T00:43:01.731 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T00:43:01.762 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T00:43:01.890 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package ceph-volume. 2024-04-19T00:43:01.917 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3154-g8b4ab94f-1jammy_all.deb ... 2024-04-19T00:43:01.956 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:02.073 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T00:43:02.100 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T00:43:02.139 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T00:43:02.259 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-19T00:43:02.276 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:43:02.318 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:02.484 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-py. 2024-04-19T00:43:02.511 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T00:43:02.550 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T00:43:02.629 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-19T00:43:02.657 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-19T00:43:02.704 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T00:43:02.862 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T00:43:02.890 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T00:43:02.928 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T00:43:03.025 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-19T00:43:03.053 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-19T00:43:03.091 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-19T00:43:03.387 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package nvme-cli. 2024-04-19T00:43:03.407 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T00:43:03.415 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-19T00:43:03.435 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T00:43:03.453 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T00:43:03.474 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T00:43:03.785 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-toml. 2024-04-19T00:43:03.812 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T00:43:03.860 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T00:43:03.884 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package pkg-config. 2024-04-19T00:43:03.912 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-19T00:43:03.950 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T00:43:04.163 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T00:43:04.190 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T00:43:04.228 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-19T00:43:04.229 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T00:43:04.256 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-19T00:43:04.295 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-19T00:43:04.583 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T00:43:04.610 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T00:43:04.649 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T00:43:04.715 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-19T00:43:04.743 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-19T00:43:04.781 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-19T00:43:04.918 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T00:43:04.934 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T00:43:04.977 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T00:43:05.084 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pastescript. 2024-04-19T00:43:05.112 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-19T00:43:05.150 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-19T00:43:05.446 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pluggy. 2024-04-19T00:43:05.473 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-19T00:43:05.503 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-19T00:43:05.714 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package radosgw. 2024-04-19T00:43:05.742 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:43:05.780 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:05.815 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-psutil. 2024-04-19T00:43:05.843 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-19T00:43:05.881 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-19T00:43:06.201 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-py. 2024-04-19T00:43:06.229 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-19T00:43:06.267 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-19T00:43:06.579 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pygments. 2024-04-19T00:43:06.583 INFO:teuthology.orchestra.run.smithi161.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T00:43:06.607 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-19T00:43:06.611 INFO:teuthology.orchestra.run.smithi161.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:43:06.650 INFO:teuthology.orchestra.run.smithi161.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:06.654 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T00:43:06.954 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T00:43:07.125 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-19T00:43:07.153 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-19T00:43:07.183 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-19T00:43:07.222 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T00:43:07.364 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T00:43:07.477 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-toml. 2024-04-19T00:43:07.506 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-19T00:43:07.544 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-19T00:43:07.646 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T00:43:07.847 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-pytest. 2024-04-19T00:43:07.875 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-19T00:43:07.930 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T00:43:07.972 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T00:43:08.275 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package python3-simplejson. 2024-04-19T00:43:08.303 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-19T00:43:08.342 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-19T00:43:08.421 INFO:teuthology.orchestra.run.smithi161.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T00:43:08.483 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:08.611 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-19T00:43:08.627 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-19T00:43:08.669 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-19T00:43:08.974 INFO:teuthology.orchestra.run.smithi161.stdout:Adding system user cephadm....done 2024-04-19T00:43:09.130 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T00:43:09.365 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package radosgw. 2024-04-19T00:43:09.394 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../116-radosgw_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:43:09.398 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T00:43:09.431 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:09.658 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T00:43:09.784 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T00:43:10.044 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T00:43:10.255 INFO:teuthology.orchestra.run.smithi043.stdout:Selecting previously unselected package rbd-fuse. 2024-04-19T00:43:10.271 INFO:teuthology.orchestra.run.smithi043.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3154-g8b4ab94f-1jammy_amd64.deb ... 2024-04-19T00:43:10.295 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T00:43:10.309 INFO:teuthology.orchestra.run.smithi043.stdout:Unpacking rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:10.421 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T00:43:10.631 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-19T00:43:10.747 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T00:43:10.898 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-19T00:43:10.999 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-19T00:43:11.100 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T00:43:11.259 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-19T00:43:11.334 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T00:43:11.572 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-19T00:43:11.610 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T00:43:11.879 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T00:43:11.981 INFO:teuthology.orchestra.run.smithi043.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-19T00:43:12.051 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:12.138 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T00:43:12.389 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T00:43:12.578 INFO:teuthology.orchestra.run.smithi043.stdout:Adding system user cephadm....done 2024-04-19T00:43:12.640 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:12.698 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-19T00:43:12.966 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T00:43:12.967 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-19T00:43:13.092 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T00:43:13.218 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-19T00:43:13.327 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-19T00:43:13.343 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T00:43:13.595 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-19T00:43:13.677 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T00:43:13.855 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-19T00:43:13.937 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T00:43:13.980 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-19T00:43:14.316 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-19T00:43:14.322 INFO:teuthology.orchestra.run.smithi161.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T00:43:14.352 INFO:teuthology.orchestra.run.smithi161.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T00:43:14.398 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T00:43:14.533 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T00:43:14.676 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-19T00:43:14.884 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T00:43:14.919 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-19T00:43:15.002 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T00:43:15.186 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-19T00:43:15.261 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T00:43:15.430 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-19T00:43:15.520 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T00:43:15.689 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-19T00:43:15.830 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T00:43:15.949 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-19T00:43:15.956 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T00:43:16.208 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-ceph-argparse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:16.476 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-19T00:43:16.602 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-19T00:43:16.633 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T00:43:16.719 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-19T00:43:16.820 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-19T00:43:16.875 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T00:43:16.985 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T00:43:17.063 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-19T00:43:17.111 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T00:43:17.400 INFO:teuthology.orchestra.run.smithi043.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-19T00:43:17.420 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T00:43:17.426 INFO:teuthology.orchestra.run.smithi043.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-19T00:43:17.492 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-19T00:43:17.617 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-19T00:43:17.805 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T00:43:17.977 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-19T00:43:18.047 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T00:43:18.103 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-19T00:43:18.371 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-19T00:43:18.441 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T00:43:18.647 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-19T00:43:18.725 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T00:43:18.965 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up zip (3.0-12build2) ... 2024-04-19T00:43:19.062 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T00:43:19.091 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-19T00:43:19.305 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T00:43:19.431 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:19.724 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T00:43:19.759 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-19T00:43:20.019 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-19T00:43:20.033 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T00:43:20.120 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-19T00:43:20.229 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-19T00:43:20.530 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-19T00:43:20.931 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-19T00:43:21.139 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T00:43:21.191 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-19T00:43:21.374 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:43:21.483 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T00:43:21.584 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-19T00:43:21.734 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T00:43:21.843 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T00:43:21.877 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-19T00:43:21.986 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T00:43:22.220 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-19T00:43:22.279 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T00:43:22.472 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-19T00:43:22.522 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T00:43:22.597 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:22.765 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:43:22.880 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T00:43:22.899 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-idna (3.3-1) ... 2024-04-19T00:43:23.167 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T00:43:23.259 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-19T00:43:23.418 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T00:43:23.677 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T00:43:24.154 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T00:43:24.404 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T00:43:24.466 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-19T00:43:24.656 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T00:43:24.717 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:43:24.781 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T00:43:24.851 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-19T00:43:25.066 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T00:43:25.128 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-19T00:43:25.167 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T00:43:25.229 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-19T00:43:25.338 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-19T00:43:25.460 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T00:43:25.648 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-19T00:43:25.761 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T00:43:25.907 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-19T00:43:26.020 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T00:43:26.146 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T00:43:26.150 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:43:26.272 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T00:43:26.276 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-19T00:43:26.361 INFO:teuthology.orchestra.run.smithi161.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T00:43:26.602 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-19T00:43:26.615 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T00:43:26.716 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T00:43:26.853 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-19T00:43:27.093 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T00:43:27.129 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-19T00:43:27.352 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T00:43:27.389 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-19T00:43:27.487 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T00:43:27.648 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-19T00:43:27.738 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:43:27.864 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T00:43:27.899 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-19T00:43:28.025 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-19T00:43:28.173 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T00:43:28.310 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-19T00:43:28.436 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-19T00:43:28.458 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T00:43:28.717 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T00:43:28.737 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-19T00:43:28.868 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T00:43:29.030 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-19T00:43:29.228 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T00:43:29.306 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-19T00:43:29.440 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-19T00:43:29.496 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T00:43:29.566 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-19T00:43:29.739 INFO:teuthology.orchestra.run.smithi043.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-19T00:43:29.815 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T00:43:30.160 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-19T00:43:30.201 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T00:43:30.285 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-19T00:43:30.469 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T00:43:30.670 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-19T00:43:30.796 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T00:43:30.897 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:30.972 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-19T00:43:31.007 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:31.097 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-19T00:43:31.133 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T00:43:31.357 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-19T00:43:31.469 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:31.483 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-19T00:43:31.595 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:31.713 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T00:43:31.834 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-19T00:43:31.973 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:32.099 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:32.152 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-19T00:43:32.225 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:32.420 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-19T00:43:32.437 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T00:43:32.437 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T00:43:32.520 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-19T00:43:32.788 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-19T00:43:32.934 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:33.061 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:33.064 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-19T00:43:33.187 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:33.298 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T00:43:33.418 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-19T00:43:33.794 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-19T00:43:34.062 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-19T00:43:34.398 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-19T00:43:34.450 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:34.524 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up librados2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:34.575 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:34.650 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:34.718 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:34.785 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-19T00:43:34.844 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:34.979 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:35.096 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libcephfs2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:35.222 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libradosstriper1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:35.347 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-19T00:43:35.455 INFO:teuthology.orchestra.run.smithi161.stdout:Adding group ceph....done 2024-04-19T00:43:35.633 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up librbd1 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:35.759 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mgr-modules-core (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:35.806 INFO:teuthology.orchestra.run.smithi161.stdout:Adding system user ceph....done 2024-04-19T00:43:35.884 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:35.976 INFO:teuthology.orchestra.run.smithi161.stdout:Setting system user ceph properties....done 2024-04-19T00:43:36.012 INFO:teuthology.orchestra.run.smithi161.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T00:43:36.103 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T00:43:36.103 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-19T00:43:36.147 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T00:43:36.467 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T00:43:36.586 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up libcephfs-dev (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:36.712 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-rados (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:36.813 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up librgw2 (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:36.922 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:36.938 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-19T00:43:37.063 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:37.553 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T00:43:37.553 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T00:43:37.950 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:38.201 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-rbd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:38.295 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T00:43:38.326 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up rbd-fuse (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:38.444 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-rgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:38.553 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up python3-cephfs (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:38.688 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-common (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:38.787 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:39.015 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T00:43:39.015 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T00:43:39.231 INFO:teuthology.orchestra.run.smithi043.stdout:Adding group ceph....done 2024-04-19T00:43:39.495 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:39.532 INFO:teuthology.orchestra.run.smithi043.stdout:Adding system user ceph....done 2024-04-19T00:43:39.701 INFO:teuthology.orchestra.run.smithi043.stdout:Setting system user ceph properties....done 2024-04-19T00:43:39.706 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T00:43:39.707 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T00:43:39.714 INFO:teuthology.orchestra.run.smithi043.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-19T00:43:39.859 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-19T00:43:40.157 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-19T00:43:40.188 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:40.464 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T00:43:40.464 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T00:43:40.631 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-test (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:40.757 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up radosgw (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:40.907 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:41.032 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:41.176 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:41.239 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T00:43:41.239 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-19T00:43:41.392 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T00:43:41.392 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T00:43:41.683 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-base (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:41.885 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:42.019 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:42.052 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-19T00:43:42.120 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:42.247 INFO:teuthology.orchestra.run.smithi161.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:42.543 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mds (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:42.574 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T00:43:42.692 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T00:43:42.788 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T00:43:42.788 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-19T00:43:42.844 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T00:43:43.303 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mgr (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:43.526 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T00:43:43.526 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-19T00:43:43.903 INFO:teuthology.orchestra.run.smithi161.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T00:43:43.988 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-osd (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:44.265 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T00:43:44.265 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-19T00:43:44.534 INFO:teuthology.orchestra.run.smithi161.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T00:43:44.540 DEBUG:teuthology.parallel:result is None 2024-04-19T00:43:44.768 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:44.886 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:45.037 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mon (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:45.254 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T00:43:45.254 INFO:teuthology.orchestra.run.smithi043.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-19T00:43:45.747 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mgr-cephadm (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:45.998 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:46.198 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-mgr-dashboard (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:46.665 INFO:teuthology.orchestra.run.smithi043.stdout:Setting up ceph-volume (19.0.0-3154-g8b4ab94f-1jammy) ... 2024-04-19T00:43:47.103 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-19T00:43:47.204 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-19T00:43:47.357 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-19T00:43:48.123 INFO:teuthology.orchestra.run.smithi043.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-19T00:43:48.780 INFO:teuthology.orchestra.run.smithi043.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-19T00:43:48.785 DEBUG:teuthology.parallel:result is None 2024-04-19T00:43:48.786 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:43:48.952 DEBUG:teuthology.orchestra.run.smithi043:> dpkg-query -W -f '${Version}' ceph 2024-04-19T00:43:48.976 INFO:teuthology.orchestra.run.smithi043.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:43:48.977 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:43:48.977 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T00:43:48.979 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:43:49.209 DEBUG:teuthology.orchestra.run.smithi161:> dpkg-query -W -f '${Version}' ceph 2024-04-19T00:43:49.229 INFO:teuthology.orchestra.run.smithi161.stdout:19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:43:49.230 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3154-g8b4ab94f-1jammy 2024-04-19T00:43:49.230 INFO:teuthology.task.install:The correct ceph version 19.0.0-3154-g8b4ab94f-1jammy is installed. 2024-04-19T00:43:49.231 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-19T00:43:49.232 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:49.232 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T00:43:49.251 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:49.251 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-19T00:43:49.288 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-19T00:43:49.288 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:49.288 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T00:43:49.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T00:43:49.363 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:49.363 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-04-19T00:43:49.376 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-19T00:43:49.430 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-19T00:43:49.431 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:49.431 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T00:43:49.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T00:43:49.501 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:49.501 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-19T00:43:49.515 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-19T00:43:49.572 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-19T00:43:49.572 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:49.572 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T00:43:49.586 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T00:43:49.641 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:49.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-04-19T00:43:49.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-19T00:43:49.710 INFO:teuthology.run_tasks:Running task ceph... 2024-04-19T00:43:49.801 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-19T00:43:49.801 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 777 /var/log/ceph 2024-04-19T00:43:49.803 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2024-04-19T00:43:49.816 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-19T00:43:49.816 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T00:43:49.858 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-19T00:43:49.870 INFO:tasks.ceph:Creating extra log directories... 2024-04-19T00:43:49.870 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T00:43:49.915 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-19T00:43:49.940 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-19T00:43:49.940 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-19T00:43:49.940 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199', 'branch': 'wip-yuri5-testing-2024-04-17-1400', 'description': 'rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7663199', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri5-testing-2024-04-17-1400'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri5-testing-2024-04-17-1400', 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mon.d', 'mon.e', 'osd.0', 'osd.1', 'osd.2'], ['mon.f', 'mon.g', 'mon.h', 'mon.i', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1937, 'sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri5-testing-2024-04-17-1400', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_8b4ab94f9c30da63487fa2f241ce12f483378860/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '8b4ab94f9c30da63487fa2f241ce12f483378860', 'targets': {'smithi043.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBI95gXZ4zVyone2S8SHXEndb9bMXVCh5+kGX01O7ftejcUi77MTgcyz8sgacIRkn980TDKv2qMlWU1Ir/Pd/70Q=', 'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxAMGsllWyXdlTRNdwf2anP9RKvXywVq3dsQ2q18q6otGTFjE4YwKJ+uTIh+eiuKsRZzSga5mia9b0v3TEltdg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-18_22:49:05', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-19T00:43:49.941 DEBUG:teuthology.orchestra.run.smithi043:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T00:43:49.970 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-19T00:43:49.976 DEBUG:teuthology.orchestra.run.smithi043:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T00:43:50.019 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2024-04-19T00:43:50.032 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:50.032 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T00:43:50.078 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T00:43:50.078 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_1 2024-04-19T00:43:50.125 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 873 Links: 1 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-19 00:38:33.924526763 +0000 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-19 00:38:33.664529086 +0000 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-19 00:38:33.664529086 +0000 2024-04-19T00:43:50.126 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-19T00:43:50.126 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T00:43:50.180 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-19T00:43:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-19T00:43:50.181 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000328667 s, 1.6 MB/s 2024-04-19T00:43:50.182 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T00:43:50.231 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_2 2024-04-19T00:43:50.277 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T00:43:50.277 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:50.277 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 897 Links: 1 2024-04-19T00:43:50.278 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:50.278 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-19 00:38:34.452522048 +0000 2024-04-19T00:43:50.278 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-19 00:38:34.184524441 +0000 2024-04-19T00:43:50.278 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-19 00:38:34.184524441 +0000 2024-04-19T00:43:50.278 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-19T00:43:50.278 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T00:43:50.332 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-19T00:43:50.332 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-19T00:43:50.332 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000263575 s, 1.9 MB/s 2024-04-19T00:43:50.333 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T00:43:50.383 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_3 2024-04-19T00:43:50.429 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T00:43:50.429 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:50.429 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-04-19T00:43:50.430 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:50.430 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-19 00:38:34.924517832 +0000 2024-04-19T00:43:50.430 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-19 00:38:34.668520117 +0000 2024-04-19T00:43:50.430 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-19 00:38:34.668520117 +0000 2024-04-19T00:43:50.430 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-19T00:43:50.430 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T00:43:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-19T00:43:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-19T00:43:50.484 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000311889 s, 1.6 MB/s 2024-04-19T00:43:50.485 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T00:43:50.535 DEBUG:teuthology.orchestra.run.smithi043:> stat /dev/vg_nvme/lv_4 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout:Device: 5h/5d Inode: 931 Links: 1 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout:Access: 2024-04-19 00:38:35.148515831 +0000 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout:Modify: 2024-04-19 00:38:35.144515867 +0000 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout:Change: 2024-04-19 00:38:35.144515867 +0000 2024-04-19T00:43:50.582 INFO:teuthology.orchestra.run.smithi043.stdout: Birth: - 2024-04-19T00:43:50.583 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T00:43:50.636 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records in 2024-04-19T00:43:50.636 INFO:teuthology.orchestra.run.smithi043.stderr:1+0 records out 2024-04-19T00:43:50.636 INFO:teuthology.orchestra.run.smithi043.stderr:512 bytes copied, 0.000294858 s, 1.7 MB/s 2024-04-19T00:43:50.637 DEBUG:teuthology.orchestra.run.smithi043:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T00:43:50.687 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T00:43:50.688 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:50.688 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-04-19T00:43:50.694 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-19T00:43:50.694 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-19 00:38:30.852648625 +0000 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-19 00:38:30.572651121 +0000 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-19 00:38:30.572651121 +0000 2024-04-19T00:43:50.740 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-19T00:43:50.741 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-19T00:43:50.794 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-19T00:43:50.794 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-19T00:43:50.794 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000445387 s, 1.1 MB/s 2024-04-19T00:43:50.796 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-19T00:43:50.845 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-19 00:38:31.100646413 +0000 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-19 00:38:31.096646447 +0000 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-19 00:38:31.096646447 +0000 2024-04-19T00:43:50.896 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-19T00:43:50.897 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-19T00:43:50.950 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-19T00:43:50.950 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-19T00:43:50.950 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000440843 s, 1.2 MB/s 2024-04-19T00:43:50.951 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-19T00:43:51.001 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-04-19T00:43:51.047 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-19T00:43:51.047 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:51.047 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-04-19T00:43:51.047 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:51.047 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-19 00:38:31.824639953 +0000 2024-04-19T00:43:51.048 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-19 00:38:31.580642129 +0000 2024-04-19T00:43:51.048 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-19 00:38:31.580642129 +0000 2024-04-19T00:43:51.048 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-19T00:43:51.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-19T00:43:51.102 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-19T00:43:51.102 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-19T00:43:51.102 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000383747 s, 1.3 MB/s 2024-04-19T00:43:51.103 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-19T00:43:51.152 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-04-19 00:38:32.320635529 +0000 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-04-19 00:38:32.072637741 +0000 2024-04-19T00:43:51.199 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-04-19 00:38:32.072637741 +0000 2024-04-19T00:43:51.200 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: - 2024-04-19T00:43:51.200 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-19T00:43:51.253 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-04-19T00:43:51.253 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-04-19T00:43:51.253 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00038253 s, 1.3 MB/s 2024-04-19T00:43:51.255 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-19T00:43:51.308 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T00:43:51.308 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-19T00:43:51.308 INFO:tasks.ceph:Generating config... 2024-04-19T00:43:51.309 INFO:tasks.ceph:[client] debug monc = 20 2024-04-19T00:43:51.310 INFO:tasks.ceph:[client] debug ms = 1 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] debug ms = 1 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] debug objecter = 20 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] debug rados = 20 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-04-19T00:43:51.310 INFO:tasks.ceph:[global] ms type = async 2024-04-19T00:43:51.310 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-19T00:43:51.310 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-19T00:43:51.310 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-19T00:43:51.310 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-19T00:43:51.310 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-19T00:43:51.310 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-04-19T00:43:51.311 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-19T00:43:51.311 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-04-19T00:43:51.311 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-04-19T00:43:51.311 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-19T00:43:51.311 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-19T00:43:51.311 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-19T00:43:51.311 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd class default list = * 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd class load list = * 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-19T00:43:51.312 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-19T00:43:51.313 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-19T00:43:51.313 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-19T00:43:51.313 INFO:tasks.ceph:Setting up mon.a... 2024-04-19T00:43:51.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-19T00:43:51.349 INFO:teuthology.orchestra.run.smithi043.stdout:creating /etc/ceph/ceph.keyring 2024-04-19T00:43:51.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-19T00:43:51.440 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T00:43:51.497 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.43'), ('mon.b', '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]'), ('mon.c', '[v2:172.21.15.43:3302,v1:172.21.15.43:6791]'), ('mon.d', '[v2:172.21.15.43:3303,v1:172.21.15.43:6792]'), ('mon.e', '[v2:172.21.15.43:3304,v1:172.21.15.43:6793]'), ('mon.f', '172.21.15.161'), ('mon.g', '[v2:172.21.15.161:3301,v1:172.21.15.161:6790]'), ('mon.h', '[v2:172.21.15.161:3302,v1:172.21.15.161:6791]'), ('mon.i', '[v2:172.21.15.161:3303,v1:172.21.15.161:6792]')] 2024-04-19T00:43:51.498 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.43,[v2:172.21.15.43:3301,v1:172.21.15.43:6790],[v2:172.21.15.43:3302,v1:172.21.15.43:6791],[v2:172.21.15.43:3303,v1:172.21.15.43:6792],[v2:172.21.15.43:3304,v1:172.21.15.43:6793],172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790],[v2:172.21.15.161:3302,v1:172.21.15.161:6791],[v2:172.21.15.161:3303,v1:172.21.15.161:6792]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mon.d': {}, 'mon.e': {}, 'mon.f': {}, 'mon.g': {}, 'mon.h': {}, 'mon.i': {}} 2024-04-19T00:43:51.499 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:51.499 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T00:43:51.546 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.43 --addv b '[v2:172.21.15.43:3301,v1:172.21.15.43:6790]' --addv c '[v2:172.21.15.43:3302,v1:172.21.15.43:6791]' --addv d '[v2:172.21.15.43:3303,v1:172.21.15.43:6792]' --addv e '[v2:172.21.15.43:3304,v1:172.21.15.43:6793]' --add f 172.21.15.161 --addv g '[v2:172.21.15.161:3301,v1:172.21.15.161:6790]' --addv h '[v2:172.21.15.161:3302,v1:172.21.15.161:6791]' --addv i '[v2:172.21.15.161:3303,v1:172.21.15.161:6792]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:51.654 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:51.654 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: generated fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:43:51.654 INFO:teuthology.orchestra.run.smithi043.stdout:setting min_mon_release = quincy 2024-04-19T00:43:51.654 INFO:teuthology.orchestra.run.smithi043.stdout:epoch 0 2024-04-19T00:43:51.654 INFO:teuthology.orchestra.run.smithi043.stdout:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:min_mon_release 17 (quincy) 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:election_strategy: 3 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:3: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:4: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:5: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:6: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:7: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:8: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:43:51.655 INFO:teuthology.orchestra.run.smithi043.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (9 monitors) 2024-04-19T00:43:51.657 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-19T00:43:51.661 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1280a06a-2032-4fc0-a20f-543d29f598e9... 2024-04-19T00:43:51.663 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T00:43:51.706 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-19T00:43:51.735 INFO:teuthology.orchestra.run.smithi043.stdout:[global] 2024-04-19T00:43:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: chdir = "" 2024-04-19T00:43:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T00:43:51.735 INFO:teuthology.orchestra.run.smithi043.stdout: auth supported = cephx 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: filestore xattr use omap = true 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: mon clock drift allowed = 1.000 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: osd crush chooseleaf type = 0 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: auth debug = true 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on old message = true 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: ms die on bug = true 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: mon pg warn max object skew = 0 2024-04-19T00:43:51.736 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default size = 2 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow primary affinity = true 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd allow pg remap = true 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on legacy crush tunables = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on crush straw calc version zero = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on no sortbitwise = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on osd down out interval zero = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on too few osds = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T00:43:51.737 INFO:teuthology.orchestra.run.smithi043.stdout: mon_allow_pool_size_one = true 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: osd default data pool replay window = 5 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: mon allow pool delete = true 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: mon cluster log file level = debug 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: debug asserts on shutdown = true 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: mon health detail to clog = false 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: mon host = "172.21.15.43,[v2:172.21.15.43:3301,v1:172.21.15.43:6790],[v2:172.21.15.43:3302,v1:172.21.15.43:6791],[v2:172.21.15.43:3303,v1:172.21.15.43:6792],[v2:172.21.15.43:3304,v1:172.21.15.43:6793],172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790],[v2:172.21.15.161:3302,v1:172.21.15.161:6791],[v2:172.21.15.161:3303,v1:172.21.15.161:6792]" 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: debug objecter = 20 2024-04-19T00:43:51.738 INFO:teuthology.orchestra.run.smithi043.stdout: debug rados = 20 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: mon client directed command retry = 5 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: mon election default strategy = 3 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: ms inject socket failures = 5000 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: ms type = async 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: fsid = 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout:[osd] 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: osd journal size = 100 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub load threshold = 5.0 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: osd scrub max interval = 600 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock profile = high_recovery_ops 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.739 INFO:teuthology.orchestra.run.smithi043.stdout: osd recover clone overlap = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd recovery max chunk = 1048576 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug shutdown = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug op order = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify stray on activate = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug trim objects = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd open classes on start = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug pg log writeout = true 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd deep scrub update digest min age = 30 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: osd map max advance = 10 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.740 INFO:teuthology.orchestra.run.smithi043.stdout: journal zero on create = true 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: filestore ondisk finisher threads = 3 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: filestore apply finisher threads = 3 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: bdev debug aio = true 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug misdirected ops = true 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: bluefs allocator = hybrid 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore allocator = hybrid 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore block size = 96636764160 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore fsck on mount = True 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: bluestore zero block detection = True 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluefs = 20 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: debug bluestore = 20 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-19T00:43:51.741 INFO:teuthology.orchestra.run.smithi043.stdout: debug osd = 20 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: debug rocksdb = 10 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd full ratio = 0.9 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd nearfull ratio = 0.8 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd class default list = * 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd class load list = * 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify cached snaps = True 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd debug verify missing on start = True 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd failsafe full ratio = 0.95 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd mclock override recovery settings = True 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd objectstore = bluestore 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd op queue = debug_random 2024-04-19T00:43:51.742 INFO:teuthology.orchestra.run.smithi043.stdout: osd op queue cut off = debug_random 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr] 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug mgr = 20 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: mgr/telemetry/nag = false 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout:[mon] 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug mon = 20 2024-04-19T00:43:51.743 INFO:teuthology.orchestra.run.smithi043.stdout: debug paxos = 20 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: debug auth = 20 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon data avail warn = 5 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon mgr mkfs grace = 240 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min pgs per osd = 4 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd reporter subtree level = osd 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon osd prime pg temp = true 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon reweight min bytes per osd = 10 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T00:43:51.744 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: # 1m isn't quite enough 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon_down_mkfs_grace = 2m 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon_warn_on_filestore_osds = false 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon min osdmap epochs = 25 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon osdmap full prune interval = 2 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon osdmap full prune min = 15 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon osdmap full prune txsize = 2 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon scrub inject crc mismatch = 0.01 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon scrub inject missing keys = 0.05 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon scrub interval = 300 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: mon warn on pool no app = False 2024-04-19T00:43:51.745 INFO:teuthology.orchestra.run.smithi043.stdout: paxos service trim min = 5 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout:[client] 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: rgw cache enabled = true 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable ops log = true 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: rgw enable usage log = true 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: debug monc = 20 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout: debug ms = 1 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.a] 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.b] 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.c] 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.d] 2024-04-19T00:43:51.746 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.e] 2024-04-19T00:43:51.747 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.f] 2024-04-19T00:43:51.747 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.g] 2024-04-19T00:43:51.747 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.h] 2024-04-19T00:43:51.747 INFO:teuthology.orchestra.run.smithi043.stdout:[mon.i] 2024-04-19T00:43:51.748 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2024-04-19T00:43:51.748 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2024-04-19T00:43:51.748 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-19T00:43:51.748 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2024-04-19T00:43:51.748 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.749 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-19T00:43:51.750 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = "172.21.15.43,[v2:172.21.15.43:3301,v1:172.21.15.43:6790],[v2:172.21.15.43:3302,v1:172.21.15.43:6791],[v2:172.21.15.43:3303,v1:172.21.15.43:6792],[v2:172.21.15.43:3304,v1:172.21.15.43:6793],172.21.15.161,[v2:172.21.15.161:3301,v1:172.21.15.161:6790],[v2:172.21.15.161:3302,v1:172.21.15.161:6791],[v2:172.21.15.161:3303,v1:172.21.15.161:6792]" 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-19T00:43:51.751 INFO:teuthology.orchestra.run.smithi161.stdout: debug objecter = 20 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: debug rados = 20 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: mon client directed command retry = 5 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: mon election default strategy = 3 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: ms inject socket failures = 5000 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: ms type = async 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.752 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug trim objects = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.753 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: bluefs allocator = hybrid 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore allocator = hybrid 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore block size = 96636764160 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore fsck on mount = True 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore zero block detection = True 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 20 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 20 2024-04-19T00:43:51.754 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 10 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd backfillfull_ratio = 0.85 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd full ratio = 0.9 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd nearfull ratio = 0.8 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd class default list = * 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd class load list = * 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify cached snaps = True 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify missing on start = True 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd failsafe full ratio = 0.95 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock override recovery settings = True 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd objectstore = bluestore 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd op queue = debug_random 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: osd op queue cut off = debug_random 2024-04-19T00:43:51.755 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2024-04-19T00:43:51.756 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-19T00:43:51.757 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon min osdmap epochs = 25 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon osdmap full prune interval = 2 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon osdmap full prune min = 15 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon osdmap full prune txsize = 2 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon scrub inject crc mismatch = 0.01 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon scrub inject missing keys = 0.05 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon scrub interval = 300 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on pool no app = False 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: paxos service trim min = 5 2024-04-19T00:43:51.758 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: debug monc = 20 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.b] 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.c] 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.d] 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.e] 2024-04-19T00:43:51.759 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.f] 2024-04-19T00:43:51.760 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.g] 2024-04-19T00:43:51.760 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.h] 2024-04-19T00:43:51.760 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.i] 2024-04-19T00:43:51.760 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-19T00:43:51.760 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-19T00:43:51.838 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-19T00:43:51.838 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:51.838 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-19T00:43:51.886 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:51.886 DEBUG:teuthology.orchestra.run.smithi043:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-19T00:43:51.933 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T00:43:51.933 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:51.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T00:43:51.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T00:43:51.998 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:43:51.998 DEBUG:teuthology.orchestra.run.smithi043:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:52.046 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T00:43:52.046 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:52.046 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-19T00:43:52.046 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-19T00:43:52.069 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:43:52.070 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:52.116 INFO:tasks.ceph:Setting up mon nodes... 2024-04-19T00:43:52.116 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-19T00:43:52.116 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T00:43:52.200 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-19T00:43:52.218 INFO:tasks.ceph:Setting up mds nodes... 2024-04-19T00:43:52.219 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-19T00:43:52.219 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-19T00:43:52.296 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-19T00:43:52.307 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-19T00:43:52.307 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi043.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-19T00:43:52.307 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-19T00:43:52.321 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T00:43:52.322 INFO:tasks.ceph:role: osd.0 2024-04-19T00:43:52.322 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T00:43:52.322 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T00:43:52.378 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T00:43:52.379 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T00:43:52.401 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-19T00:43:52.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-19T00:43:52.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-19T00:43:52.462 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-19T00:43:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:43:52.515 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T00:43:52.516 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-19T00:43:52.569 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T00:43:52.569 INFO:tasks.ceph:role: osd.1 2024-04-19T00:43:52.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T00:43:52.569 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T00:43:52.624 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T00:43:52.625 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T00:43:52.625 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T00:43:52.648 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-19T00:43:52.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-19T00:43:52.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-19T00:43:52.708 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-19T00:43:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:43:52.761 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T00:43:52.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-19T00:43:52.815 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-19T00:43:52.815 INFO:tasks.ceph:role: osd.2 2024-04-19T00:43:52.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T00:43:52.815 DEBUG:teuthology.orchestra.run.smithi043:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T00:43:52.870 INFO:teuthology.orchestra.run.smithi043.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T00:43:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T00:43:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T00:43:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T00:43:52.870 INFO:teuthology.orchestra.run.smithi043.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T00:43:52.870 INFO:teuthology.orchestra.run.smithi043.stdout: = sunit=0 swidth=0 blks 2024-04-19T00:43:52.871 INFO:teuthology.orchestra.run.smithi043.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T00:43:52.871 INFO:teuthology.orchestra.run.smithi043.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T00:43:52.871 INFO:teuthology.orchestra.run.smithi043.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T00:43:52.871 INFO:teuthology.orchestra.run.smithi043.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T00:43:52.894 INFO:teuthology.orchestra.run.smithi043.stdout:Discarding blocks...Done. 2024-04-19T00:43:52.896 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi043.front.sepia.ceph.com -o noatime 2024-04-19T00:43:52.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-19T00:43:52.958 DEBUG:teuthology.orchestra.run.smithi043:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-19T00:43:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:43:53.012 INFO:teuthology.orchestra.run.smithi043.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T00:43:53.012 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:53.124+0000 7f724c5807c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-19T00:43:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:53.124+0000 7f724c5807c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-19T00:43:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:53.124+0000 7f724c5807c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T00:43:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:53.124+0000 7f724c5807c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T00:43:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:53.124+0000 7f724c5807c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-19T00:43:53.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:53.124+0000 7f724c5807c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-19T00:43:56.118 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-19T00:43:56.132 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:56.212+0000 7f51f59d77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-19T00:43:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:56.212+0000 7f51f59d77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-19T00:43:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:56.212+0000 7f51f59d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T00:43:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:56.212+0000 7f51f59d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T00:43:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:56.212+0000 7f51f59d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-19T00:43:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:56.216+0000 7f51f59d77c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-19T00:43:59.201 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-19T00:43:59.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:43:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:59.352+0000 7f2ad9f337c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-19T00:43:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:59.352+0000 7f2ad9f337c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-19T00:43:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:59.352+0000 7f2ad9f337c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T00:43:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:59.352+0000 7f2ad9f337c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T00:43:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:59.352+0000 7f2ad9f337c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-19T00:43:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:43:59.352+0000 7f2ad9f337c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-19T00:44:02.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-19T00:44:02.375 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-19T00:44:02.390 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T00:44:02.391 INFO:tasks.ceph:role: osd.3 2024-04-19T00:44:02.391 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T00:44:02.391 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-19T00:44:02.455 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T00:44:02.455 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T00:44:02.456 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T00:44:02.909 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-04-19T00:44:02.911 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-04-19T00:44:02.911 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-19T00:44:03.003 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-19T00:44:03.012 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T00:44:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:44:03.013 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-19T00:44:03.064 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T00:44:03.064 INFO:tasks.ceph:role: osd.4 2024-04-19T00:44:03.064 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T00:44:03.065 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-19T00:44:03.126 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T00:44:03.127 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T00:44:03.588 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-04-19T00:44:03.590 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-04-19T00:44:03.590 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-19T00:44:03.759 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-19T00:44:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:44:03.771 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T00:44:03.771 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-19T00:44:03.826 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-19T00:44:03.826 INFO:tasks.ceph:role: osd.5 2024-04-19T00:44:03.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T00:44:03.827 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-19T00:44:03.891 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-19T00:44:04.368 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-04-19T00:44:04.370 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-04-19T00:44:04.371 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-19T00:44:04.532 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-19T00:44:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:44:04.546 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: /sbin/restorecon: command not found 2024-04-19T00:44:04.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:44:04.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:04.662+0000 7fbebc3927c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-19T00:44:04.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:04.662+0000 7fbebc3927c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-19T00:44:04.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:04.662+0000 7fbebc3927c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T00:44:04.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:04.662+0000 7fbebc3927c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T00:44:04.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:04.662+0000 7fbebc3927c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-19T00:44:04.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:04.666+0000 7fbebc3927c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-19T00:44:07.642 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-19T00:44:07.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:44:07.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:07.734+0000 7f73cd4117c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-19T00:44:07.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:07.734+0000 7f73cd4117c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-19T00:44:07.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:07.734+0000 7f73cd4117c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T00:44:07.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:07.734+0000 7f73cd4117c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T00:44:07.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:07.734+0000 7f73cd4117c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-19T00:44:07.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:07.734+0000 7f73cd4117c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-19T00:44:10.711 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-19T00:44:10.768 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:44:10.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:10.846+0000 7fec161ce7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-19T00:44:10.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:10.846+0000 7fec161ce7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-19T00:44:10.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:10.846+0000 7fec161ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T00:44:10.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:10.846+0000 7fec161ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T00:44:10.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:10.846+0000 7fec161ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-19T00:44:10.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:10.850+0000 7fec161ce7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-19T00:44:13.826 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-19T00:44:13.884 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-19T00:44:13.884 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:44:13.884 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-19T00:44:13.899 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:44:13.899 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-19T00:44:13.953 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:44:13.953 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-19T00:44:14.009 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:14.009 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-19T00:44:14.022 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:14.022 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-19T00:44:14.078 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:14.078 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-19T00:44:14.130 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:14.130 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-19T00:44:14.186 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:14.186 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-19T00:44:14.235 INFO:tasks.ceph:Adding keys to all mons... 2024-04-19T00:44:14.235 DEBUG:teuthology.orchestra.run.smithi043:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T00:44:14.237 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.0] 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBJviFmPGOgBxAAOPenvjZ1RjaDlxFx8O5o+A== 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.1] 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBMviFmBD35DBAAhhjnhYCkSFg55KW5AjpONw== 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.2] 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBPviFmV6dMFRAAjK1lR/TKOhADM+jPrlPTjA== 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout:[mgr.x] 2024-04-19T00:44:14.281 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBIviFmfqoJDBAAn9dkodnOPlyOPQI+Vpbr1A== 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.3] 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBUviFml/PBJxAAzE+HskMeRyVpTh8bfq59dQ== 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.4] 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBXviFmR/oBLBAAMSkR9yXnbktz9IAcm4ADHw== 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout:[osd.5] 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBaviFmeXO9MhAA01ADvIu8w+1JwQa5PmYtzg== 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout:[client.0] 2024-04-19T00:44:14.282 INFO:teuthology.orchestra.run.smithi043.stdout: key = AQBIviFmmM2/ERAAw35WISp8l1Bcwr8VfEToAg== 2024-04-19T00:44:14.288 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2024-04-19T00:44:14.288 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBJviFmPGOgBxAAOPenvjZ1RjaDlxFx8O5o+A== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBMviFmBD35DBAAhhjnhYCkSFg55KW5AjpONw== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBPviFmV6dMFRAAjK1lR/TKOhADM+jPrlPTjA== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBIviFmfqoJDBAAn9dkodnOPlyOPQI+Vpbr1A== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.3] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBUviFml/PBJxAAzE+HskMeRyVpTh8bfq59dQ== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.4] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBXviFmR/oBLBAAMSkR9yXnbktz9IAcm4ADHw== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.5] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBaviFmeXO9MhAA01ADvIu8w+1JwQa5PmYtzg== 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-04-19T00:44:14.289 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBIviFmmM2/ERAAw35WISp8l1Bcwr8VfEToAg== 2024-04-19T00:44:14.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.326 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.379 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.381 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.423 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.467 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T00:44:14.550 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-19T00:44:14.595 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.634 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.678 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.763 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.814 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-19T00:44:14.867 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T00:44:14.901 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-19T00:44:14.967 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-19T00:44:14.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-19T00:44:14.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:15.080 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-19T00:44:15.094 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-19T00:44:15.154 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:15.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-19T00:44:15.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-19T00:44:15.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:15.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-19T00:44:15.420 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-d 2024-04-19T00:44:15.474 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i d --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:15.561 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-d 2024-04-19T00:44:15.571 DEBUG:teuthology.orchestra.run.smithi043:> sudo mkdir -p /var/lib/ceph/mon/ceph-e 2024-04-19T00:44:15.622 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i e --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:15.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-e 2024-04-19T00:44:15.765 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-f 2024-04-19T00:44:15.779 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i f --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:15.874 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-f 2024-04-19T00:44:15.885 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-g 2024-04-19T00:44:15.935 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i g --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:16.027 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-g 2024-04-19T00:44:16.038 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-h 2024-04-19T00:44:16.093 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i h --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:16.183 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-h 2024-04-19T00:44:16.194 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-i 2024-04-19T00:44:16.247 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i i --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-19T00:44:16.335 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-i 2024-04-19T00:44:16.346 DEBUG:teuthology.orchestra.run.smithi043:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:44:16.348 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-19T00:44:16.398 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-19T00:44:16.398 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T00:44:16.399 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T00:44:16.401 INFO:tasks.ceph.mon.a:Started 2024-04-19T00:44:16.401 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T00:44:16.401 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T00:44:16.404 INFO:tasks.ceph.mon.b:Started 2024-04-19T00:44:16.404 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T00:44:16.404 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T00:44:16.406 INFO:tasks.ceph.mon.c:Started 2024-04-19T00:44:16.407 INFO:tasks.ceph.mon.d:Restarting daemon 2024-04-19T00:44:16.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-04-19T00:44:16.409 INFO:tasks.ceph.mon.d:Started 2024-04-19T00:44:16.409 INFO:tasks.ceph.mon.e:Restarting daemon 2024-04-19T00:44:16.409 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-04-19T00:44:16.411 INFO:tasks.ceph.mon.e:Started 2024-04-19T00:44:16.411 INFO:tasks.ceph.mon.f:Restarting daemon 2024-04-19T00:44:16.411 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-04-19T00:44:16.444 INFO:tasks.ceph.mon.f:Started 2024-04-19T00:44:16.445 INFO:tasks.ceph.mon.g:Restarting daemon 2024-04-19T00:44:16.445 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-04-19T00:44:16.447 INFO:tasks.ceph.mon.g:Started 2024-04-19T00:44:16.448 INFO:tasks.ceph.mon.h:Restarting daemon 2024-04-19T00:44:16.448 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-04-19T00:44:16.450 INFO:tasks.ceph.mon.h:Started 2024-04-19T00:44:16.450 INFO:tasks.ceph.mon.i:Restarting daemon 2024-04-19T00:44:16.450 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-04-19T00:44:16.452 INFO:tasks.ceph.mon.i:Started 2024-04-19T00:44:16.452 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-19T00:44:16.452 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-19T00:44:16.453 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-19T00:44:16.454 INFO:tasks.ceph.mgr.x:Started 2024-04-19T00:44:16.455 INFO:tasks.ceph:Setting crush tunables to default 2024-04-19T00:44:16.455 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd crush tunables default 2024-04-19T00:44:16.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient: get_monmap_and_config 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient: build_initial_monmap 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient: monmap: 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:16.591339+0000 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:16.591339+0000 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:16.590 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:16.591 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:16.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 Processor -- start 2024-04-19T00:44:16.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 -- start start 2024-04-19T00:44:16.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient: init 2024-04-19T00:44:16.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:16.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient: _add_conns ranks=[0,4,8,9,3,10,1,7,6,2,5] 2024-04-19T00:44:16.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec7809da40 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): picked mon.noname-c con 0x7fec7809ffa0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): picked mon.noname-e con 0x7fec780a03c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec780a4da0 con 0x7fec7809da40 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fec780a1560 con 0x7fec7809ffa0 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fec780a0fa0 con 0x7fec780a03c0 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec7dea9640 10 monclient(hunting): authenticate will time out at 926.718243s 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:16.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec777fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fec7809da40 0x7fec7809de20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:59544/0 (socket says 172.21.15.43:59544) 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 learned_addr learned my addr 172.21.15.43:0/3620889992 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec780a03c0 auth_method 0 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec7809ffa0 auth_method 0 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:16.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:16.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:16.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:16.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:44:16.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.588+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:44:16.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:16.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec780a03c0 auth_method 0 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec7809ffa0 auth_method 0 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:16.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:16.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:16.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:16.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:44:16.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:16.792+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:44:17.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec7809ffa0 auth_method 0 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec780a03c0 auth_method 0 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:17.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:44:17.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:17.196+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec780a03c0 auth_method 0 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec7809ffa0 auth_method 0 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:18.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:44:18.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:18.000+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): tick 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 monclient(hunting): continuing hunt 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 msgr2=0x7fec7813d280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fec780a03c0 0x7fec7813d280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 msgr2=0x7fec7813ad30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec7809ffa0 0x7fec7813ad30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 >> v1:172.21.15.43:6789/0 conn(0x7fec7809da40 legacy=0x7fec7809de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:19.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient: _add_conns ranks=[2,7,10,6,1,5,9,3] 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): picked mon.noname-b con 0x7fec64005a90 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): picked mon.noname-i con 0x7fec640084f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): picked mon.noname-f con 0x7fec6400b050 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fec780a1560 con 0x7fec6400b050 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fec780a0fa0 con 0x7fec64005a90 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fec6400dc00 con 0x7fec640084f0 2024-04-19T00:44:19.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec757fa640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 10 monclient(hunting): get_auth_request con 0x7fec64005a90 auth_method 0 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec6400b050 auth_method 0 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:19.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec640084f0 auth_method 0 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:19.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.592+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 10 monclient(hunting): get_auth_request con 0x7fec64005a90 auth_method 0 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec6400b050 auth_method 0 2024-04-19T00:44:19.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec640084f0 auth_method 0 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:19.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:44:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:44:19.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:19.796+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 10 monclient(hunting): get_auth_request con 0x7fec64005a90 auth_method 0 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:20.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec6400b050 auth_method 0 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec640084f0 auth_method 0 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:44:20.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:20.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:20.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:20.200+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 10 monclient(hunting): get_auth_request con 0x7fec64005a90 auth_method 0 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:21.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec640084f0 auth_method 0 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec6400b050 auth_method 0 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:21.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:44:21.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:44:21.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:21.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:44:21.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:21.004+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:44:22.417 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:22.413+0000 7f2a53c28180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-19T00:44:22.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): tick 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 monclient(hunting): continuing hunt 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 msgr2=0x7fec6400a980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec640084f0 0x7fec6400a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 msgr2=0x7fec64007ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec64005a90 0x7fec64007ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 msgr2=0x7fec6400d510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.161:3300/0 conn(0x7fec6400b050 0x7fec6400d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 reap_dead start 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient: _add_conns ranks=[3,6,9,5,1] 2024-04-19T00:44:22.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec6400e1f0 0x7fec640105e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): picked mon.noname-g con 0x7fec6400e1f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64010bc0 0x7fec64012fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): picked mon.noname-d con 0x7fec64010bc0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 1 -- 172.21.15.43:0/3620889992 reap_dead start 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.43:3300/0 conn(0x7fec64005e90 0x7fec64008970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): picked mon.noname-a con 0x7fec64005e90 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fec780a1560 con 0x7fec64005e90 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fec780a0fa0 con 0x7fec6400e1f0 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 1 -- 172.21.15.43:0/3620889992 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fec6400dc00 con 0x7fec64010bc0 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec757fa640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:22.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64010bc0 0x7fec64012fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): get_auth_request con 0x7fec64010bc0 auth_method 0 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.43:3300/0 conn(0x7fec64005e90 0x7fec64008970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec6400e1f0 0x7fec640105e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): get_auth_request con 0x7fec64005e90 auth_method 0 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): get_auth_request con 0x7fec6400e1f0 auth_method 0 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:22.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.588+0000 7fec777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 10 monclient(hunting): handle_auth_done global_id 4119 payload 274 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec6400e1f0 msgr2=0x7fec640105e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec6400e1f0 0x7fec640105e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 >> v2:172.21.15.43:3300/0 conn(0x7fec64005e90 msgr2=0x7fec64008970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.43:3300/0 conn(0x7fec64005e90 0x7fec64008970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 1 monclient: found mon.noname-d 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec76ffd640 1 -- 172.21.15.43:0/3620889992 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec780a1280 con 0x7fec64010bc0 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.592+0000 7fec77fff640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.43:3300/0 conn(0x7fec64005e90 0x7fec64008970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec76ffd640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64010bc0 0x7fec64012fb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fec7806f870 tx=0x7fec6c01e2e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=3f5f6a47a4f8551e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:22.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 1 -- 172.21.15.43:0/3620889992 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec6c018450 con 0x7fec64010bc0 2024-04-19T00:44:22.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: dump: 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:22.600 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 1 -- 172.21.15.43:0/3620889992 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c018630 con 0x7fec64010bc0 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 1 -- 172.21.15.43:0/3620889992 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec6c018c50 con 0x7fec64010bc0 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:22.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec767fc640 10 monclient: dump: 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:22.602 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:22.603 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:22.603 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:22.603 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:22.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 5 monclient: authenticate success, global_id 4119 2024-04-19T00:44:22.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:22.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 4 set_mon_vals no callback set 2024-04-19T00:44:22.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 10 monclient: shutdown 2024-04-19T00:44:22.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:22.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 1 -- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64010bc0 msgr2=0x7fec64012fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.596+0000 7fec7dea9640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64010bc0 0x7fec64012fb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fec7806f870 tx=0x7fec6c01e2e0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 -- 172.21.15.43:0/3620889992 shutdown_connections 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 --2- 172.21.15.43:0/3620889992 >> v2:172.21.15.43:3300/0 conn(0x7fec64005e90 0x7fec64008970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64010bc0 0x7fec64012fb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 --2- 172.21.15.43:0/3620889992 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec6400e1f0 0x7fec640105e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 -- 172.21.15.43:0/3620889992 >> 172.21.15.43:0/3620889992 conn(0x7fec780a3ef0 msgr2=0x7fec780a42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 -- 172.21.15.43:0/3620889992 shutdown_connections 2024-04-19T00:44:22.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 -- 172.21.15.43:0/3620889992 wait complete. 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 10 monclient: build_initial_monmap 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 10 monclient: monmap: 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:22.606 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 librados: starting msgr at 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 librados: starting objecter 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 Processor -- start 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 -- start start 2024-04-19T00:44:22.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 librados: setting wanted keys 2024-04-19T00:44:22.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 1 librados: calling monclient init 2024-04-19T00:44:22.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.600+0000 7fec7dea9640 10 monclient: init 2024-04-19T00:44:22.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:22.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _add_conns ranks=[6,2,7,0,8,5,3,4,1] 2024-04-19T00:44:22.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 0x7fec78072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:22.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): picked mon.noname-d con 0x7fec64005e90 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec6400e1f0 0x7fec78072740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): picked mon.noname-b con 0x7fec6400e1f0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec64010bc0 0x7fec78072c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): picked mon.noname-i con 0x7fec64010bc0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fec781365b0 con 0x7fec6400e1f0 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fec78136430 con 0x7fec64005e90 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fec78136730 con 0x7fec64010bc0 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient(hunting): authenticate will time out at 932.732823s 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 0x7fec78072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 0x7fec78072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33338/0 (socket says 172.21.15.43:33338) 2024-04-19T00:44:22.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 -- 172.21.15.43:0/33307471 learned_addr learned my addr 172.21.15.43:0/33307471 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): get_auth_request con 0x7fec64005e90 auth_method 0 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec76ffd640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec6400e1f0 0x7fec78072740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 -- 172.21.15.43:0/33307471 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec64010bc0 msgr2=0x7fec78072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec77fff640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec64010bc0 0x7fec78072c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec64010bc0 0x7fec78072c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 -- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec6400e1f0 msgr2=0x7fec78072740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:22.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec6400e1f0 0x7fec78072740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 monclient: found mon.noname-d 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 -- 172.21.15.43:0/33307471 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec78132790 con 0x7fec64005e90 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec76ffd640 10 monclient: get_auth_request con 0x7fec6400e1f0 auth_method 0 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec76ffd640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec6400e1f0 0x7fec78072740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec777fe640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 0x7fec78072200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fec68002230 tx=0x7fec680073e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 client.?.objecter ms_handle_connect 0x7fec64005e90 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec68004d80 con 0x7fec64005e90 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: dump: 2024-04-19T00:44:22.615 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:22.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: _finish_auth 0 2024-04-19T00:44:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6801c030 con 0x7fec64005e90 2024-04-19T00:44:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec68005930 con 0x7fec64005e90 2024-04-19T00:44:22.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:22.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:22.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 monclient: dump: 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:22.619 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 5 monclient: authenticate success, global_id 4128 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec75ffb640 4 set_mon_vals no callback set 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _renew_subs 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec781312e0 con 0x7fec64005e90 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _renew_subs 2024-04-19T00:44:22.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fec781324b0 con 0x7fec64005e90 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 librados: init done 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fec680053e0 con 0x7fec64005e90 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fec68010030 con 0x7fec64005e90 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 10 client.4128.objecter ms_dispatch 0x7fec78000ba0 osd_map(1..1 src has 1..1) 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 3 client.4128.objecter handle_osd_map got epochs [1,1] > 0 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 3 client.4128.objecter handle_osd_map decoding full epoch 1 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec780693b0 con 0x7fec64005e90 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.604+0000 7fec5ffff640 20 client.4128.objecter dump_active .. 0 homeless 2024-04-19T00:44:22.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.608+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fec68018cb0 con 0x7fec64005e90 2024-04-19T00:44:22.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.608+0000 7fec5ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:22.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.608+0000 7fec5ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:22.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.672+0000 7fec7dea9640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-19T00:44:22.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.672+0000 7fec7dea9640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-19T00:44:22.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.672+0000 7fec7dea9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:22.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.672+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fec7806d800 con 0x7fec64005e90 2024-04-19T00:44:22.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec5ffff640 1 -- 172.21.15.43:0/33307471 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fec68018440 con 0x7fec64005e90 2024-04-19T00:44:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec5ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-19T00:44:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec5ffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-19T00:44:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:adjusted tunables profile to default 2024-04-19T00:44:22.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 10 librados: watch_flush enter 2024-04-19T00:44:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 10 librados: watch_flush exit 2024-04-19T00:44:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 20 client.4128.objecter shutdown clearing up homeless session... 2024-04-19T00:44:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 10 client.4128.objecter successfully canceled tick 2024-04-19T00:44:22.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 10 monclient: shutdown 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 msgr2=0x7fec78072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 0x7fec78072200 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fec68002230 tx=0x7fec680073e0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 shutdown_connections 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec64010bc0 0x7fec78072c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec6400e1f0 0x7fec78072740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 --2- 172.21.15.43:0/33307471 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec64005e90 0x7fec78072200 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 >> 172.21.15.43:0/33307471 conn(0x7fec780a3ef0 msgr2=0x7fec781bfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 shutdown_connections 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 -- 172.21.15.43:0/33307471 wait complete. 2024-04-19T00:44:22.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:22.680+0000 7fec7dea9640 1 librados: shutdown 2024-04-19T00:44:22.700 INFO:tasks.ceph:check_enable_crimson: False 2024-04-19T00:44:22.700 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-19T00:44:22.700 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:44:22.700 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-19T00:44:22.713 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:44:22.713 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-19T00:44:22.771 DEBUG:teuthology.orchestra.run.smithi043:> set -ex 2024-04-19T00:44:22.771 DEBUG:teuthology.orchestra.run.smithi043:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-19T00:44:22.825 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:22.825 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-19T00:44:22.839 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:22.839 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-19T00:44:22.895 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:44:22.895 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-19T00:44:22.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new e366653d-95f2-4573-8eee-ccdd82879a6f 0 2024-04-19T00:44:22.959 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:22.957+0000 7f2a53c28180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.077+0000 7f72c8786640 10 monclient: get_monmap_and_config 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.077+0000 7f72c8786640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.077+0000 7f72c8786640 10 monclient: monmap: 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:44:23.082673+0000 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:44:23.082673+0000 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.081 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.082 INFO:teuthology.orchestra.run.smithi161.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:23.083 INFO:teuthology.orchestra.run.smithi161.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:23.083 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.077+0000 7f72c8786640 1 Processor -- start 2024-04-19T00:44:23.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.077+0000 7f72c8786640 1 -- start start 2024-04-19T00:44:23.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.077+0000 7f72c8786640 10 monclient: init 2024-04-19T00:44:23.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient: _add_conns ranks=[10,9,0,6,5,2,3,7,8,4,1] 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72c0132330 0x7f72c0132710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): picked mon.noname-f con 0x7f72c0132330 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f72c009f9f0 0x7f72c00a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): picked mon.noname-a con 0x7f72c009f9f0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72c00a3490 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c00a0b40 con 0x7f72c00a3490 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f72c00a0e20 con 0x7f72c009f9f0 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f72c00a1850 con 0x7f72c0132330 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient(hunting): authenticate will time out at 966.028640s 2024-04-19T00:44:23.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72c0132330 0x7f72c0132710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f72c0132330 0x7f72c0132710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35772/0 (socket says 172.21.15.161:35772) 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 -- 172.21.15.161:0/1342009775 learned_addr learned my addr 172.21.15.161:0/1342009775 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): get_auth_request con 0x7f72c0132330 auth_method 0 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 -- 172.21.15.161:0/1342009775 >> v2:172.21.15.43:3300/0 conn(0x7f72c009f9f0 msgr2=0x7f72c00a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 --2- 172.21.15.161:0/1342009775 >> v2:172.21.15.43:3300/0 conn(0x7f72c009f9f0 0x7f72c00a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 -- 172.21.15.161:0/1342009775 >> v1:172.21.15.43:6789/0 conn(0x7f72c00a3490 legacy=0x7f72c013c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 monclient: found mon.noname-f 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 -- 172.21.15.161:0/1342009775 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c00a4df0 con 0x7f72c0132330 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c6522640 1 --2- 172.21.15.161:0/1342009775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c0132330 0x7f72c0132710 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f72c009e700 tx=0x7f72bc01eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23980c1e1827827b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 1 -- 172.21.15.161:0/1342009775 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72bc0ad070 con 0x7f72c0132330 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: dump: 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.090 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 1 -- 172.21.15.161:0/1342009775 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc0a7b30 con 0x7f72c0132330 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 5 monclient: authenticate success, global_id 4105 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c8786640 4 set_mon_vals no callback set 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 1 -- 172.21.15.161:0/1342009775 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72bc0a7d60 con 0x7f72c0132330 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.081+0000 7f72c5520640 10 monclient: dump: 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.092 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.093 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: shutdown 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1342009775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c0132330 msgr2=0x7f72c0132710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 --2- 172.21.15.161:0/1342009775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c0132330 0x7f72c0132710 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f72c009e700 tx=0x7f72bc01eab0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1342009775 shutdown_connections 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 --2- 172.21.15.161:0/1342009775 >> v2:172.21.15.43:3300/0 conn(0x7f72c009f9f0 0x7f72c00a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 --2- 172.21.15.161:0/1342009775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72c0132330 0x7f72c0132710 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1342009775 >> 172.21.15.161:0/1342009775 conn(0x7f72c00a4040 msgr2=0x7f72c00a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1342009775 shutdown_connections 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1342009775 wait complete. 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: monmap: 2024-04-19T00:44:23.094 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.095 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 librados: starting msgr at 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 librados: starting objecter 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 Processor -- start 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- start start 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 librados: setting wanted keys 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 librados: calling monclient init 2024-04-19T00:44:23.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: init 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _add_conns ranks=[2,8,3,0,5,7,4,6,1] 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72c009f9f0 0x7f72c01ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): picked mon.noname-b con 0x7f72c009f9f0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f72c00a3490 0x7f72c01cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): picked mon.noname-e con 0x7f72c00a3490 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 0x7f72c01cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): picked mon.noname-g con 0x7f72c0132330 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:23.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f72c013ff40 con 0x7f72c009f9f0 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f72c013fdc0 con 0x7f72c0132330 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f72c01400c0 con 0x7f72c00a3490 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient(hunting): authenticate will time out at 966.033805s 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 0x7f72c01cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 0x7f72c01cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:34222/0 (socket says 172.21.15.161:34222) 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 -- 172.21.15.161:0/1055813705 learned_addr learned my addr 172.21.15.161:0/1055813705 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): get_auth_request con 0x7f72c0132330 auth_method 0 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6522640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72c009f9f0 0x7f72c01ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c5d21640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f72c00a3490 0x7f72c01cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6522640 10 monclient(hunting): get_auth_request con 0x7f72c009f9f0 auth_method 0 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6522640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6522640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6522640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 -- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f72c00a3490 msgr2=0x7f72c01cd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f72c00a3490 0x7f72c01cd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 -- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72c009f9f0 msgr2=0x7f72c01ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72c009f9f0 0x7f72c01ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 monclient: found mon.noname-g 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 -- 172.21.15.161:0/1055813705 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c009af50 con 0x7f72c0132330 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c5d21640 10 monclient: get_auth_request con 0x7f72c00a3490 auth_method 0 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c5d21640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f72c00a3490 0x7f72c01cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c6d23640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 0x7f72c01cda50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72b800c440 tx=0x7f72b800c900 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 client.?.objecter ms_handle_connect 0x7f72c0132330 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72b8017070 con 0x7f72c0132330 2024-04-19T00:44:23.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: dump: 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.104 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b80052e0 con 0x7f72c0132330 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 5 monclient: authenticate success, global_id 4116 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _renew_subs 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c009ff90 con 0x7f72c0132330 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f72b8010300 con 0x7f72c0132330 2024-04-19T00:44:23.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c4d1f640 4 set_mon_vals no callback set 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 monclient: dump: 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.106 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _renew_subs 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c012f7a0 con 0x7f72c0132330 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 librados: init done 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f72b8003600 con 0x7f72c0132330 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f72b8010b20 con 0x7f72c0132330 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 10 client.4116.objecter ms_dispatch 0x7f72c0000ba0 osd_map(2..2 src has 1..2) 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 3 client.4116.objecter handle_osd_map got epochs [2,2] > 0 2024-04-19T00:44:23.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 3 client.4116.objecter handle_osd_map decoding full epoch 2 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72b6ffd640 20 client.4116.objecter dump_active .. 0 homeless 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.085+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c00693b0 con 0x7f72c0132330 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.089+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f72b8020020 con 0x7f72c0132330 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.089+0000 7f72b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.089+0000 7f72b6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:23.157 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:23.153+0000 7f2a53c28180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-19T00:44:23.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.157+0000 7f72c8786640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e366653d-95f2-4573-8eee-ccdd82879a6f", "id": 0}] 2024-04-19T00:44:23.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.157+0000 7f72c8786640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e366653d-95f2-4573-8eee-ccdd82879a6f", "id": 0}] 2024-04-19T00:44:23.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.157+0000 7f72c8786640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.157+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e366653d-95f2-4573-8eee-ccdd82879a6f", "id": 0} v 0) -- 0x7f72c00973e0 con 0x7f72c0132330 2024-04-19T00:44:23.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72b6ffd640 1 -- 172.21.15.161:0/1055813705 <== mon.3 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e366653d-95f2-4573-8eee-ccdd82879a6f", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f72b8010d20 con 0x7f72c0132330 2024-04-19T00:44:23.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e366653d-95f2-4573-8eee-ccdd82879a6f", "id": 0}] 2024-04-19T00:44:23.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72b6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:23.168 INFO:teuthology.orchestra.run.smithi161.stdout:0 2024-04-19T00:44:23.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 10 librados: watch_flush enter 2024-04-19T00:44:23.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 10 librados: watch_flush exit 2024-04-19T00:44:23.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 20 client.4116.objecter shutdown clearing up homeless session... 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 10 client.4116.objecter successfully canceled tick 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 10 monclient: shutdown 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 msgr2=0x7f72c01cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 0x7f72c01cda50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72b800c440 tx=0x7f72b800c900 comp rx=0 tx=0).stop 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 shutdown_connections 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f72c0132330 0x7f72c01cda50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f72c00a3490 0x7f72c01cd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 --2- 172.21.15.161:0/1055813705 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f72c009f9f0 0x7f72c01ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 >> 172.21.15.161:0/1055813705 conn(0x7f72c00a4040 msgr2=0x7f72c0096510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 shutdown_connections 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 -- 172.21.15.161:0/1055813705 wait complete. 2024-04-19T00:44:23.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.165+0000 7f72c8786640 1 librados: shutdown 2024-04-19T00:44:23.186 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 3efcb199-3b20-4ce2-bd9e-9457316d298e 1 2024-04-19T00:44:23.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient: get_monmap_and_config 2024-04-19T00:44:23.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient: monmap: 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:44:23.302038+0000 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:44:23.302038+0000 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:23.301 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:23.302 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 Processor -- start 2024-04-19T00:44:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 -- start start 2024-04-19T00:44:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient: init 2024-04-19T00:44:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient: _add_conns ranks=[5,2,10,9,6,8,7,0,3,1,4] 2024-04-19T00:44:23.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 0x7f0a64135c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): picked mon.noname-h con 0x7f0a64072a30 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a64072e10 0x7f0a64073270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): picked mon.noname-b con 0x7f0a64072e10 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a640a4950 0x7f0a640a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): picked mon.noname-f con 0x7f0a640a4950 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0a64136160 con 0x7f0a640a4950 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0a6406f860 con 0x7f0a64072e10 2024-04-19T00:44:23.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a6406e660 con 0x7f0a64072a30 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6acb0640 10 monclient(hunting): authenticate will time out at 966.247650s 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 0x7f0a64135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6924d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a640a4950 0x7f0a640a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 0x7f0a64135c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.161:53768/0 (socket says 172.21.15.161:53768) 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 1 -- 172.21.15.161:0/60328454 learned_addr learned my addr 172.21.15.161:0/60328454 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 10 monclient(hunting): get_auth_request con 0x7f0a64072a30 auth_method 0 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6924d640 10 monclient(hunting): get_auth_request con 0x7f0a640a4950 auth_method 0 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6924d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6924d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a6924d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.297+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6924d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6924d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6924d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a63fff640 1 --2- 172.21.15.161:0/60328454 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a64072e10 0x7f0a64073270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a63fff640 10 monclient(hunting): get_auth_request con 0x7f0a64072e10 auth_method 0 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a63fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 -- 172.21.15.161:0/60328454 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a64072e10 msgr2=0x7f0a64073270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 --2- 172.21.15.161:0/60328454 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a64072e10 0x7f0a64073270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 -- 172.21.15.161:0/60328454 >> v2:172.21.15.161:3300/0 conn(0x7f0a640a4950 msgr2=0x7f0a640a4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 --2- 172.21.15.161:0/60328454 >> v2:172.21.15.161:3300/0 conn(0x7f0a640a4950 0x7f0a640a4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 monclient: found mon.noname-h 2024-04-19T00:44:23.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:44:23.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 -- 172.21.15.161:0/60328454 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6406f480 con 0x7f0a64072a30 2024-04-19T00:44:23.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6924d640 1 --2- 172.21.15.161:0/60328454 >> v2:172.21.15.161:3300/0 conn(0x7f0a640a4950 0x7f0a640a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a68a4c640 1 --2- 172.21.15.161:0/60328454 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 0x7f0a64135c20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a580098a0 tx=0x7f0a5801c400 comp rx=0 tx=0).ready entity=mon.5 client_cookie=2162b524a93aaf98 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 1 -- 172.21.15.161:0/60328454 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a580b0070 con 0x7f0a64072a30 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: dump: 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.317 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.318 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6acb0640 5 monclient: authenticate success, global_id 4109 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 1 -- 172.21.15.161:0/60328454 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a580aa810 con 0x7f0a64072a30 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6acb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6acb0640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a6acb0640 4 set_mon_vals no callback set 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 1 -- 172.21.15.161:0/60328454 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a580a9300 con 0x7f0a64072a30 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.301+0000 7f0a637fe640 10 monclient: dump: 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.320 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: shutdown 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/60328454 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 msgr2=0x7f0a64135c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/60328454 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 0x7f0a64135c20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a580098a0 tx=0x7f0a5801c400 comp rx=0 tx=0).stop 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/60328454 shutdown_connections 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/60328454 >> v2:172.21.15.161:3300/0 conn(0x7f0a640a4950 0x7f0a640a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/60328454 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a64072e10 0x7f0a64073270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/60328454 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072a30 0x7f0a64135c20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/60328454 >> 172.21.15.161:0/60328454 conn(0x7f0a640a9770 msgr2=0x7f0a640a5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/60328454 shutdown_connections 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/60328454 wait complete. 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: monmap: 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.323 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 librados: starting msgr at 2024-04-19T00:44:23.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 librados: starting objecter 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 Processor -- start 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- start start 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 librados: setting wanted keys 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 librados: calling monclient init 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: init 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: _add_conns ranks=[1,5,6,0,3,8,4,2,7] 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 0x7f0a641d0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): picked mon.noname-f con 0x7f0a64072a30 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072e10 0x7f0a6413f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): picked mon.noname-h con 0x7f0a64072e10 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a640a4950 0x7f0a6413f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): picked mon.noname-d con 0x7f0a640a4950 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a64142970 con 0x7f0a64072a30 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a641427f0 con 0x7f0a64072e10 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0a64142af0 con 0x7f0a640a4950 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient(hunting): authenticate will time out at 966.254017s 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 0x7f0a641d0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072e10 0x7f0a6413f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 0x7f0a641d0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35794/0 (socket says 172.21.15.161:35794) 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 -- 172.21.15.161:0/2826487506 learned_addr learned my addr 172.21.15.161:0/2826487506 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): get_auth_request con 0x7f0a64072a30 auth_method 0 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): get_auth_request con 0x7f0a64072e10 auth_method 0 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6924d640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a640a4950 0x7f0a6413f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 -- 172.21.15.161:0/2826487506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a640a4950 msgr2=0x7f0a6413f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a640a4950 0x7f0a6413f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 -- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072e10 msgr2=0x7f0a6413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072e10 0x7f0a6413f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 monclient: found mon.noname-f 2024-04-19T00:44:23.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 -- 172.21.15.161:0/2826487506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6413c980 con 0x7f0a64072a30 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6924d640 10 monclient: get_auth_request con 0x7f0a640a4950 auth_method 0 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6924d640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a640a4950 0x7f0a6413f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a63fff640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072e10 0x7f0a6413f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a68a4c640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 0x7f0a641d0d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0a580ae040 tx=0x7f0a580191a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 client.?.objecter ms_handle_connect 0x7f0a64072a30 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a580b0040 con 0x7f0a64072a30 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: dump: 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.330 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 5 monclient: authenticate success, global_id 4123 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a580044b0 con 0x7f0a64072a30 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a580aacf0 con 0x7f0a64072a30 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a62ffd640 4 set_mon_vals no callback set 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 monclient: dump: 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.332 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: _renew_subs 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6406e890 con 0x7f0a64072a30 2024-04-19T00:44:23.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: _renew_subs 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a641d14b0 con 0x7f0a64072a30 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a6acb0640 1 librados: init done 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0a58003620 con 0x7f0a64072a30 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f0a580bda00 con 0x7f0a64072a30 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 10 client.4123.objecter ms_dispatch 0x7f0a64000ba0 osd_map(3..3 src has 1..3) 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.305+0000 7f0a617fa640 20 client.4123.objecter dump_active .. 0 homeless 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a6acb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a6acb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a6acb0640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:44:23.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a640693b0 con 0x7f0a64072a30 2024-04-19T00:44:23.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0a58016110 con 0x7f0a64072a30 2024-04-19T00:44:23.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.309+0000 7f0a617fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:23.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.381+0000 7f0a6acb0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3efcb199-3b20-4ce2-bd9e-9457316d298e", "id": 1}] 2024-04-19T00:44:23.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.381+0000 7f0a6acb0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3efcb199-3b20-4ce2-bd9e-9457316d298e", "id": 1}] 2024-04-19T00:44:23.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.381+0000 7f0a6acb0640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:44:23.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.381+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3efcb199-3b20-4ce2-bd9e-9457316d298e", "id": 1} v 0) -- 0x7f0a6413a690 con 0x7f0a64072a30 2024-04-19T00:44:23.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a617fa640 1 -- 172.21.15.161:0/2826487506 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3efcb199-3b20-4ce2-bd9e-9457316d298e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f0a580bdc00 con 0x7f0a64072a30 2024-04-19T00:44:23.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3efcb199-3b20-4ce2-bd9e-9457316d298e", "id": 1}] 2024-04-19T00:44:23.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a617fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:23.393 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 10 librados: watch_flush enter 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 10 librados: watch_flush exit 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 20 client.4123.objecter shutdown clearing up homeless session... 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 10 client.4123.objecter successfully canceled tick 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 10 monclient: shutdown 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 msgr2=0x7f0a641d0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 0x7f0a641d0d90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0a580ae040 tx=0x7f0a580191a0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 shutdown_connections 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a640a4950 0x7f0a6413f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a64072e10 0x7f0a6413f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 --2- 172.21.15.161:0/2826487506 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a64072a30 0x7f0a641d0d90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 >> 172.21.15.161:0/2826487506 conn(0x7f0a640a9770 msgr2=0x7f0a640a6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 shutdown_connections 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 -- 172.21.15.161:0/2826487506 wait complete. 2024-04-19T00:44:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.389+0000 7f0a6acb0640 1 librados: shutdown 2024-04-19T00:44:23.410 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 08524c86-3cef-40d9-b1b6-89660fd4a0e6 2 2024-04-19T00:44:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient: get_monmap_and_config 2024-04-19T00:44:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient: monmap: 2024-04-19T00:44:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:44:23.530178+0000 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:44:23.530178+0000 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:23.530 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 Processor -- start 2024-04-19T00:44:23.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 -- start start 2024-04-19T00:44:23.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient: init 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient: _add_conns ranks=[8,7,2,4,10,5,1,3,9,0,6] 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc0a1ed0 0x7f99fc0a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): picked mon.noname-e con 0x7f99fc0a1ed0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 0x7f99fc0a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): picked mon.noname-i con 0x7f99fc09ecb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f99fc0a4350 0x7f99fc13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): picked mon.noname-b con 0x7f99fc0a4350 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f99fc06c710 con 0x7f99fc0a4350 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f99fc09f430 con 0x7f99fc09ecb0 2024-04-19T00:44:23.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f99fc0a0260 con 0x7f99fc0a1ed0 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f9a01a6e640 10 monclient(hunting): authenticate will time out at 966.475419s 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 0x7f99fc0a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 0x7f99fc0a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.161:43440/0 (socket says 172.21.15.161:43440) 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 -- 172.21.15.161:0/199490852 learned_addr learned my addr 172.21.15.161:0/199490852 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): get_auth_request con 0x7f99fc09ecb0 auth_method 0 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fb7fe640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc0a1ed0 0x7f99fc0a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fbfff640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f99fc0a4350 0x7f99fc13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fb7fe640 10 monclient(hunting): get_auth_request con 0x7f99fc0a1ed0 auth_method 0 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fbfff640 10 monclient(hunting): get_auth_request con 0x7f99fc0a4350 auth_method 0 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 -- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc0a1ed0 msgr2=0x7f99fc0a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc0a1ed0 0x7f99fc0a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 -- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f99fc0a4350 msgr2=0x7f99fc13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f99fc0a4350 0x7f99fc13fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 monclient: found mon.noname-i 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.525+0000 7f99faffd640 1 -- 172.21.15.161:0/199490852 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc0a0540 con 0x7f99fc09ecb0 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99faffd640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 0x7f99fc0a3e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098d0 tx=0x7f99f001c440 comp rx=0 tx=0).ready entity=mon.7 client_cookie=7b5334af3a04cffa server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 1 -- 172.21.15.161:0/199490852 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99f00b0070 con 0x7f99fc09ecb0 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: dump: 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.543 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.544 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f9a01a6e640 5 monclient: authenticate success, global_id 4111 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f9a01a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 1 -- 172.21.15.161:0/199490852 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f00aa810 con 0x7f99fc09ecb0 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f9a01a6e640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f9a01a6e640 4 set_mon_vals no callback set 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 1 -- 172.21.15.161:0/199490852 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99f00a9300 con 0x7f99fc09ecb0 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.529+0000 7f99fa7fc640 10 monclient: dump: 2024-04-19T00:44:23.545 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.546 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: shutdown 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/199490852 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 msgr2=0x7f99fc0a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 0x7f99fc0a3e10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098d0 tx=0x7f99f001c440 comp rx=0 tx=0).stop 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/199490852 shutdown_connections 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f99fc0a4350 0x7f99fc13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f99fc09ecb0 0x7f99fc0a3e10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/199490852 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc0a1ed0 0x7f99fc0a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/199490852 >> 172.21.15.161:0/199490852 conn(0x7f99fc0a8840 msgr2=0x7f99fc0a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/199490852 shutdown_connections 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/199490852 wait complete. 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: monmap: 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.548 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 librados: starting msgr at 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 librados: starting objecter 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 Processor -- start 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- start start 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 librados: setting wanted keys 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 librados: calling monclient init 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: init 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _add_conns ranks=[8,6,3,2,4,5,7,1,0] 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc09ecb0 0x7f99fc13ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): picked mon.noname-e con 0x7f99fc09ecb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f99fc0a1ed0 0x7f99fc139be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): picked mon.noname-d con 0x7f99fc0a1ed0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 0x7f99fc122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): picked mon.noname-g con 0x7f99fc0a4350 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f99fc1429a0 con 0x7f99fc0a4350 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f99fc142820 con 0x7f99fc0a1ed0 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f99fc142b20 con 0x7f99fc09ecb0 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient(hunting): authenticate will time out at 966.481683s 2024-04-19T00:44:23.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 0x7f99fc122920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 0x7f99fc122920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:34230/0 (socket says 172.21.15.161:34230) 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 -- 172.21.15.161:0/4175499494 learned_addr learned my addr 172.21.15.161:0/4175499494 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): get_auth_request con 0x7f99fc0a4350 auth_method 0 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fb7fe640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc09ecb0 0x7f99fc13ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fb7fe640 10 monclient(hunting): get_auth_request con 0x7f99fc09ecb0 auth_method 0 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99faffd640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f99fc0a1ed0 0x7f99fc139be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99faffd640 10 monclient(hunting): get_auth_request con 0x7f99fc0a1ed0 auth_method 0 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99faffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99faffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient(hunting): handle_auth_done global_id 4125 payload 824 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 -- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc09ecb0 msgr2=0x7f99fc13ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc09ecb0 0x7f99fc13ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 -- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f99fc0a1ed0 msgr2=0x7f99fc139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f99fc0a1ed0 0x7f99fc139be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 monclient: found mon.noname-g 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 -- 172.21.15.161:0/4175499494 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99fc1252a0 con 0x7f99fc0a4350 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fb7fe640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc09ecb0 0x7f99fc13ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99fbfff640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 0x7f99fc122920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009fd0 tx=0x7f99ec00e4b0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 client.?.objecter ms_handle_connect 0x7f99fc0a4350 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99ec00edd0 con 0x7f99fc0a4350 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: dump: 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.553 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99ec019070 con 0x7f99fc0a4350 2024-04-19T00:44:23.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99ec014d00 con 0x7f99fc0a4350 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 5 monclient: authenticate success, global_id 4125 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _renew_subs 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99fc124030 con 0x7f99fc0a4350 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99f9ffb640 4 set_mon_vals no callback set 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 monclient: dump: 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.555 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:23.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _renew_subs 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f99fc13f950 con 0x7f99fc0a4350 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 librados: init done 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f99ec014420 con 0x7f99fc0a4350 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f99ec01e460 con 0x7f99fc0a4350 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 10 client.4125.objecter ms_dispatch 0x7f99fc000ba0 osd_map(4..4 src has 1..4) 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 3 client.4125.objecter handle_osd_map got epochs [4,4] > 0 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 3 client.4125.objecter handle_osd_map decoding full epoch 4 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f99e3fff640 20 client.4125.objecter dump_active .. 0 homeless 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.533+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99fc06cc30 con 0x7f99fc0a4350 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.537+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f99ec014710 con 0x7f99fc0a4350 2024-04-19T00:44:23.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.537+0000 7f99e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.537+0000 7f99e3fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:23.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.605+0000 7f9a01a6e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "08524c86-3cef-40d9-b1b6-89660fd4a0e6", "id": 2}] 2024-04-19T00:44:23.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.605+0000 7f9a01a6e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "08524c86-3cef-40d9-b1b6-89660fd4a0e6", "id": 2}] 2024-04-19T00:44:23.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.605+0000 7f9a01a6e640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.605+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "08524c86-3cef-40d9-b1b6-89660fd4a0e6", "id": 2} v 0) -- 0x7f99fc123df0 con 0x7f99fc0a4350 2024-04-19T00:44:23.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f99e3fff640 1 -- 172.21.15.161:0/4175499494 <== mon.3 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08524c86-3cef-40d9-b1b6-89660fd4a0e6", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f99ec0149b0 con 0x7f99fc0a4350 2024-04-19T00:44:23.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f99e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "08524c86-3cef-40d9-b1b6-89660fd4a0e6", "id": 2}] 2024-04-19T00:44:23.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f99e3fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:23.616 INFO:teuthology.orchestra.run.smithi161.stdout:2 2024-04-19T00:44:23.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 10 librados: watch_flush enter 2024-04-19T00:44:23.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 10 librados: watch_flush exit 2024-04-19T00:44:23.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 20 client.4125.objecter shutdown clearing up homeless session... 2024-04-19T00:44:23.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 10 client.4125.objecter successfully canceled tick 2024-04-19T00:44:23.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 10 monclient: shutdown 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 msgr2=0x7f99fc122920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 0x7f99fc122920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99ec009fd0 tx=0x7f99ec00e4b0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 shutdown_connections 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f99fc0a4350 0x7f99fc122920 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f99fc0a1ed0 0x7f99fc139be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 --2- 172.21.15.161:0/4175499494 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f99fc09ecb0 0x7f99fc13ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 >> 172.21.15.161:0/4175499494 conn(0x7f99fc0a8840 msgr2=0x7f99fc13b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 shutdown_connections 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 -- 172.21.15.161:0/4175499494 wait complete. 2024-04-19T00:44:23.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.613+0000 7f9a01a6e640 1 librados: shutdown 2024-04-19T00:44:23.632 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 9b1f698f-5881-4edf-b694-3025af87e7dc 3 2024-04-19T00:44:23.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient: get_monmap_and_config 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient: monmap: 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:44:23.745498+0000 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:44:23.745498+0000 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.744 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:23.745 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 Processor -- start 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 -- start start 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient: init 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient: _add_conns ranks=[7,9,6,1,5,8,0,10,4,2,3] 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 0x7f06b41370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): picked mon.noname-i con 0x7f06b40a0b10 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06b40a0ef0 0x7f06b40a1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): picked mon.noname-a con 0x7f06b40a0ef0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06b40a2f50 0x7f06b40a3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): picked mon.noname-d con 0x7f06b40a2f50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f06b40a19c0 con 0x7f06b40a0ef0 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f06b409ed40 con 0x7f06b40a2f50 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f06b409f020 con 0x7f06b40a0b10 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06bbb2a640 10 monclient(hunting): authenticate will time out at 966.690695s 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 0x7f06b41370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 0x7f06b41370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.161:43454/0 (socket says 172.21.15.161:43454) 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 -- 172.21.15.161:0/190101313 learned_addr learned my addr 172.21.15.161:0/190101313 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): get_auth_request con 0x7f06b40a0b10 auth_method 0 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b90c5640 1 --2- 172.21.15.161:0/190101313 >> v2:172.21.15.43:3300/0 conn(0x7f06b40a0ef0 0x7f06b40a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06ba0c7640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06b40a2f50 0x7f06b40a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 -- 172.21.15.161:0/190101313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06b40a2f50 msgr2=0x7f06b40a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06b40a2f50 0x7f06b40a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 -- 172.21.15.161:0/190101313 >> v2:172.21.15.43:3300/0 conn(0x7f06b40a0ef0 msgr2=0x7f06b40a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 --2- 172.21.15.161:0/190101313 >> v2:172.21.15.43:3300/0 conn(0x7f06b40a0ef0 0x7f06b40a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 monclient: found mon.noname-i 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b98c6640 1 -- 172.21.15.161:0/190101313 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b40767a0 con 0x7f06b40a0b10 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b90c5640 10 monclient: get_auth_request con 0x7f06b40a0ef0 auth_method 0 2024-04-19T00:44:23.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06b90c5640 1 --2- 172.21.15.161:0/190101313 >> v2:172.21.15.43:3300/0 conn(0x7f06b40a0ef0 0x7f06b40a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06ba0c7640 10 monclient: get_auth_request con 0x7f06b40a2f50 auth_method 0 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.741+0000 7f06ba0c7640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06b40a2f50 0x7f06b40a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b98c6640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 0x7f06b41370a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009870 tx=0x7f06b001c420 comp rx=0 tx=0).ready entity=mon.7 client_cookie=f3fb044ff9c7fb86 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 1 -- 172.21.15.161:0/190101313 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06b00b0070 con 0x7f06b40a0b10 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: dump: 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.752 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 1 -- 172.21.15.161:0/190101313 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b00aa810 con 0x7f06b40a0b10 2024-04-19T00:44:23.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 1 -- 172.21.15.161:0/190101313 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06b00a9300 con 0x7f06b40a0b10 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 5 monclient: authenticate success, global_id 4120 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 4 set_mon_vals no callback set 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06b88c4640 10 monclient: dump: 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.754 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.755 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: shutdown 2024-04-19T00:44:23.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- 172.21.15.161:0/190101313 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 msgr2=0x7f06b41370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 0x7f06b41370a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f06b0009870 tx=0x7f06b001c420 comp rx=0 tx=0).stop 2024-04-19T00:44:23.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- 172.21.15.161:0/190101313 shutdown_connections 2024-04-19T00:44:23.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06b40a2f50 0x7f06b40a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/190101313 >> v2:172.21.15.43:3300/0 conn(0x7f06b40a0ef0 0x7f06b40a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/190101313 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a0b10 0x7f06b41370a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- 172.21.15.161:0/190101313 >> 172.21.15.161:0/190101313 conn(0x7f06b40a4040 msgr2=0x7f06b40a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- 172.21.15.161:0/190101313 shutdown_connections 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- 172.21.15.161:0/190101313 wait complete. 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: monmap: 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.761 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 librados: starting msgr at 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 librados: starting objecter 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 Processor -- start 2024-04-19T00:44:23.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- start start 2024-04-19T00:44:23.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 librados: setting wanted keys 2024-04-19T00:44:23.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 librados: calling monclient init 2024-04-19T00:44:23.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: init 2024-04-19T00:44:23.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient: _add_conns ranks=[2,8,7,3,0,1,6,5,4] 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40a0b10 0x7f06b413d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): picked mon.noname-b con 0x7f06b40a0b10 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f06b40a0ef0 0x7f06b409b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): picked mon.noname-e con 0x7f06b40a0ef0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 0x7f06b409ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): picked mon.noname-i con 0x7f06b40a2f50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f06b4140860 con 0x7f06b40a0b10 2024-04-19T00:44:23.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f06b41406e0 con 0x7f06b40a2f50 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f06b41409e0 con 0x7f06b40a0ef0 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06bbb2a640 10 monclient(hunting): authenticate will time out at 966.695745s 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 0x7f06b409ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 0x7f06b409ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.161:43470/0 (socket says 172.21.15.161:43470) 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 1 -- 172.21.15.161:0/2750044693 learned_addr learned my addr 172.21.15.161:0/2750044693 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 10 monclient(hunting): get_auth_request con 0x7f06b40a2f50 auth_method 0 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.745+0000 7f06ba0c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b90c5640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f06b40a0ef0 0x7f06b409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b98c6640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40a0b10 0x7f06b413d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b90c5640 10 monclient(hunting): get_auth_request con 0x7f06b40a0ef0 auth_method 0 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b90c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b90c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b90c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b98c6640 10 monclient(hunting): get_auth_request con 0x7f06b40a0b10 auth_method 0 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b98c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b98c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06b98c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-04-19T00:44:23.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 -- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f06b40a0ef0 msgr2=0x7f06b409b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f06b40a0ef0 0x7f06b409b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 -- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40a0b10 msgr2=0x7f06b413d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40a0b10 0x7f06b413d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 monclient: found mon.noname-i 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 -- 172.21.15.161:0/2750044693 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b413c890 con 0x7f06b40a2f50 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06ba0c7640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 0x7f06b409ba80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00cf20 tx=0x7f06ac00a7d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 client.?.objecter ms_handle_connect 0x7f06b40a2f50 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06ac01a070 con 0x7f06b40a2f50 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: dump: 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.768 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ac00dab0 con 0x7f06b40a2f50 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 5 monclient: authenticate success, global_id 4129 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 monclient: _renew_subs 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b409dd60 con 0x7f06b40a2f50 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06ac00dc60 con 0x7f06b40a2f50 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a7fff640 4 set_mon_vals no callback set 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: dump: 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.770 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 monclient: _renew_subs 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b413b100 con 0x7f06b40a2f50 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 1 librados: init done 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f06ac00d640 con 0x7f06b40a2f50 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b40693b0 con 0x7f06b40a2f50 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f06ac00de00 con 0x7f06b40a2f50 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 client.4129.objecter ms_dispatch 0x7f06b4000ba0 osd_map(5..5 src has 1..5) 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 20 client.4129.objecter dump_active .. 0 homeless 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f06ac023070 con 0x7f06b40a2f50 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.749+0000 7f06a67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:23.773 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:23.757+0000 7f2a53c28180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-19T00:44:23.817 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:23.813+0000 7f2a53c28180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-19T00:44:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.821+0000 7f06bbb2a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9b1f698f-5881-4edf-b694-3025af87e7dc", "id": 3}] 2024-04-19T00:44:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.821+0000 7f06bbb2a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9b1f698f-5881-4edf-b694-3025af87e7dc", "id": 3}] 2024-04-19T00:44:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.821+0000 7f06bbb2a640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:44:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.821+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_command({"prefix": "osd new", "uuid": "9b1f698f-5881-4edf-b694-3025af87e7dc", "id": 3} v 0) -- 0x7f06b40a71d0 con 0x7f06b40a2f50 2024-04-19T00:44:23.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.825+0000 7f06a67fc640 1 -- 172.21.15.161:0/2750044693 <== mon.7 v2:172.21.15.161:3303/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b1f698f-5881-4edf-b694-3025af87e7dc", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f06ac015440 con 0x7f06b40a2f50 2024-04-19T00:44:23.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.825+0000 7f06a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9b1f698f-5881-4edf-b694-3025af87e7dc", "id": 3}] 2024-04-19T00:44:23.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.825+0000 7f06a67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:23.830 INFO:teuthology.orchestra.run.smithi161.stdout:3 2024-04-19T00:44:23.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 10 librados: watch_flush enter 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 10 librados: watch_flush exit 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 20 client.4129.objecter shutdown clearing up homeless session... 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 10 client.4129.objecter successfully canceled tick 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 10 monclient: shutdown 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 msgr2=0x7f06b409ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 0x7f06b409ba80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00cf20 tx=0x7f06ac00a7d0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 shutdown_connections 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06b40a2f50 0x7f06b409ba80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f06b40a0ef0 0x7f06b409b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 --2- 172.21.15.161:0/2750044693 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f06b40a0b10 0x7f06b413d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 >> 172.21.15.161:0/2750044693 conn(0x7f06b40a4040 msgr2=0x7f06b4134780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 shutdown_connections 2024-04-19T00:44:23.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 -- 172.21.15.161:0/2750044693 wait complete. 2024-04-19T00:44:23.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.829+0000 7f06bbb2a640 1 librados: shutdown 2024-04-19T00:44:23.846 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a 4 2024-04-19T00:44:23.875 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:23.873+0000 7f2a53c28180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-19T00:44:23.954 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:23.953+0000 7f2a53c28180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-19T00:44:23.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient: get_monmap_and_config 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient: monmap: 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:44:23.965645+0000 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:44:23.965645+0000 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:23.964 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:23.965 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 Processor -- start 2024-04-19T00:44:23.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 -- start start 2024-04-19T00:44:23.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient: init 2024-04-19T00:44:23.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient: _add_conns ranks=[8,2,5,0,7,9,1,4,3,10,6] 2024-04-19T00:44:23.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8c70072950 0x7f8c70072d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): picked mon.noname-e con 0x7f8c70072950 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c700a32d0 0x7f8c70138bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): picked mon.noname-b con 0x7f8c700a32d0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 0x7f8c70139100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): picked mon.noname-h con 0x7f8c700a36f0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8c700a0ab0 con 0x7f8c700a32d0 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8c700a16a0 con 0x7f8c700a36f0 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8c700a1980 con 0x7f8c70072950 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c77b86640 10 monclient(hunting): authenticate will time out at 966.910747s 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 0x7f8c70139100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 0x7f8c70139100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.161:53792/0 (socket says 172.21.15.161:53792) 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 -- 172.21.15.161:0/2204504176 learned_addr learned my addr 172.21.15.161:0/2204504176 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): get_auth_request con 0x7f8c700a36f0 auth_method 0 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75121640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c700a32d0 0x7f8c70138bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75922640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8c70072950 0x7f8c70072d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75121640 10 monclient(hunting): get_auth_request con 0x7f8c700a32d0 auth_method 0 2024-04-19T00:44:23.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75121640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75121640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75922640 10 monclient(hunting): get_auth_request con 0x7f8c70072950 auth_method 0 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75922640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75922640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c75922640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient(hunting): handle_auth_done global_id 4127 payload 274 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 -- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8c70072950 msgr2=0x7f8c70072d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8c70072950 0x7f8c70072d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 -- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c700a32d0 msgr2=0x7f8c70138bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c700a32d0 0x7f8c70138bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 monclient: found mon.noname-h 2024-04-19T00:44:23.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.961+0000 7f8c76123640 1 -- 172.21.15.161:0/2204504176 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c700a1c60 con 0x7f8c700a36f0 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c76123640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 0x7f8c70139100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009e80 tx=0x7f8c6801c910 comp rx=0 tx=0).ready entity=mon.5 client_cookie=2d00d966f17f3e3c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 1 -- 172.21.15.161:0/2204504176 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c680090d0 con 0x7f8c700a36f0 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: dump: 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.977 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 1 -- 172.21.15.161:0/2204504176 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c680a5a70 con 0x7f8c700a36f0 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 5 monclient: authenticate success, global_id 4127 2024-04-19T00:44:23.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 1 -- 172.21.15.161:0/2204504176 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c680a5c60 con 0x7f8c700a36f0 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 4 set_mon_vals no callback set 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c74920640 10 monclient: dump: 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.979 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: shutdown 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- 172.21.15.161:0/2204504176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 msgr2=0x7f8c70139100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 0x7f8c70139100 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c68009e80 tx=0x7f8c6801c910 comp rx=0 tx=0).stop 2024-04-19T00:44:23.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- 172.21.15.161:0/2204504176 shutdown_connections 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c700a36f0 0x7f8c70139100 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c700a32d0 0x7f8c70138bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- 172.21.15.161:0/2204504176 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8c70072950 0x7f8c70072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- 172.21.15.161:0/2204504176 >> 172.21.15.161:0/2204504176 conn(0x7f8c700a7f90 msgr2=0x7f8c700a4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- 172.21.15.161:0/2204504176 shutdown_connections 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- 172.21.15.161:0/2204504176 wait complete. 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: build_initial_monmap 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: monmap: 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-04-19T00:44:23.981 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 librados: starting msgr at 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 librados: starting objecter 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 Processor -- start 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- start start 2024-04-19T00:44:23.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 librados: setting wanted keys 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 librados: calling monclient init 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: init 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient: _add_conns ranks=[1,3,4,2,0,6,7,8,5] 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 0x7f8c70128f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): picked mon.noname-f con 0x7f8c70072950 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 0x7f8c70129490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): picked mon.noname-g con 0x7f8c700a32d0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8c700a36f0 0x7f8c701299d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): picked mon.noname-c con 0x7f8c700a36f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c70142ae0 con 0x7f8c70072950 2024-04-19T00:44:23.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8c70142960 con 0x7f8c700a32d0 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8c70142c60 con 0x7f8c700a36f0 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c77b86640 10 monclient(hunting): authenticate will time out at 966.915689s 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 0x7f8c70129490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 0x7f8c70128f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 0x7f8c70128f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35800/0 (socket says 172.21.15.161:35800) 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 0x7f8c70129490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:34238/0 (socket says 172.21.15.161:34238) 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 1 -- 172.21.15.161:0/3501951020 learned_addr learned my addr 172.21.15.161:0/3501951020 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): get_auth_request con 0x7f8c70072950 auth_method 0 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): get_auth_request con 0x7f8c700a32d0 auth_method 0 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.965+0000 7f8c75922640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c76123640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8c700a36f0 0x7f8c701299d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c76123640 10 monclient(hunting): get_auth_request con 0x7f8c700a36f0 auth_method 0 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c76123640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c76123640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:23.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c76123640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 10 monclient(hunting): handle_auth_done global_id 4134 payload 824 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 -- 172.21.15.161:0/3501951020 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8c700a36f0 msgr2=0x7f8c701299d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8c700a36f0 0x7f8c701299d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 -- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 msgr2=0x7f8c70128f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 0x7f8c70128f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 10 monclient: _finish_hunting 0 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 monclient: found mon.noname-g 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 -- 172.21.15.161:0/3501951020 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c70135420 con 0x7f8c700a32d0 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75922640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 0x7f8c70128f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c75121640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 0x7f8c70129490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000d5d0 tx=0x7f8c6000da90 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:23.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 client.?.objecter ms_handle_connect 0x7f8c700a32d0 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c60004240 con 0x7f8c700a32d0 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: dump: 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.987 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c60004400 con 0x7f8c700a32d0 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 5 monclient: authenticate success, global_id 4134 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 monclient: _renew_subs 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c700a0ec0 con 0x7f8c700a32d0 2024-04-19T00:44:23.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c60012420 con 0x7f8c700a32d0 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c67fff640 4 set_mon_vals no callback set 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: dump: 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:23.989 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 monclient: _renew_subs 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c701356c0 con 0x7f8c700a32d0 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 1 librados: init done 2024-04-19T00:44:23.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8c60003620 con 0x7f8c700a32d0 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f8c6000f040 con 0x7f8c700a32d0 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 client.4134.objecter ms_dispatch 0x7f8c70000ba0 osd_map(6..6 src has 1..6) 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 3 client.4134.objecter handle_osd_map got epochs [6,6] > 0 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 3 client.4134.objecter handle_osd_map decoding full epoch 6 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 20 client.4134.objecter dump_active .. 0 homeless 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:23.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c700693b0 con 0x7f8c700a32d0 2024-04-19T00:44:23.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8c600125c0 con 0x7f8c700a32d0 2024-04-19T00:44:23.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:23.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:23.969+0000 7f8c667fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:24.009 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:24.005+0000 7f2a53c28180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-19T00:44:24.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.037+0000 7f8c77b86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a", "id": 4}] 2024-04-19T00:44:24.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.037+0000 7f8c77b86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a", "id": 4}] 2024-04-19T00:44:24.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.037+0000 7f8c77b86640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:24.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.037+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a", "id": 4} v 0) -- 0x7f8c7006d920 con 0x7f8c700a32d0 2024-04-19T00:44:24.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.045+0000 7f8c667fc640 1 -- 172.21.15.161:0/3501951020 <== mon.3 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f8c60012b10 con 0x7f8c700a32d0 2024-04-19T00:44:24.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.045+0000 7f8c667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a", "id": 4}] 2024-04-19T00:44:24.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.045+0000 7f8c667fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:24.050 INFO:teuthology.orchestra.run.smithi161.stdout:4 2024-04-19T00:44:24.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 10 librados: watch_flush enter 2024-04-19T00:44:24.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 10 librados: watch_flush exit 2024-04-19T00:44:24.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 20 client.4134.objecter shutdown clearing up homeless session... 2024-04-19T00:44:24.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 10 client.4134.objecter successfully canceled tick 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 10 monclient: shutdown 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 msgr2=0x7f8c70129490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 0x7f8c70129490 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8c6000d5d0 tx=0x7f8c6000da90 comp rx=0 tx=0).stop 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 shutdown_connections 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8c700a36f0 0x7f8c701299d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c700a32d0 0x7f8c70129490 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 --2- 172.21.15.161:0/3501951020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c70072950 0x7f8c70128f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 >> 172.21.15.161:0/3501951020 conn(0x7f8c700a7f90 msgr2=0x7f8c7012f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 shutdown_connections 2024-04-19T00:44:24.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 -- 172.21.15.161:0/3501951020 wait complete. 2024-04-19T00:44:24.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.049+0000 7f8c77b86640 1 librados: shutdown 2024-04-19T00:44:24.067 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 3d8e2a0e-809f-460b-a629-1aaa317f38b9 5 2024-04-19T00:44:24.091 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:24.085+0000 7f2a53c28180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-19T00:44:24.143 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:24.141+0000 7f2a53c28180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-19T00:44:24.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient: get_monmap_and_config 2024-04-19T00:44:24.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient: build_initial_monmap 2024-04-19T00:44:24.204 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient: monmap: 2024-04-19T00:44:24.204 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:24.204 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:44:24.205944+0000 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:44:24.205944+0000 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:24.205 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:24.206 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:24.206 INFO:teuthology.orchestra.run.smithi161.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:24.206 INFO:teuthology.orchestra.run.smithi161.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:24.206 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 Processor -- start 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 -- start start 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient: init 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient: _add_conns ranks=[7,2,6,1,8,3,10,4,5,9,0] 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 0x7fd6000a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): picked mon.noname-i con 0x7fd6000a20d0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd6000a3f60 0x7fd6000a4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): picked mon.noname-b con 0x7fd6000a3f60 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd60013dc00 0x7fd60013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): picked mon.noname-d con 0x7fd60013dc00 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd6000a0b80 con 0x7fd6000a3f60 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd6000a48a0 con 0x7fd60013dc00 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd60006c140 con 0x7fd6000a20d0 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd607da7640 10 monclient(hunting): authenticate will time out at 967.151113s 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 0x7fd6000a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 0x7fd6000a3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.161:43484/0 (socket says 172.21.15.161:43484) 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 -- 172.21.15.161:0/3296626855 learned_addr learned my addr 172.21.15.161:0/3296626855 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:24.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): get_auth_request con 0x7fd6000a20d0 auth_method 0 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605342640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd6000a3f60 0x7fd6000a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd606344640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd60013dc00 0x7fd60013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 -- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd60013dc00 msgr2=0x7fd60013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd60013dc00 0x7fd60013fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 -- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd6000a3f60 msgr2=0x7fd6000a4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd6000a3f60 0x7fd6000a4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient: _finish_hunting 0 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 monclient: found mon.noname-i 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605b43640 1 -- 172.21.15.161:0/3296626855 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6000a0250 con 0x7fd6000a20d0 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605342640 10 monclient: get_auth_request con 0x7fd6000a3f60 auth_method 0 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.201+0000 7fd605342640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd6000a3f60 0x7fd6000a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd605b43640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 0x7fd6000a3a20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0098d0 tx=0x7fd5fc01c440 comp rx=0 tx=0).ready entity=mon.7 client_cookie=2583ba42c0648880 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 1 -- 172.21.15.161:0/3296626855 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5fc0b0070 con 0x7fd6000a20d0 2024-04-19T00:44:24.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: dump: 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.218 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: _finish_auth 0 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: _check_auth_tickets 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:24.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 1 -- 172.21.15.161:0/3296626855 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc0aa810 con 0x7fd6000a20d0 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 5 monclient: authenticate success, global_id 4138 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 1 -- 172.21.15.161:0/3296626855 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5fc0a9300 con 0x7fd6000a20d0 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd604b41640 10 monclient: dump: 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.220 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.221 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:24.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:24.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 4 set_mon_vals no callback set 2024-04-19T00:44:24.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 10 monclient: shutdown 2024-04-19T00:44:24.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:24.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 -- 172.21.15.161:0/3296626855 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 msgr2=0x7fd6000a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 0x7fd6000a3a20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0098d0 tx=0x7fd5fc01c440 comp rx=0 tx=0).stop 2024-04-19T00:44:24.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 -- 172.21.15.161:0/3296626855 shutdown_connections 2024-04-19T00:44:24.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd60013dc00 0x7fd60013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd6000a3f60 0x7fd6000a4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 --2- 172.21.15.161:0/3296626855 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6000a20d0 0x7fd6000a3a20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 -- 172.21.15.161:0/3296626855 >> 172.21.15.161:0/3296626855 conn(0x7fd6000a5630 msgr2=0x7fd60006d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 -- 172.21.15.161:0/3296626855 shutdown_connections 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 -- 172.21.15.161:0/3296626855 wait complete. 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 10 monclient: build_initial_monmap 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 10 monclient: monmap: 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-04-19T00:44:24.224 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 librados: starting msgr at 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 librados: starting objecter 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 Processor -- start 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 -- start start 2024-04-19T00:44:24.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 librados: setting wanted keys 2024-04-19T00:44:24.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 1 librados: calling monclient init 2024-04-19T00:44:24.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.205+0000 7fd607da7640 10 monclient: init 2024-04-19T00:44:24.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:24.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _add_conns ranks=[3,1,2,4,5,7,8,0,6] 2024-04-19T00:44:24.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 0x7fd600133de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): picked mon.noname-g con 0x7fd6000a20d0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:24.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6000a3f60 0x7fd600134320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): picked mon.noname-f con 0x7fd6000a3f60 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:24.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd60013dc00 0x7fd60013c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): picked mon.noname-b con 0x7fd60013dc00 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd60006e640 con 0x7fd6000a3f60 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd6000a1520 con 0x7fd60013dc00 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd60009f510 con 0x7fd6000a20d0 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:24.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient(hunting): authenticate will time out at 967.156259s 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 0x7fd600133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 0x7fd600133de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:34244/0 (socket says 172.21.15.161:34244) 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 -- 172.21.15.161:0/540270984 learned_addr learned my addr 172.21.15.161:0/540270984 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): get_auth_request con 0x7fd6000a20d0 auth_method 0 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6000a3f60 0x7fd600134320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): get_auth_request con 0x7fd6000a3f60 auth_method 0 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605342640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd606344640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd60013dc00 0x7fd60013c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd606344640 10 monclient(hunting): get_auth_request con 0x7fd60013dc00 auth_method 0 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd606344640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd606344640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd606344640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient(hunting): handle_auth_done global_id 4143 payload 824 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 -- 172.21.15.161:0/540270984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd60013dc00 msgr2=0x7fd60013c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd60013dc00 0x7fd60013c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 -- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6000a3f60 msgr2=0x7fd600134320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6000a3f60 0x7fd600134320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient: _finish_hunting 0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 monclient: found mon.noname-g 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 -- 172.21.15.161:0/540270984 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60009da50 con 0x7fd6000a20d0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd605b43640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 0x7fd600133de0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0098d0 tx=0x7fd5fc0191e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 client.?.objecter ms_handle_connect 0x7fd6000a20d0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5fc0b9070 con 0x7fd6000a20d0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: dump: 2024-04-19T00:44:24.231 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.232 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 5 monclient: authenticate success, global_id 4143 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc0aae30 con 0x7fd6000a20d0 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _renew_subs 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60009dd30 con 0x7fd6000a20d0 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5fc0b0040 con 0x7fd6000a20d0 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:24.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 monclient: dump: 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.234 INFO:teuthology.orchestra.run.smithi161.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f7fff640 4 set_mon_vals no callback set 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _renew_subs 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd600073500 con 0x7fd6000a20d0 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 librados: init done 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd5fc004040 con 0x7fd6000a20d0 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fd5fc003cb0 con 0x7fd6000a20d0 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 10 client.4143.objecter ms_dispatch 0x7fd600000ba0 osd_map(7..7 src has 1..7) 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 3 client.4143.objecter handle_osd_map got epochs [7,7] > 0 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 3 client.4143.objecter handle_osd_map decoding full epoch 7 2024-04-19T00:44:24.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd5f67fc640 20 client.4143.objecter dump_active .. 0 homeless 2024-04-19T00:44:24.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:24.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:24.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:24.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.209+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6000693b0 con 0x7fd6000a20d0 2024-04-19T00:44:24.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.213+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd5fc016150 con 0x7fd6000a20d0 2024-04-19T00:44:24.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.213+0000 7fd5f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:24.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.213+0000 7fd5f67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:24.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.281+0000 7fd607da7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3d8e2a0e-809f-460b-a629-1aaa317f38b9", "id": 5}] 2024-04-19T00:44:24.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.281+0000 7fd607da7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3d8e2a0e-809f-460b-a629-1aaa317f38b9", "id": 5}] 2024-04-19T00:44:24.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.281+0000 7fd607da7640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:44:24.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.281+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3d8e2a0e-809f-460b-a629-1aaa317f38b9", "id": 5} v 0) -- 0x7fd60009bcb0 con 0x7fd6000a20d0 2024-04-19T00:44:24.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.289+0000 7fd5f67fc640 1 -- 172.21.15.161:0/540270984 <== mon.3 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3d8e2a0e-809f-460b-a629-1aaa317f38b9", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fd5fc003620 con 0x7fd6000a20d0 2024-04-19T00:44:24.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.289+0000 7fd5f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3d8e2a0e-809f-460b-a629-1aaa317f38b9", "id": 5}] 2024-04-19T00:44:24.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.289+0000 7fd5f67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:24.295 INFO:teuthology.orchestra.run.smithi161.stdout:5 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.289+0000 7fd607da7640 10 librados: watch_flush enter 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 10 librados: watch_flush exit 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 20 client.4143.objecter shutdown clearing up homeless session... 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 10 client.4143.objecter successfully canceled tick 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 10 monclient: shutdown 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 msgr2=0x7fd600133de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 0x7fd600133de0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0098d0 tx=0x7fd5fc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 shutdown_connections 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd60013dc00 0x7fd60013c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6000a3f60 0x7fd600134320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 --2- 172.21.15.161:0/540270984 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6000a20d0 0x7fd600133de0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 >> 172.21.15.161:0/540270984 conn(0x7fd6000a5630 msgr2=0x7fd60013a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:24.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 shutdown_connections 2024-04-19T00:44:24.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 -- 172.21.15.161:0/540270984 wait complete. 2024-04-19T00:44:24.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-04-19T00:44:24.293+0000 7fd607da7640 1 librados: shutdown 2024-04-19T00:44:24.298 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:24.297+0000 7f2a53c28180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-19T00:44:24.310 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-19T00:44:24.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-19T00:44:24.313 INFO:tasks.ceph.osd.0:Started 2024-04-19T00:44:24.313 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-19T00:44:24.313 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-19T00:44:24.316 INFO:tasks.ceph.osd.1:Started 2024-04-19T00:44:24.316 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-19T00:44:24.316 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-19T00:44:24.318 INFO:tasks.ceph.osd.2:Started 2024-04-19T00:44:24.318 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-19T00:44:24.318 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-19T00:44:24.321 INFO:tasks.ceph.osd.3:Started 2024-04-19T00:44:24.321 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-19T00:44:24.321 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-19T00:44:24.323 INFO:tasks.ceph.osd.4:Started 2024-04-19T00:44:24.323 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-19T00:44:24.323 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-19T00:44:24.325 INFO:tasks.ceph.osd.5:Started 2024-04-19T00:44:24.325 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T00:44:24.365 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:24.361+0000 7f2a53c28180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-19T00:44:24.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.440+0000 7f44a5621640 10 monclient: get_monmap_and_config 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.440+0000 7f44a5621640 10 monclient: build_initial_monmap 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.440+0000 7f44a5621640 10 monclient: monmap: 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:24.445527+0000 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:24.445527+0000 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:24.444 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:24.445 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 Processor -- start 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 -- start start 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient: init 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient: _add_conns ranks=[4,1,2,8,5,0,10,9,7,3,6] 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 0x7f44a00a3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): picked mon.noname-c con 0x7f44a00729f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f44a0072dd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 0x7f44a0136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): picked mon.noname-b con 0x7f44a00731f0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44a00a11a0 con 0x7f44a0072dd0 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f44a00a0100 con 0x7f44a00731f0 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f44a00a1480 con 0x7f44a00729f0 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient(hunting): authenticate will time out at 934.572876s 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 0x7f44a00a3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 0x7f44a0136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 0x7f44a0136fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35606/0 (socket says 172.21.15.43:35606) 2024-04-19T00:44:24.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 1 -- 172.21.15.43:0/1848160480 learned_addr learned my addr 172.21.15.43:0/1848160480 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): get_auth_request con 0x7f44a00731f0 auth_method 0 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 0x7f44a00a3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36298/0 (socket says 172.21.15.43:36298) 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): get_auth_request con 0x7f44a00729f0 auth_method 0 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 -- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 msgr2=0x7f44a0136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 --2- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 0x7f44a0136fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 -- 172.21.15.43:0/1848160480 >> v1:172.21.15.161:6789/0 conn(0x7f44a0072dd0 legacy=0x7f44a01368b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 monclient: found mon.noname-c 2024-04-19T00:44:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 -- 172.21.15.43:0/1848160480 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a00a38b0 con 0x7f44a00729f0 2024-04-19T00:44:24.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449f7fe640 1 --2- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 0x7f44a0136fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449effd640 1 --2- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 0x7f44a00a3370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4490009820 tx=0x7f449001c380 comp rx=0 tx=0).ready entity=mon.4 client_cookie=924a42b4c99c48ec server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 1 -- 172.21.15.43:0/1848160480 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f44900b0070 con 0x7f44a00729f0 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: dump: 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.457 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:24.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 1 -- 172.21.15.43:0/1848160480 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44900aa810 con 0x7f44a00729f0 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 1 -- 172.21.15.43:0/1848160480 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f44900a9300 con 0x7f44a00729f0 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f449dffb640 10 monclient: dump: 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:24.459 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 5 monclient: authenticate success, global_id 4117 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.444+0000 7f44a5621640 4 set_mon_vals no callback set 2024-04-19T00:44:24.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: shutdown 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 msgr2=0x7f44a00a3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 --2- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 0x7f44a00a3370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4490009820 tx=0x7f449001c380 comp rx=0 tx=0).stop 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/1848160480 shutdown_connections 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 --2- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f44a00731f0 0x7f44a0136fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 --2- 172.21.15.43:0/1848160480 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00729f0 0x7f44a00a3370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/1848160480 >> 172.21.15.43:0/1848160480 conn(0x7f44a00a4460 msgr2=0x7f44a00a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/1848160480 shutdown_connections 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/1848160480 wait complete. 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: build_initial_monmap 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: monmap: 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:24.461 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:24.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 librados: starting msgr at 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 librados: starting objecter 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 Processor -- start 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- start start 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 librados: setting wanted keys 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 librados: calling monclient init 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: init 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: _add_conns ranks=[6,0,4,7,5,1,2,8,3] 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 0x7f44a0128050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): picked mon.noname-d con 0x7f44a00729f0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a0072dd0 0x7f44a0128590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): picked mon.noname-a con 0x7f44a0072dd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00731f0 0x7f44a0128ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): picked mon.noname-c con 0x7f44a00731f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:24.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f44a01437f0 con 0x7f44a0072dd0 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f44a0137710 con 0x7f44a00731f0 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f44a0143970 con 0x7f44a00729f0 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient(hunting): authenticate will time out at 934.578084s 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 0x7f44a0128050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a0072dd0 0x7f44a0128590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 0x7f44a0128050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33366/0 (socket says 172.21.15.43:33366) 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 -- 172.21.15.43:0/3632922355 learned_addr learned my addr 172.21.15.43:0/3632922355 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): get_auth_request con 0x7f44a00729f0 auth_method 0 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00731f0 0x7f44a0128ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): get_auth_request con 0x7f44a0072dd0 auth_method 0 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): get_auth_request con 0x7f44a00731f0 auth_method 0 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient(hunting): handle_auth_done global_id 4173 payload 824 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 -- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00731f0 msgr2=0x7f44a0128ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00731f0 0x7f44a0128ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 -- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a0072dd0 msgr2=0x7f44a0128590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a0072dd0 0x7f44a0128590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 monclient: found mon.noname-d 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:24.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 -- 172.21.15.43:0/3632922355 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a00a0870 con 0x7f44a00729f0 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449effd640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 0x7f44a0128050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f44900ae040 tx=0x7f4490019120 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 client.?.objecter ms_handle_connect 0x7f44a00729f0 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f44900b9070 con 0x7f44a00729f0 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: dump: 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:24.467 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 5 monclient: authenticate success, global_id 4173 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44900044a0 con 0x7f44a00729f0 2024-04-19T00:44:24.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f44900b0040 con 0x7f44a00729f0 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f449d7fa640 4 set_mon_vals no callback set 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 monclient: dump: 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:24.469 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: _renew_subs 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a01300a0 con 0x7f44a00729f0 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: _renew_subs 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:24.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a0134cc0 con 0x7f44a00729f0 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44a5621640 1 librados: init done 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f44900aac90 con 0x7f44a00729f0 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f44900b4a20 con 0x7f44a00729f0 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 10 client.4173.objecter ms_dispatch 0x7f44a0000ba0 osd_map(8..8 src has 1..8) 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 3 client.4173.objecter handle_osd_map got epochs [8,8] > 0 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 3 client.4173.objecter handle_osd_map decoding full epoch 8 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.448+0000 7f44877fe640 20 client.4173.objecter dump_active .. 0 homeless 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.452+0000 7f44a5621640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.452+0000 7f44a5621640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.452+0000 7f44a5621640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:24.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.452+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a01306f0 con 0x7f44a00729f0 2024-04-19T00:44:24.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.464+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4490016090 con 0x7f44a00729f0 2024-04-19T00:44:24.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.464+0000 7f44877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:24.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.464+0000 7f44877fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:24.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:24.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:24.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:24.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44a006dab0 con 0x7f44a00729f0 2024-04-19T00:44:24.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44877fe640 1 -- 172.21.15.43:0/3632922355 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f44900aa7e0 con 0x7f44a00729f0 2024-04-19T00:44:24.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:24.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44877fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:24.527 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:24.528 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":8,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T00:44:24.286950+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 librados: watch_flush enter 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 librados: watch_flush exit 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 20 client.4173.objecter shutdown clearing up homeless session... 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 client.4173.objecter successfully canceled tick 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 10 monclient: shutdown 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 msgr2=0x7f44a0128050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 0x7f44a0128050 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f44900ae040 tx=0x7f4490019120 comp rx=0 tx=0).stop 2024-04-19T00:44:24.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 shutdown_connections 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f44a00731f0 0x7f44a0128ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f44a0072dd0 0x7f44a0128590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 --2- 172.21.15.43:0/3632922355 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f44a00729f0 0x7f44a0128050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 >> 172.21.15.43:0/3632922355 conn(0x7f44a00a4460 msgr2=0x7f44a012ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 shutdown_connections 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.524+0000 7f44a5621640 1 -- 172.21.15.43:0/3632922355 wait complete. 2024-04-19T00:44:24.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:24.528+0000 7f44a5621640 1 librados: shutdown 2024-04-19T00:44:24.545 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-19T00:44:24.545 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-19T00:44:24.936 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-04-19T00:44:24.933+0000 7f348fa597c0 -1 Falling back to public interface 2024-04-19T00:44:24.937 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-04-19T00:44:24.933+0000 7fab3ab317c0 -1 Falling back to public interface 2024-04-19T00:44:24.937 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-04-19T00:44:24.933+0000 7f5c43bde7c0 -1 Falling back to public interface 2024-04-19T00:44:24.943 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-04-19T00:44:24.940+0000 7fe8e1fca7c0 -1 Falling back to public interface 2024-04-19T00:44:24.943 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-04-19T00:44:24.940+0000 7f36b69787c0 -1 Falling back to public interface 2024-04-19T00:44:24.944 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-04-19T00:44:24.940+0000 7fa7d6a207c0 -1 Falling back to public interface 2024-04-19T00:44:24.945 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-04-19T00:44:24.944+0000 7fa7d59e0640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6812/0: (98) Address already in use 2024-04-19T00:44:24.993 INFO:tasks.ceph.mgr.x.smithi161.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-19T00:44:24.993 INFO:tasks.ceph.mgr.x.smithi161.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-19T00:44:24.993 INFO:tasks.ceph.mgr.x.smithi161.stderr: from numpy import show_config as show_numpy_config 2024-04-19T00:44:25.041 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:25.037+0000 7f2a53c28180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-19T00:44:25.150 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T00:44:25.194 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:25.189+0000 7f2a53c28180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-19T00:44:25.275 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient: get_monmap_and_config 2024-04-19T00:44:25.275 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient: build_initial_monmap 2024-04-19T00:44:25.275 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient: monmap: 2024-04-19T00:44:25.275 INFO:teuthology.misc.health.smithi043.stderr:epoch 0 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:44:25.276317+0000 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:44:25.276317+0000 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:25.276 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 Processor -- start 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 -- start start 2024-04-19T00:44:25.277 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient: init 2024-04-19T00:44:25.287 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient: _add_conns ranks=[3,4,5,6,10,2,9,0,8,1,7] 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf0075d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): picked mon.noname-g con 0x7f1bf0075970 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 0x7f1bf0076290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf009eb80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1bf0138bb0 0x7f1bf0139010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): picked mon.noname-h con 0x7f1bf0138bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf00767d0 con 0x7f1bf0075970 2024-04-19T00:44:25.288 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f1bf00a3e40 con 0x7f1bf009eb80 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f1bf00a0030 con 0x7f1bf0138bb0 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bf4dc7640 10 monclient(hunting): authenticate will time out at 935.403507s 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 0x7f1bf0076290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 0x7f1bf0076290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36316/0 (socket says 172.21.15.43:36316) 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 1 -- 172.21.15.43:0/2102883330 learned_addr learned my addr 172.21.15.43:0/2102883330 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:25.289 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): get_auth_request con 0x7f1bf009eb80 auth_method 0 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:25.290 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bee59c640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf0075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1beed9d640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1bf0138bb0 0x7f1bf0139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1beed9d640 10 monclient(hunting): get_auth_request con 0x7f1bf0138bb0 auth_method 0 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1beed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1beed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1beed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bee59c640 10 monclient(hunting): get_auth_request con 0x7f1bf0075970 auth_method 0 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bee59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:25.291 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 1 -- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1bf0138bb0 msgr2=0x7f1bf0139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.272+0000 7f1bedd9b640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1bf0138bb0 0x7f1bf0139010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 1 -- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 msgr2=0x7f1bf0075d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf0075d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 10 monclient: _finish_hunting 0 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 1 monclient: found mon.noname-c 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:25.292 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 1 -- 172.21.15.43:0/2102883330 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf00a05f0 con 0x7f1bf009eb80 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bee59c640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bedd9b640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 0x7f1bf0076290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1be40098a0 tx=0x7f1be401c3e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7bb05c36802c525f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 1 -- 172.21.15.43:0/2102883330 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1be40b0070 con 0x7f1bf009eb80 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: dump: 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:25.293 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:25.294 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: _finish_auth 0 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: _check_auth_tickets 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 1 -- 172.21.15.43:0/2102883330 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be40aa810 con 0x7f1bf009eb80 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 5 monclient: authenticate success, global_id 4135 2024-04-19T00:44:25.295 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 1 -- 172.21.15.43:0/2102883330 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1be40a9300 con 0x7f1bf009eb80 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 4 set_mon_vals no callback set 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bed59a640 10 monclient: dump: 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:25.296 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:25.297 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 10 monclient: shutdown 2024-04-19T00:44:25.298 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/2102883330 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 msgr2=0x7f1bf0076290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 0x7f1bf0076290 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1be40098a0 tx=0x7f1be401c3e0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/2102883330 shutdown_connections 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1bf0138bb0 0x7f1bf0139010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf009eb80 0x7f1bf0076290 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/2102883330 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf0075d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/2102883330 >> 172.21.15.43:0/2102883330 conn(0x7f1bf00a4790 msgr2=0x7f1bf00a4ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:25.299 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/2102883330 shutdown_connections 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/2102883330 wait complete. 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.276+0000 7f1bf4dc7640 10 monclient: build_initial_monmap 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: monmap: 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:epoch 0 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:created 0.000000 2024-04-19T00:44:25.300 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:25.301 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 librados: starting msgr at 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 librados: starting objecter 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 Processor -- start 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- start start 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 librados: setting wanted keys 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 librados: calling monclient init 2024-04-19T00:44:25.302 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: init 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _add_conns ranks=[3,7,4,0,6,8,1,2,5] 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf01d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): picked mon.noname-g con 0x7f1bf0075970 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1bf009eb80 0x7f1bf0137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): picked mon.noname-i con 0x7f1bf009eb80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 0x7f1bf013cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf0138bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:25.303 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf00a0e70 con 0x7f1bf0075970 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f1bf00a0cf0 con 0x7f1bf0138bb0 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1bf0142690 con 0x7f1bf009eb80 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:25.304 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient(hunting): authenticate will time out at 935.409254s 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 0x7f1bf013cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 0x7f1bf013cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36318/0 (socket says 172.21.15.43:36318) 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 -- 172.21.15.43:0/1223169009 learned_addr learned my addr 172.21.15.43:0/1223169009 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): get_auth_request con 0x7f1bf0138bb0 auth_method 0 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:25.305 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:25.306 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf01d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:25.306 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): get_auth_request con 0x7f1bf0075970 auth_method 0 2024-04-19T00:44:25.306 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:25.306 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:25.306 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:25.306 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1bf009eb80 0x7f1bf0137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): get_auth_request con 0x7f1bf009eb80 auth_method 0 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:25.307 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient(hunting): handle_auth_done global_id 4144 payload 824 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 -- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1bf009eb80 msgr2=0x7f1bf0137bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1bf009eb80 0x7f1bf0137bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.308 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 -- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 msgr2=0x7f1bf01d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf01d07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 monclient: found mon.noname-c 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 -- 172.21.15.43:0/1223169009 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf013f580 con 0x7f1bf0138bb0 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1beed9d640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 0x7f1bf013cda0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007d80 tx=0x7f1be800e4b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 client.?.objecter ms_handle_connect 0x7f1bf0138bb0 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:25.309 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1be800edd0 con 0x7f1bf0138bb0 2024-04-19T00:44:25.310 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:25.310 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:25.310 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: dump: 2024-04-19T00:44:25.310 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:25.310 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:25.310 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.314 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.314 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:25.315 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:25.315 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:25.315 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:25.315 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:25.315 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:25.315 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:25.316 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:25.316 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:25.316 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:25.316 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:25.316 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:25.316 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:25.317 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:25.317 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:25.317 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be8019070 con 0x7f1bf0138bb0 2024-04-19T00:44:25.317 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:25.317 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1be8014d00 con 0x7f1bf0138bb0 2024-04-19T00:44:25.317 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 5 monclient: authenticate success, global_id 4144 2024-04-19T00:44:25.318 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _renew_subs 2024-04-19T00:44:25.318 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:25.318 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdffff640 4 set_mon_vals no callback set 2024-04-19T00:44:25.318 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf0072f20 con 0x7f1bf0138bb0 2024-04-19T00:44:25.318 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:25.318 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 monclient: dump: 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:25.319 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:25.320 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _renew_subs 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf0123590 con 0x7f1bf0138bb0 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 librados: init done 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1be8014420 con 0x7f1bf0138bb0 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f1be801e460 con 0x7f1bf0138bb0 2024-04-19T00:44:25.321 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 10 client.4144.objecter ms_dispatch 0x7f1bf0000ba0 osd_map(8..8 src has 1..8) 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 3 client.4144.objecter handle_osd_map got epochs [8,8] > 0 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 3 client.4144.objecter handle_osd_map decoding full epoch 8 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bdeffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.280+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf0068110 con 0x7f1bf0138bb0 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.284+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1be8014710 con 0x7f1bf0138bb0 2024-04-19T00:44:25.322 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.284+0000 7f1bdeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:25.323 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.284+0000 7f1bdeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:25.344 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.340+0000 7f1bf4dc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:25.344 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.340+0000 7f1bf4dc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:25.344 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.340+0000 7f1bf4dc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:25.344 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.340+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bf0138920 con 0x7f1bf0138bb0 2024-04-19T00:44:25.345 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bdeffd640 1 -- 172.21.15.43:0/1223169009 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f1be80148f0 con 0x7f1bf0138bb0 2024-04-19T00:44:25.345 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bdeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:25.345 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bdeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:25.346 INFO:teuthology.misc.health.smithi043.stdout: 2024-04-19T00:44:25.346 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":8,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T00:44:24.286950+0000","last_up_change":"0.000000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T00:44:25.347 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 10 librados: watch_flush enter 2024-04-19T00:44:25.347 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 10 librados: watch_flush exit 2024-04-19T00:44:25.347 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 20 client.4144.objecter shutdown clearing up homeless session... 2024-04-19T00:44:25.347 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 10 client.4144.objecter successfully canceled tick 2024-04-19T00:44:25.347 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 10 monclient: shutdown 2024-04-19T00:44:25.347 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 msgr2=0x7f1bf013cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 0x7f1bf013cda0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007d80 tx=0x7f1be800e4b0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 shutdown_connections 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1bf0138bb0 0x7f1bf013cda0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1bf009eb80 0x7f1bf0137bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 --2- 172.21.15.43:0/1223169009 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1bf0075970 0x7f1bf01d07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 >> 172.21.15.43:0/1223169009 conn(0x7f1bf00a4790 msgr2=0x7f1bf0070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 shutdown_connections 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 -- 172.21.15.43:0/1223169009 wait complete. 2024-04-19T00:44:25.348 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:25.344+0000 7f1bf4dc7640 1 librados: shutdown 2024-04-19T00:44:25.349 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:25.345+0000 7f2a53c28180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-19T00:44:25.361 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-19T00:44:25.414 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:25.409+0000 7f2a53c28180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-19T00:44:25.545 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:25.541+0000 7f2a53c28180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-19T00:44:26.165 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:26.161+0000 7f2a53c28180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-19T00:44:26.239 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:26.237+0000 7f2a53c28180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-19T00:44:26.418 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:26.417+0000 7f2a53c28180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-19T00:44:26.491 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:26.489+0000 7f2a53c28180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-19T00:44:26.780 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:26.777+0000 7f2a53c28180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-19T00:44:26.841 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:26.837+0000 7f2a53c28180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-19T00:44:27.892 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.885+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-19T00:44:27.892 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.889+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-19T00:44:27.892 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.889+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.e 2024-04-19T00:44:27.892 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.889+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.d 2024-04-19T00:44:27.893 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.889+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.h 2024-04-19T00:44:27.894 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.893+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.f 2024-04-19T00:44:27.897 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.893+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.g 2024-04-19T00:44:27.897 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-04-19T00:44:27.893+0000 7f29fb071640 -1 mgr.server handle_report got status from non-daemon mon.i 2024-04-19T00:44:28.697 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-04-19T00:44:28.696+0000 7fa7d6a207c0 -1 osd.2 0 log_to_monitors true 2024-04-19T00:44:28.762 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-04-19T00:44:28.761+0000 7fab3ab317c0 -1 osd.4 0 log_to_monitors true 2024-04-19T00:44:28.794 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-04-19T00:44:28.792+0000 7fe8e1fca7c0 -1 osd.0 0 log_to_monitors true 2024-04-19T00:44:28.802 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-04-19T00:44:28.801+0000 7f348fa597c0 -1 osd.3 0 log_to_monitors true 2024-04-19T00:44:28.834 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-04-19T00:44:28.832+0000 7f36b69787c0 -1 osd.1 0 log_to_monitors true 2024-04-19T00:44:28.863 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-04-19T00:44:28.861+0000 7f5c43bde7c0 -1 osd.5 0 log_to_monitors true 2024-04-19T00:44:29.950 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-04-19T00:44:29.948+0000 7fe8ddf73640 -1 osd.0 0 waiting for initial osdmap 2024-04-19T00:44:29.953 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-04-19T00:44:29.952+0000 7f36b2921640 -1 osd.1 0 waiting for initial osdmap 2024-04-19T00:44:29.965 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-04-19T00:44:29.964+0000 7fe8d9daf640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:29.982 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-04-19T00:44:29.980+0000 7f36ae75d640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:29.989 INFO:tasks.ceph.osd.0.smithi043.stderr:2024-04-19T00:44:29.988+0000 7fe8d58d9640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:30.005 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-04-19T00:44:30.004+0000 7f36aa287640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:30.650 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-04-19T00:44:30.648+0000 7fa7d31db640 -1 osd.2 0 waiting for initial osdmap 2024-04-19T00:44:30.694 INFO:tasks.ceph.osd.2.smithi043.stderr:2024-04-19T00:44:30.672+0000 7fa7ca7fd640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:31.735 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-04-19T00:44:31.733+0000 7fab372ec640 -1 osd.4 0 waiting for initial osdmap 2024-04-19T00:44:31.736 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-04-19T00:44:31.733+0000 7f348ba02640 -1 osd.3 0 waiting for initial osdmap 2024-04-19T00:44:31.759 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-04-19T00:44:31.757+0000 7f5c3fb87640 -1 osd.5 0 waiting for initial osdmap 2024-04-19T00:44:31.763 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-04-19T00:44:31.761+0000 7f3483836640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:31.763 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-04-19T00:44:31.761+0000 7fab2e90e640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:31.779 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-04-19T00:44:31.777+0000 7f5c379bb640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-19T00:44:31.967 DEBUG:teuthology.orchestra.run.smithi043:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-19T00:44:32.096 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.092+0000 7f849ff95640 10 monclient: get_monmap_and_config 2024-04-19T00:44:32.096 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.092+0000 7f849ff95640 10 monclient: build_initial_monmap 2024-04-19T00:44:32.096 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.092+0000 7f849ff95640 10 monclient: monmap: 2024-04-19T00:44:32.096 INFO:teuthology.misc.health.smithi043.stderr:epoch 0 2024-04-19T00:44:32.096 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.096 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:44:32.097824+0000 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:44:32.097824+0000 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:32.097 INFO:teuthology.misc.health.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:32.098 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:32.099 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 Processor -- start 2024-04-19T00:44:32.099 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 -- start start 2024-04-19T00:44:32.099 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient: init 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient: _add_conns ranks=[3,4,8,9,10,6,1,2,7,5,0] 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8498135a60 0x7f8498135e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): picked mon.noname-g con 0x7f8498135a60 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981388b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): picked mon.noname-c con 0x7f84980a51c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 0x7f8498138df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): picked mon.noname-e con 0x7f84980a19e0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f84980a78d0 con 0x7f8498135a60 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f84980a3ca0 con 0x7f84980a51c0 2024-04-19T00:44:32.100 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f84980a39c0 con 0x7f84980a19e0 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient(hunting): authenticate will time out at 942.224920s 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 0x7f8498138df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 0x7f8498138df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59346/0 (socket says 172.21.15.43:59346) 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 -- 172.21.15.43:0/3081925655 learned_addr learned my addr 172.21.15.43:0/3081925655 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): get_auth_request con 0x7f84980a19e0 auth_method 0 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849d530640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:32.101 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849d530640 10 monclient(hunting): get_auth_request con 0x7f84980a51c0 auth_method 0 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849d530640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849d530640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849d530640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849dd31640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8498135a60 0x7f8498135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849dd31640 10 monclient(hunting): get_auth_request con 0x7f8498135a60 auth_method 0 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849dd31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849dd31640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849dd31640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 -- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 msgr2=0x7f84981388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 -- 172.21.15.43:0/3081925655 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8498135a60 msgr2=0x7f8498135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:32.102 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8498135a60 0x7f8498135e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.103 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient: _finish_hunting 0 2024-04-19T00:44:32.103 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 monclient: found mon.noname-e 2024-04-19T00:44:32.103 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:32.103 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:32.103 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 -- 172.21.15.43:0/3081925655 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84980a4970 con 0x7f84980a19e0 2024-04-19T00:44:32.109 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849e532640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 0x7f8498138df0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8490009eb0 tx=0x7f849001c8f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=91c8d802ea9a5af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:32.109 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 1 -- 172.21.15.43:0/3081925655 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84900090d0 con 0x7f84980a19e0 2024-04-19T00:44:32.109 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.109 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: dump: 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.110 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: _finish_auth 0 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: _check_auth_tickets 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 1 -- 172.21.15.43:0/3081925655 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84900a6950 con 0x7f84980a19e0 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 5 monclient: authenticate success, global_id 4193 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 1 -- 172.21.15.43:0/3081925655 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f84900af480 con 0x7f84980a19e0 2024-04-19T00:44:32.111 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849ff95640 4 set_mon_vals no callback set 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.096+0000 7f849cd2f640 10 monclient: dump: 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.112 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: shutdown 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 msgr2=0x7f8498138df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 0x7f8498138df0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8490009eb0 tx=0x7f849001c8f0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/3081925655 shutdown_connections 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f84980a19e0 0x7f8498138df0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- 172.21.15.43:0/3081925655 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8498135a60 0x7f8498135e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.113 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/3081925655 >> 172.21.15.43:0/3081925655 conn(0x7f84980aaf80 msgr2=0x7f84980ab390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/3081925655 shutdown_connections 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/3081925655 wait complete. 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: build_initial_monmap 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: monmap: 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:epoch 0 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:created 0.000000 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:32.114 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 librados: starting msgr at 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 librados: starting objecter 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 Processor -- start 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- start start 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 librados: setting wanted keys 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 librados: calling monclient init 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: init 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _add_conns ranks=[2,4,5,0,8,1,7,3,6] 2024-04-19T00:44:32.115 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 0x7f84981d0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): picked mon.noname-b con 0x7f84980a19e0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981d1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): picked mon.noname-c con 0x7f84980a51c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8498135a60 0x7f84981d1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): picked mon.noname-h con 0x7f8498135a60 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8498142650 con 0x7f84980a19e0 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.116 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f84981427d0 con 0x7f84980a51c0 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8498142950 con 0x7f8498135a60 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient(hunting): authenticate will time out at 942.229892s 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981d1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 0x7f84981d0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981d1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48468/0 (socket says 172.21.15.43:48468) 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 -- 172.21.15.43:0/1036313847 learned_addr learned my addr 172.21.15.43:0/1036313847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 0x7f84981d0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35220/0 (socket says 172.21.15.43:35220) 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): get_auth_request con 0x7f84980a51c0 auth_method 0 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.117 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): get_auth_request con 0x7f84980a19e0 auth_method 0 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849e532640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8498135a60 0x7f84981d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849e532640 10 monclient(hunting): get_auth_request con 0x7f8498135a60 auth_method 0 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849e532640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849e532640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.118 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849e532640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient(hunting): handle_auth_done global_id 4189 payload 824 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 -- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8498135a60 msgr2=0x7f84981d1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8498135a60 0x7f84981d1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 -- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 msgr2=0x7f84981d0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 0x7f84981d0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient: _finish_hunting 0 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 monclient: found mon.noname-c 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 -- 172.21.15.43:0/1036313847 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84980a2ef0 con 0x7f84980a51c0 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849dd31640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 0x7f84981d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849d530640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981d1030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f848800d830 tx=0x7f848800dcf0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 client.?.objecter ms_handle_connect 0x7f84980a51c0 2024-04-19T00:44:32.119 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f848800bb30 con 0x7f84980a51c0 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: dump: 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.120 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8488019070 con 0x7f84980a51c0 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 5 monclient: authenticate success, global_id 4189 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _renew_subs 2024-04-19T00:44:32.121 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849813e3d0 con 0x7f84980a51c0 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f848800bcf0 con 0x7f84980a51c0 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 10 monclient: dump: 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:epoch 1 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.122 INFO:teuthology.misc.health.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr: 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848ffff640 4 set_mon_vals no callback set 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 client.4189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _renew_subs 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:32.123 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f849813e6b0 con 0x7f84980a51c0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 librados: init done 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f8488014430 con 0x7f84980a51c0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.100+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84980a1ec0 con 0x7f84980a51c0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f847c038d80 0x7f847c03b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(12..12 src has 1..12) ==== 3607+0+0 (secure 0 0 0) 0x7f848800f040 con 0x7f84980a51c0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 10 client.4189.objecter ms_dispatch 0x7f8498000ba0 osd_map(12..12 src has 1..12) 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 3 client.4189.objecter handle_osd_map got epochs [12,12] > 0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 3 client.4189.objecter handle_osd_map decoding full epoch 12 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 20 client.4189.objecter dump_active .. 0 homeless 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f849dd31640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f847c038d80 0x7f847c03b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.124 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f849dd31640 10 monclient: get_auth_request con 0x7f847c038d80 auth_method 0 2024-04-19T00:44:32.125 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f849dd31640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f847c038d80 0x7f847c03b240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8494007d30 tx=0x7f8494008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:32.125 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.104+0000 7f848e7fc640 10 client.4189.objecter ms_handle_connect 0x7f847c038d80 2024-04-19T00:44:32.125 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.108+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8488022020 con 0x7f84980a51c0 2024-04-19T00:44:32.125 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.108+0000 7f848e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:32.125 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.108+0000 7f848e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:32.272 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.268+0000 7f849ff95640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:32.272 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.268+0000 7f849ff95640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:32.272 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.268+0000 7f849ff95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:32.272 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.268+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8498068110 con 0x7f84980a51c0 2024-04-19T00:44:32.272 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f848e7fc640 1 -- 172.21.15.43:0/1036313847 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+8554 (secure 0 0 0) 0x7f84980a2ef0 con 0x7f84980a51c0 2024-04-19T00:44:32.273 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f848e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:32.273 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f848e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:32.273 INFO:teuthology.misc.health.smithi043.stdout: 2024-04-19T00:44:32.273 INFO:teuthology.misc.health.smithi043.stdout:{"epoch":12,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T00:44:32.017099+0000","last_up_change":"2024-04-19T00:44:32.017099+0000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6804","nonce":110940973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6809","nonce":110940973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6821","nonce":110940973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6815","nonce":110940973}]},"public_addr":"172.21.15.43:6804/110940973","cluster_addr":"172.21.15.43:6809/110940973","heartbeat_back_addr":"172.21.15.43:6821/110940973","heartbeat_front_addr":"172.21.15.43:6815/110940973","state":["exists","up"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6802","nonce":1380723486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6808","nonce":1380723486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6820","nonce":1380723486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6814","nonce":1380723486}]},"public_addr":"172.21.15.43:6802/1380723486","cluster_addr":"172.21.15.43:6808/1380723486","heartbeat_back_addr":"172.21.15.43:6820/1380723486","heartbeat_front_addr":"172.21.15.43:6814/1380723486","state":["exists","up"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6806","nonce":1290935720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6813","nonce":1290935720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6823","nonce":1290935720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6819","nonce":1290935720}]},"public_addr":"172.21.15.43:6806/1290935720","cluster_addr":"172.21.15.43:6813/1290935720","heartbeat_back_addr":"172.21.15.43:6823/1290935720","heartbeat_front_addr":"172.21.15.43:6819/1290935720","state":["exists","up"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6801","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6804","nonce":2204255832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6810","nonce":2204255832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6821","nonce":2204255832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6815","nonce":2204255832}]},"public_addr":"172.21.15.161:6804/2204255832","cluster_addr":"172.21.15.161:6810/2204255832","heartbeat_back_addr":"172.21.15.161:6821/2204255832","heartbeat_front_addr":"172.21.15.161:6815/2204255832","state":["exists","up"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6803","nonce":3653492594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6809","nonce":3653492594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6822","nonce":3653492594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6816","nonce":3653492594}]},"public_addr":"172.21.15.161:6803/3653492594","cluster_addr":"172.21.15.161:6809/3653492594","heartbeat_back_addr":"172.21.15.161:6822/3653492594","heartbeat_front_addr":"172.21.15.161:6816/3653492594","state":["exists","up"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6805","nonce":760479844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6811","nonce":760479844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6823","nonce":760479844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6817","nonce":760479844}]},"public_addr":"172.21.15.161:6805/760479844","cluster_addr":"172.21.15.161:6811/760479844","heartbeat_back_addr":"172.21.15.161:6823/760479844","heartbeat_front_addr":"172.21.15.161:6817/760479844","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.756279+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.840443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.708058+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T00:44:32.275 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 10 librados: watch_flush enter 2024-04-19T00:44:32.275 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 10 librados: watch_flush exit 2024-04-19T00:44:32.275 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 20 client.4189.objecter shutdown clearing up homeless session... 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 10 client.4189.objecter successfully canceled tick 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f847c038d80 msgr2=0x7f847c03b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f847c038d80 0x7f847c03b240 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8494007d30 tx=0x7f8494008040 comp rx=0 tx=0).stop 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 10 monclient: shutdown 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 msgr2=0x7f84981d1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981d1030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f848800d830 tx=0x7f848800dcf0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 shutdown_connections 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f847c038d80 0x7f847c03b240 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8498135a60 0x7f84981d1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f84980a51c0 0x7f84981d1030 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 --2- 172.21.15.43:0/1036313847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f84980a19e0 0x7f84981d0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 >> 172.21.15.43:0/1036313847 conn(0x7f84980aaf80 msgr2=0x7f84980a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:32.276 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 shutdown_connections 2024-04-19T00:44:32.277 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 -- 172.21.15.43:0/1036313847 wait complete. 2024-04-19T00:44:32.277 INFO:teuthology.misc.health.smithi043.stderr:2024-04-19T00:44:32.272+0000 7f849ff95640 1 librados: shutdown 2024-04-19T00:44:32.291 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-19T00:44:32.291 INFO:tasks.ceph:Creating RBD pool 2024-04-19T00:44:32.291 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient: get_monmap_and_config 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient: build_initial_monmap 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient: monmap: 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:32.418597+0000 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:32.418597+0000 2024-04-19T00:44:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:32.419 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 Processor -- start 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 -- start start 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient: init 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient: _add_conns ranks=[0,6,8,9,7,10,3,1,5,2,4] 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54d0138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f54d0139540 0x7f54d013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): picked mon.noname-d con 0x7f54d0139540 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f54d00a5bb0 0x7f54d0140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): picked mon.noname-e con 0x7f54d00a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54d00a0d40 con 0x7f54d0138a50 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f54d00a1020 con 0x7f54d0139540 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f54d00a0780 con 0x7f54d00a5bb0 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54d50a6640 10 monclient(hunting): authenticate will time out at 942.545719s 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f54d00a5bb0 0x7f54d0140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ced9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f54d0138a50 0x7f54d0138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44732/0 (socket says 172.21.15.43:44732) 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ced9d640 1 -- 172.21.15.43:0/506473732 learned_addr learned my addr 172.21.15.43:0/506473732 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): get_auth_request con 0x7f54d00a5bb0 auth_method 0 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 1 --2- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f54d0139540 0x7f54d013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): get_auth_request con 0x7f54d0139540 auth_method 0 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cdd9b640 1 -- 172.21.15.43:0/506473732 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037481419 0 0) 0x7f54d00a0d40 con 0x7f54d0138a50 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cdd9b640 10 monclient(hunting): _negotiate 2024-04-19T00:44:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cdd9b640 10 monclient(hunting): my global_id is 4203 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cdd9b640 1 -- 172.21.15.43:0/506473732 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54b8003530 con 0x7f54d0138a50 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient(hunting): handle_auth_done global_id 4202 payload 274 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 1 -- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f54d0139540 msgr2=0x7f54d013dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 1 --2- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f54d0139540 0x7f54d013dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 1 -- 172.21.15.43:0/506473732 >> v1:172.21.15.43:6789/0 conn(0x7f54d0138a50 legacy=0x7f54d0138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 1 monclient: found mon.noname-e 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.416+0000 7f54cf59e640 1 -- 172.21.15.43:0/506473732 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d00a0a60 con 0x7f54d00a5bb0 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 --2- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f54d00a5bb0 0x7f54d0140530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009dc0 tx=0x7f54c801c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=50a1b0b40c2a8956 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 1 -- 172.21.15.43:0/506473732 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54c80b2070 con 0x7f54d00a5bb0 2024-04-19T00:44:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: dump: 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: _finish_auth 0 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 5 monclient: authenticate success, global_id 4202 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 1 -- 172.21.15.43:0/506473732 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c80a5a50 con 0x7f54d00a5bb0 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 1 -- 172.21.15.43:0/506473732 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54c80a5c40 con 0x7f54d00a5bb0 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 4 set_mon_vals no callback set 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cdd9b640 10 monclient: dump: 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: shutdown 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f54d00a5bb0 msgr2=0x7f54d0140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 --2- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f54d00a5bb0 0x7f54d0140530 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009dc0 tx=0x7f54c801c850 comp rx=0 tx=0).stop 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/506473732 shutdown_connections 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 --2- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f54d00a5bb0 0x7f54d0140530 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 --2- 172.21.15.43:0/506473732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f54d0139540 0x7f54d013dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/506473732 >> 172.21.15.43:0/506473732 conn(0x7f54d00a7cb0 msgr2=0x7f54d00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/506473732 shutdown_connections 2024-04-19T00:44:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/506473732 wait complete. 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: build_initial_monmap 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: monmap: 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 librados: starting msgr at 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 librados: starting objecter 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 Processor -- start 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- start start 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 librados: setting wanted keys 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 librados: calling monclient init 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: init 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: _add_conns ranks=[7,3,0,1,6,5,4,8,2] 2024-04-19T00:44:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f54d00a5bb0 0x7f54d0137770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): picked mon.noname-i con 0x7f54d00a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54d0138a50 0x7f54d013c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): picked mon.noname-g con 0x7f54d0138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 0x7f54d013ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): picked mon.noname-a con 0x7f54d0139540 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f54d0143620 con 0x7f54d0139540 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f54d01434a0 con 0x7f54d0138a50 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f54d01437a0 con 0x7f54d00a5bb0 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient(hunting): authenticate will time out at 942.550881s 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 0x7f54d013ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 0x7f54d013ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54130/0 (socket says 172.21.15.43:54130) 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 -- 172.21.15.43:0/2048563887 learned_addr learned my addr 172.21.15.43:0/2048563887 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): get_auth_request con 0x7f54d0139540 auth_method 0 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54ced9d640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f54d00a5bb0 0x7f54d0137770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54ced9d640 10 monclient(hunting): get_auth_request con 0x7f54d00a5bb0 auth_method 0 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54ced9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient(hunting): handle_auth_done global_id 4212 payload 824 2024-04-19T00:44:32.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54ce59c640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54d0138a50 0x7f54d013c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 -- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f54d00a5bb0 msgr2=0x7f54d0137770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f54d00a5bb0 0x7f54d0137770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 -- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54d0138a50 msgr2=0x7f54d013c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54d0138a50 0x7f54d013c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 monclient: found mon.noname-a 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 -- 172.21.15.43:0/2048563887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d013ed30 con 0x7f54d0139540 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cf59e640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 0x7f54d013ca40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f54c80b0040 tx=0x7f54c80195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 client.?.objecter ms_handle_connect 0x7f54d0139540 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54c80b2050 con 0x7f54d0139540 2024-04-19T00:44:32.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: dump: 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.439 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 5 monclient: authenticate success, global_id 4212 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: _renew_subs 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c80be070 con 0x7f54d0139540 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d009fc90 con 0x7f54d0139540 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f54c8003e10 con 0x7f54d0139540 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:32.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54b77fe640 10 monclient: dump: 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:32.441 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: _renew_subs 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54cd59a640 4 set_mon_vals no callback set 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d0121e20 con 0x7f54d0139540 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.420+0000 7f54d50a6640 1 librados: init done 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54d50a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54d50a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54d50a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f54c80039a0 con 0x7f54d0139540 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d013ea80 con 0x7f54d0139540 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f54bc038d80 0x7f54bc03b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:32.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3607+0+0 (secure 0 0 0) 0x7f54c80e3770 con 0x7f54d0139540 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 10 client.4212.objecter ms_dispatch 0x7f54d0000ba0 osd_map(12..12 src has 1..12) 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 3 client.4212.objecter handle_osd_map got epochs [12,12] > 0 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 3 client.4212.objecter handle_osd_map decoding full epoch 12 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 20 client.4212.objecter dump_active .. 0 homeless 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54ced9d640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f54bc038d80 0x7f54bc03b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54ced9d640 10 monclient: get_auth_request con 0x7f54bc038d80 auth_method 0 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54b77fe640 10 client.4212.objecter ms_handle_connect 0x7f54bc038d80 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.424+0000 7f54ced9d640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f54bc038d80 0x7f54bc03b240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54c0007a70 tx=0x7f54c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.428+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f54c8016560 con 0x7f54d0139540 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.428+0000 7f54b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:32.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.428+0000 7f54b77fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.596+0000 7f54d50a6640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-19T00:44:32.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.596+0000 7f54d50a6640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-19T00:44:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.596+0000 7f54d50a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:32.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:32.596+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f54d01d1440 con 0x7f54d0139540 2024-04-19T00:44:33.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.040+0000 7f54b77fe640 1 -- 172.21.15.43:0/2048563887 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7f54c80ad840 con 0x7f54d0139540 2024-04-19T00:44:33.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.040+0000 7f54b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-19T00:44:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.040+0000 7f54b77fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-19T00:44:33.042 INFO:teuthology.orchestra.run.smithi043.stderr:pool 'rbd' created 2024-04-19T00:44:33.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 10 librados: watch_flush enter 2024-04-19T00:44:33.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 10 librados: watch_flush exit 2024-04-19T00:44:33.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 20 client.4212.objecter shutdown clearing up homeless session... 2024-04-19T00:44:33.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 10 client.4212.objecter successfully canceled tick 2024-04-19T00:44:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f54bc038d80 msgr2=0x7f54bc03b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f54bc038d80 0x7f54bc03b240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f54c0007a70 tx=0x7f54c0009040 comp rx=0 tx=0).stop 2024-04-19T00:44:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 10 monclient: shutdown 2024-04-19T00:44:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 msgr2=0x7f54d013ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:33.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 0x7f54d013ca40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f54c80b0040 tx=0x7f54c80195f0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 shutdown_connections 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f54bc038d80 0x7f54bc03b240 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f54d0139540 0x7f54d013ca40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f54d0138a50 0x7f54d013c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 --2- 172.21.15.43:0/2048563887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f54d00a5bb0 0x7f54d0137770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.044+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 >> 172.21.15.43:0/2048563887 conn(0x7f54d00a7cb0 msgr2=0x7f54d00a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.048+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 shutdown_connections 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.048+0000 7f54d50a6640 1 -- 172.21.15.43:0/2048563887 wait complete. 2024-04-19T00:44:33.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.048+0000 7f54d50a6640 1 librados: shutdown 2024-04-19T00:44:33.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-19T00:44:33.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient: get_monmap_and_config 2024-04-19T00:44:33.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient: build_initial_monmap 2024-04-19T00:44:33.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient: monmap: 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:33.195619+0000 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:33.195619+0000 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:33.195 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:33.196 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 Processor -- start 2024-04-19T00:44:33.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 -- start start 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient: init 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient: _add_conns ranks=[0,7,8,3,4,2,9,6,10,1,5] 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f50138d60 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6f50139140 0x7f6f50139520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): picked mon.noname-i con 0x7f6f50139140 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f5013e080 0x7f6f50140470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): picked mon.noname-e con 0x7f6f5013e080 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f5009f790 con 0x7f6f50138d60 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6f500a7d40 con 0x7f6f50139140 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:33.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6f5009f4e0 con 0x7f6f5013e080 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f57a6e640 10 monclient(hunting): authenticate will time out at 943.323097s 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5580a640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6f50138d60 0x7f6f500a6080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44746/0 (socket says 172.21.15.43:44746) 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5580a640 1 -- 172.21.15.43:0/614632423 learned_addr learned my addr 172.21.15.43:0/614632423 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f5013e080 0x7f6f50140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): get_auth_request con 0x7f6f5013e080 auth_method 0 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f55009640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6f50139140 0x7f6f50139520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f55009640 10 monclient(hunting): get_auth_request con 0x7f6f50139140 auth_method 0 2024-04-19T00:44:33.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f55009640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f55009640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f55009640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient(hunting): handle_auth_done global_id 4211 payload 274 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f54808640 1 -- 172.21.15.43:0/614632423 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3294413756 0 0) 0x7f6f5009f790 con 0x7f6f50138d60 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 1 -- 172.21.15.43:0/614632423 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6f50139140 msgr2=0x7f6f50139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6f50139140 0x7f6f50139520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 1 -- 172.21.15.43:0/614632423 >> v1:172.21.15.43:6789/0 conn(0x7f6f50138d60 legacy=0x7f6f500a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient: _finish_hunting 0 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 1 monclient: found mon.noname-e 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f5600b640 1 -- 172.21.15.43:0/614632423 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f500a0bd0 con 0x7f6f5013e080 2024-04-19T00:44:33.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.192+0000 7f6f54808640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f5600b640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f5013e080 0x7f6f50140470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009df0 tx=0x7f6f4801c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=30f2d8926b08964 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 1 -- 172.21.15.43:0/614632423 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f480a6960 con 0x7f6f5013e080 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: dump: 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:33.207 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: _finish_auth 0 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: _check_auth_tickets 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 1 -- 172.21.15.43:0/614632423 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f480a6b20 con 0x7f6f5013e080 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 5 monclient: authenticate success, global_id 4211 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:33.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 4 set_mon_vals no callback set 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 1 -- 172.21.15.43:0/614632423 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f480a6cd0 con 0x7f6f5013e080 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f54808640 10 monclient: dump: 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:33.209 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: shutdown 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- 172.21.15.43:0/614632423 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f5013e080 msgr2=0x7f6f50140470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f5013e080 0x7f6f50140470 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009df0 tx=0x7f6f4801c850 comp rx=0 tx=0).stop 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- 172.21.15.43:0/614632423 shutdown_connections 2024-04-19T00:44:33.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f5013e080 0x7f6f50140470 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/614632423 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6f50139140 0x7f6f50139520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- 172.21.15.43:0/614632423 >> 172.21.15.43:0/614632423 conn(0x7f6f500ab590 msgr2=0x7f6f500ab9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- 172.21.15.43:0/614632423 shutdown_connections 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- 172.21.15.43:0/614632423 wait complete. 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: build_initial_monmap 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: monmap: 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:33.211 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 librados: starting msgr at 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 librados: starting objecter 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 Processor -- start 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- start start 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 librados: setting wanted keys 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 librados: calling monclient init 2024-04-19T00:44:33.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: init 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient: _add_conns ranks=[8,5,3,2,6,1,4,7,0] 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 0x7f6f5013d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): picked mon.noname-e con 0x7f6f50138d60 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f50139140 0x7f6f50137be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): picked mon.noname-h con 0x7f6f50139140 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f5013e080 0x7f6f501d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): picked mon.noname-g con 0x7f6f5013e080 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6f50143560 con 0x7f6f5013e080 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6f501433e0 con 0x7f6f50139140 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6f501436e0 con 0x7f6f50138d60 2024-04-19T00:44:33.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.196+0000 7f6f57a6e640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient(hunting): authenticate will time out at 943.328008s 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 0x7f6f5013d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 0x7f6f5013d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59380/0 (socket says 172.21.15.43:59380) 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 -- 172.21.15.43:0/3700525998 learned_addr learned my addr 172.21.15.43:0/3700525998 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): get_auth_request con 0x7f6f50138d60 auth_method 0 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f55009640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f50139140 0x7f6f50137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5600b640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f5013e080 0x7f6f501d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:33.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 -- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f50139140 msgr2=0x7f6f50137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f50139140 0x7f6f50137be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 -- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f5013e080 msgr2=0x7f6f501d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f5013e080 0x7f6f501d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient: _finish_hunting 0 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 monclient: found mon.noname-e 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 -- 172.21.15.43:0/3700525998 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5013bea0 con 0x7f6f50138d60 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f55009640 10 monclient: get_auth_request con 0x7f6f50139140 auth_method 0 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f55009640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f50139140 0x7f6f50137be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5600b640 10 monclient: get_auth_request con 0x7f6f5013e080 auth_method 0 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5600b640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f5013e080 0x7f6f501d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:33.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f5580a640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 0x7f6f5013d180 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00c450 tx=0x7f6f4c00c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 client.?.objecter ms_handle_connect 0x7f6f50138d60 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f4c018070 con 0x7f6f50138d60 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: dump: 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:33.216 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 5 monclient: authenticate success, global_id 4220 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient: _renew_subs 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4c0047f0 con 0x7f6f50138d60 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:33.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f50121fb0 con 0x7f6f50138d60 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f4c004e50 con 0x7f6f50138d60 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f47fff640 4 set_mon_vals no callback set 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 monclient: dump: 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:33.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient: _renew_subs 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f5009f120 con 0x7f6f50138d60 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 1 librados: init done 2024-04-19T00:44:33.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f6f4c0049e0 con 0x7f6f50138d60 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f500693b0 con 0x7f6f50138d60 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f34038f20 0x7f6f3403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(13..13 src has 1..13) ==== 4324+0+0 (secure 0 0 0) 0x7f6f4c047d00 con 0x7f6f50138d60 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 client.4220.objecter ms_dispatch 0x7f6f50000ba0 osd_map(13..13 src has 1..13) 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 3 client.4220.objecter handle_osd_map got epochs [13,13] > 0 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 3 client.4220.objecter handle_osd_map decoding full epoch 13 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 20 client.4220.objecter dump_active .. 0 homeless 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f55009640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f34038f20 0x7f6f3403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f55009640 10 monclient: get_auth_request con 0x7f6f34038f20 auth_method 0 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f467fc640 10 client.4220.objecter ms_handle_connect 0x7f6f34038f20 2024-04-19T00:44:33.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.200+0000 7f6f55009640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f34038f20 0x7f6f3403b3e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f5009fdb0 tx=0x7f6f40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.204+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6f4c01ae30 con 0x7f6f50138d60 2024-04-19T00:44:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.204+0000 7f6f467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:33.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.204+0000 7f6f467fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:33.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.368+0000 7f6f57a6e640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-19T00:44:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.368+0000 7f6f57a6e640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-19T00:44:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.368+0000 7f6f57a6e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:33.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:33.368+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f6f50121a10 con 0x7f6f50138d60 2024-04-19T00:44:34.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.056+0000 7f6f467fc640 1 -- 172.21.15.43:0/3700525998 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 173+0+0 (secure 0 0 0) 0x7f6f4c00edd0 con 0x7f6f50138d60 2024-04-19T00:44:34.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.056+0000 7f6f467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-19T00:44:34.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.056+0000 7f6f467fc640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-19T00:44:34.061 INFO:teuthology.orchestra.run.smithi043.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-19T00:44:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.060+0000 7f6f57a6e640 10 librados: watch_flush enter 2024-04-19T00:44:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.060+0000 7f6f57a6e640 10 librados: watch_flush exit 2024-04-19T00:44:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.060+0000 7f6f57a6e640 20 client.4220.objecter shutdown clearing up homeless session... 2024-04-19T00:44:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 10 client.4220.objecter successfully canceled tick 2024-04-19T00:44:34.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f34038f20 msgr2=0x7f6f3403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f34038f20 0x7f6f3403b3e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f5009fdb0 tx=0x7f6f40008040 comp rx=0 tx=0).stop 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 10 monclient: shutdown 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 msgr2=0x7f6f5013d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 0x7f6f5013d180 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c00c450 tx=0x7f6f4c00c910 comp rx=0 tx=0).stop 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 shutdown_connections 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f5013e080 0x7f6f501d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f50139140 0x7f6f50137be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f34038f20 0x7f6f3403b3e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 --2- 172.21.15.43:0/3700525998 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6f50138d60 0x7f6f5013d180 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 >> 172.21.15.43:0/3700525998 conn(0x7f6f500ab590 msgr2=0x7f6f5013f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 shutdown_connections 2024-04-19T00:44:34.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 -- 172.21.15.43:0/3700525998 wait complete. 2024-04-19T00:44:34.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.064+0000 7f6f57a6e640 1 librados: shutdown 2024-04-19T00:44:34.091 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-19T00:44:34.092 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-19T00:44:34.092 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-19T00:44:34.093 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-19T00:44:34.093 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.220+0000 7fd0296d6640 10 monclient: get_monmap_and_config 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.220+0000 7fd0296d6640 10 monclient: build_initial_monmap 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.220+0000 7fd0296d6640 10 monclient: monmap: 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:34.225485+0000 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:34.225485+0000 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:34.224 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:34.225 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.220+0000 7fd0296d6640 1 Processor -- start 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.220+0000 7fd0296d6640 1 -- start start 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.220+0000 7fd0296d6640 10 monclient: init 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient: _add_conns ranks=[7,8,1,10,5,6,3,4,0,2,9] 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd024138a50 0x7fd024138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): picked mon.noname-i con 0x7fd024138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 0x7fd024139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): picked mon.noname-e con 0x7fd0240a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd024137960 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240a0d40 con 0x7fd024137960 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd0240a1020 con 0x7fd024138a50 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd0241398b0 con 0x7fd0240a5bb0 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient(hunting): authenticate will time out at 944.352673s 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 0x7fd024139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 0x7fd024139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59390/0 (socket says 172.21.15.43:59390) 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 -- 172.21.15.43:0/2499324244 learned_addr learned my addr 172.21.15.43:0/2499324244 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): get_auth_request con 0x7fd0240a5bb0 auth_method 0 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd022ffd640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd024138a50 0x7fd024138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd022ffd640 10 monclient(hunting): get_auth_request con 0x7fd024138a50 auth_method 0 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd022ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd022ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 -- 172.21.15.43:0/2499324244 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd024138a50 msgr2=0x7fd024138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd024138a50 0x7fd024138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 -- 172.21.15.43:0/2499324244 >> v1:172.21.15.161:6789/0 conn(0x7fd024137960 legacy=0x7fd024137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient: _finish_hunting 0 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 monclient: found mon.noname-e 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 -- 172.21.15.43:0/2499324244 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0240a0a60 con 0x7fd0240a5bb0 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd022ffd640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd024138a50 0x7fd024138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0227fc640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 0x7fd024139370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd0180098a0 tx=0x7fd01801c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5ab26575017d9673 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 1 -- 172.21.15.43:0/2499324244 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd0180b1070 con 0x7fd0240a5bb0 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: dump: 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 5 monclient: authenticate success, global_id 4229 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 1 -- 172.21.15.43:0/2499324244 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0180aa7b0 con 0x7fd0240a5bb0 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd0296d6640 4 set_mon_vals no callback set 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 1 -- 172.21.15.43:0/2499324244 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd0180aae50 con 0x7fd0240a5bb0 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.224+0000 7fd021ffb640 10 monclient: dump: 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.236 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: shutdown 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2499324244 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 msgr2=0x7fd024139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 0x7fd024139370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd0180098a0 tx=0x7fd01801c440 comp rx=0 tx=0).stop 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2499324244 shutdown_connections 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd0240a5bb0 0x7fd024139370 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2499324244 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd024138a50 0x7fd024138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2499324244 >> 172.21.15.43:0/2499324244 conn(0x7fd0240a7cb0 msgr2=0x7fd0240a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2499324244 shutdown_connections 2024-04-19T00:44:34.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2499324244 wait complete. 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: build_initial_monmap 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: monmap: 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:34.238 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 librados: starting msgr at 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 librados: starting objecter 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 Processor -- start 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- start start 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 librados: setting wanted keys 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 librados: calling monclient init 2024-04-19T00:44:34.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: init 2024-04-19T00:44:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: _add_conns ranks=[5,8,4,3,1,0,7,6,2] 2024-04-19T00:44:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd0240a5bb0 0x7fd024136060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): picked mon.noname-h con 0x7fd0240a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 0x7fd0241365a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): picked mon.noname-e con 0x7fd024137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 0x7fd02413c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): picked mon.noname-c con 0x7fd024138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd024143bb0 con 0x7fd024138a50 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fd024143a30 con 0x7fd0240a5bb0 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd024143d30 con 0x7fd024137960 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient(hunting): authenticate will time out at 944.358029s 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 0x7fd0241365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 0x7fd02413c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 0x7fd0241365a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59406/0 (socket says 172.21.15.43:59406) 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 0x7fd02413c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48480/0 (socket says 172.21.15.43:48480) 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 1 -- 172.21.15.43:0/2317142017 learned_addr learned my addr 172.21.15.43:0/2317142017 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): get_auth_request con 0x7fd024137960 auth_method 0 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): get_auth_request con 0x7fd024138a50 auth_method 0 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd0240a5bb0 0x7fd024136060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): get_auth_request con 0x7fd0240a5bb0 auth_method 0 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd022ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 -- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 msgr2=0x7fd0241365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 0x7fd0241365a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 -- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd0240a5bb0 msgr2=0x7fd024136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd0240a5bb0 0x7fd024136060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 monclient: found mon.noname-c 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 -- 172.21.15.43:0/2317142017 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02409ee60 con 0x7fd024138a50 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0227fc640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 0x7fd0241365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0237fe640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 0x7fd02413c5b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c700 tx=0x7fd01c00cbc0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 client.?.objecter ms_handle_connect 0x7fd024138a50 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd01c00ce10 con 0x7fd024138a50 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: dump: 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd01c003df0 con 0x7fd024138a50 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd01c00acc0 con 0x7fd024138a50 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd00b7fe640 10 monclient: dump: 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 5 monclient: authenticate success, global_id 4207 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: _renew_subs 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0217fa640 4 set_mon_vals no callback set 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd02413e1a0 con 0x7fd024138a50 2024-04-19T00:44:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: _renew_subs 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fd024140490 con 0x7fd024138a50 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.228+0000 7fd0296d6640 1 librados: init done 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd0296d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd0296d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd0296d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7fd01c00a850 con 0x7fd024138a50 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0240008e0 con 0x7fd024138a50 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd010038d80 0x7fd01003b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(14..14 src has 1..14) ==== 4346+0+0 (secure 0 0 0) 0x7fd01c048820 con 0x7fd024138a50 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 10 client.4207.objecter ms_dispatch 0x7fd024000ba0 osd_map(14..14 src has 1..14) 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 3 client.4207.objecter handle_osd_map got epochs [14,14] > 0 2024-04-19T00:44:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 3 client.4207.objecter handle_osd_map decoding full epoch 14 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 20 client.4207.objecter dump_active .. 0 homeless 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd022ffd640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd010038d80 0x7fd01003b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd022ffd640 10 monclient: get_auth_request con 0x7fd010038d80 auth_method 0 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd00b7fe640 10 client.4207.objecter ms_handle_connect 0x7fd010038d80 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.232+0000 7fd022ffd640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd010038d80 0x7fd01003b240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd014006e90 tx=0x7fd014008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.236+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd01c020020 con 0x7fd024138a50 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.236+0000 7fd00b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.236+0000 7fd00b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd0296d6640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-19T00:44:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd0296d6640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-19T00:44:34.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd0296d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:34.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fd02413e9f0 con 0x7fd024138a50 2024-04-19T00:44:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd00b7fe640 1 -- 172.21.15.43:0/2317142017 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+84686 (secure 0 0 0) 0x7fd01c01a570 con 0x7fd024138a50 2024-04-19T00:44:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd00b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-19T00:44:34.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.448+0000 7fd00b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:34.453 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:34.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 10 librados: watch_flush enter 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 10 librados: watch_flush exit 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 20 client.4207.objecter shutdown clearing up homeless session... 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 10 client.4207.objecter successfully canceled tick 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd010038d80 msgr2=0x7fd01003b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd010038d80 0x7fd01003b240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd014006e90 tx=0x7fd014008040 comp rx=0 tx=0).stop 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 10 monclient: shutdown 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:34.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 msgr2=0x7fd02413c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 0x7fd02413c5b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c700 tx=0x7fd01c00cbc0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 shutdown_connections 2024-04-19T00:44:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd024138a50 0x7fd02413c5b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd010038d80 0x7fd01003b240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd024137960 0x7fd0241365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 --2- 172.21.15.43:0/2317142017 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd0240a5bb0 0x7fd024136060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 >> 172.21.15.43:0/2317142017 conn(0x7fd0240a7cb0 msgr2=0x7fd0240a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 shutdown_connections 2024-04-19T00:44:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 -- 172.21.15.43:0/2317142017 wait complete. 2024-04-19T00:44:34.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.452+0000 7fd0296d6640 1 librados: shutdown 2024-04-19T00:44:34.473 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":4,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":2963482743},{"type":"v1","addr":"172.21.15.161:6825","nonce":2963482743}]},"active_addr":"172.21.15.161:6825/2963482743","active_change":"2024-04-19T00:44:26.863985+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":4112485778}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":1270235543}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.161:0","nonce":3696551305}]}]} 2024-04-19T00:44:34.479 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-19T00:44:34.480 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-19T00:44:34.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T00:44:34.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient: get_monmap_and_config 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient: build_initial_monmap 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient: monmap: 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:34.600295+0000 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:34.600295+0000 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:34.599 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:34.600 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 Processor -- start 2024-04-19T00:44:34.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 -- start start 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient: init 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient: _add_conns ranks=[7,1,3,6,8,0,4,2,10,9,5] 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fed8c138a50 0x7fed8c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): picked mon.noname-i con 0x7fed8c138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fed8c0a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c139370 0x7fed8c138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): picked mon.noname-g con 0x7fed8c139370 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed8c0a0d40 con 0x7fed8c0a5bb0 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fed8c0a1020 con 0x7fed8c139370 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fed8c0a0780 con 0x7fed8c138a50 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed932bc640 10 monclient(hunting): authenticate will time out at 944.727437s 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed90857640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fed8c0a5bb0 0x7fed8c137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:48316/0 (socket says 172.21.15.43:48316) 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed90857640 1 -- 172.21.15.43:0/566466612 learned_addr learned my addr 172.21.15.43:0/566466612 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91058640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fed8c138a50 0x7fed8c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91058640 10 monclient(hunting): get_auth_request con 0x7fed8c138a50 auth_method 0 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91859640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c139370 0x7fed8c138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91058640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91058640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91859640 10 monclient(hunting): get_auth_request con 0x7fed8c139370 auth_method 0 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91859640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.596+0000 7fed91859640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91058640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 1 -- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fed8c138a50 msgr2=0x7fed8c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fed8c138a50 0x7fed8c138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 1 -- 172.21.15.43:0/566466612 >> v1:172.21.15.161:6789/0 conn(0x7fed8c0a5bb0 legacy=0x7fed8c137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 1 -- 172.21.15.43:0/566466612 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358838072 0 0) 0x7fed8c0a0d40 con 0x7fed8c0a5bb0 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 10 monclient: _finish_hunting 0 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 1 monclient: found mon.noname-g 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:44:34.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 1 -- 172.21.15.43:0/566466612 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c0a0a60 con 0x7fed8c139370 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed91859640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c139370 0x7fed8c138070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fed84009dc0 tx=0x7fed8401c7f0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=fc07d4325bdd02c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 1 -- 172.21.15.43:0/566466612 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed840b2070 con 0x7fed8c139370 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: dump: 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.608 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 1 -- 172.21.15.43:0/566466612 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed840a5a00 con 0x7fed8c139370 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 5 monclient: authenticate success, global_id 4242 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 1 -- 172.21.15.43:0/566466612 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed840a5bb0 con 0x7fed8c139370 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 4 set_mon_vals no callback set 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed83fff640 10 monclient: dump: 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.611 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.612 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.612 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 10 monclient: shutdown 2024-04-19T00:44:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 -- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c139370 msgr2=0x7fed8c138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c139370 0x7fed8c138070 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fed84009dc0 tx=0x7fed8401c7f0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 -- 172.21.15.43:0/566466612 shutdown_connections 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c139370 0x7fed8c138070 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 --2- 172.21.15.43:0/566466612 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fed8c138a50 0x7fed8c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 -- 172.21.15.43:0/566466612 >> 172.21.15.43:0/566466612 conn(0x7fed8c0a7cb0 msgr2=0x7fed8c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 -- 172.21.15.43:0/566466612 shutdown_connections 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.600+0000 7fed932bc640 1 -- 172.21.15.43:0/566466612 wait complete. 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: build_initial_monmap 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: monmap: 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:34.618 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:34.619 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 librados: starting msgr at 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 librados: starting objecter 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 Processor -- start 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- start start 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 librados: setting wanted keys 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 librados: calling monclient init 2024-04-19T00:44:34.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: init 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _add_conns ranks=[3,8,1,2,6,0,4,5,7] 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c0a5bb0 0x7fed8c1d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): picked mon.noname-g con 0x7fed8c0a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 0x7fed8c1d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): picked mon.noname-e con 0x7fed8c138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8c139370 0x7fed8c134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): picked mon.noname-f con 0x7fed8c139370 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fed8c138730 con 0x7fed8c139370 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fed8c1385b0 con 0x7fed8c0a5bb0 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fed8c143820 con 0x7fed8c138a50 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient(hunting): authenticate will time out at 944.733818s 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 0x7fed8c1d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 0x7fed8c1d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59416/0 (socket says 172.21.15.43:59416) 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c0a5bb0 0x7fed8c1d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 -- 172.21.15.43:0/3790024587 learned_addr learned my addr 172.21.15.43:0/3790024587 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): get_auth_request con 0x7fed8c138a50 auth_method 0 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): get_auth_request con 0x7fed8c0a5bb0 auth_method 0 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8c139370 0x7fed8c134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): get_auth_request con 0x7fed8c139370 auth_method 0 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 -- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c0a5bb0 msgr2=0x7fed8c1d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c0a5bb0 0x7fed8c1d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 -- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8c139370 msgr2=0x7fed8c134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8c139370 0x7fed8c134dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient: _finish_hunting 0 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 monclient: found mon.noname-e 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:34.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 -- 172.21.15.43:0/3790024587 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8c1373c0 con 0x7fed8c138a50 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91058640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c0a5bb0 0x7fed8c1d0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed91859640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8c139370 0x7fed8c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed90857640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 0x7fed8c1d0ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c420 tx=0x7fed7c00c8e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 client.?.objecter ms_handle_connect 0x7fed8c138a50 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed7c0047f0 con 0x7fed8c138a50 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: dump: 2024-04-19T00:44:34.625 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.626 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed7c004990 con 0x7fed8c138a50 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 5 monclient: authenticate success, global_id 4247 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _renew_subs 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed8c09fc90 con 0x7fed8c138a50 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:34.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fed7c0133c0 con 0x7fed8c138a50 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 10 monclient: dump: 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.628 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed837fe640 4 set_mon_vals no callback set 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _renew_subs 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fed8c1382b0 con 0x7fed8c138a50 2024-04-19T00:44:34.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 librados: init done 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7fed7c011070 con 0x7fed8c138a50 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.604+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed8c13a160 con 0x7fed8c138a50 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fed70038d80 0x7fed7003b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(14..14 src has 1..14) ==== 4346+0+0 (secure 0 0 0) 0x7fed7c048760 con 0x7fed8c138a50 2024-04-19T00:44:34.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 10 client.4247.objecter ms_dispatch 0x7fed8c000ba0 osd_map(14..14 src has 1..14) 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 3 client.4247.objecter handle_osd_map got epochs [14,14] > 0 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 3 client.4247.objecter handle_osd_map decoding full epoch 14 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 20 client.4247.objecter dump_active .. 0 homeless 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed91058640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fed70038d80 0x7fed7003b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed91058640 10 monclient: get_auth_request con 0x7fed70038d80 auth_method 0 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed81ffb640 10 client.4247.objecter ms_handle_connect 0x7fed70038d80 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.608+0000 7fed91058640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fed70038d80 0x7fed7003b240 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed88009f50 tx=0x7fed88009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.612+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fed7c01c790 con 0x7fed8c138a50 2024-04-19T00:44:34.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.612+0000 7fed81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.612+0000 7fed81ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed932bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed932bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed932bc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fed8c076190 con 0x7fed8c138a50 2024-04-19T00:44:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed81ffb640 1 -- 172.21.15.43:0/3790024587 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+12255 (secure 0 0 0) 0x7fed7c01c9f0 con 0x7fed8c138a50 2024-04-19T00:44:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.784+0000 7fed81ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:34.787 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:34.788 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":14,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T00:44:34.042780+0000","last_up_change":"2024-04-19T00:44:32.017099+0000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T00:44:32.605528+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T00:44:32.926773+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6804","nonce":110940973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6809","nonce":110940973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6821","nonce":110940973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6815","nonce":110940973}]},"public_addr":"172.21.15.43:6804/110940973","cluster_addr":"172.21.15.43:6809/110940973","heartbeat_back_addr":"172.21.15.43:6821/110940973","heartbeat_front_addr":"172.21.15.43:6815/110940973","state":["exists","up"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6802","nonce":1380723486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6808","nonce":1380723486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6820","nonce":1380723486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6814","nonce":1380723486}]},"public_addr":"172.21.15.43:6802/1380723486","cluster_addr":"172.21.15.43:6808/1380723486","heartbeat_back_addr":"172.21.15.43:6820/1380723486","heartbeat_front_addr":"172.21.15.43:6814/1380723486","state":["exists","up"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6806","nonce":1290935720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6813","nonce":1290935720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6823","nonce":1290935720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6819","nonce":1290935720}]},"public_addr":"172.21.15.43:6806/1290935720","cluster_addr":"172.21.15.43:6813/1290935720","heartbeat_back_addr":"172.21.15.43:6823/1290935720","heartbeat_front_addr":"172.21.15.43:6819/1290935720","state":["exists","up"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6801","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6804","nonce":2204255832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6810","nonce":2204255832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6821","nonce":2204255832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6815","nonce":2204255832}]},"public_addr":"172.21.15.161:6804/2204255832","cluster_addr":"172.21.15.161:6810/2204255832","heartbeat_back_addr":"172.21.15.161:6821/2204255832","heartbeat_front_addr":"172.21.15.161:6815/2204255832","state":["exists","up"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6803","nonce":3653492594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6809","nonce":3653492594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6822","nonce":3653492594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6816","nonce":3653492594}]},"public_addr":"172.21.15.161:6803/3653492594","cluster_addr":"172.21.15.161:6809/3653492594","heartbeat_back_addr":"172.21.15.161:6822/3653492594","heartbeat_front_addr":"172.21.15.161:6816/3653492594","state":["exists","up"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6805","nonce":760479844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6811","nonce":760479844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6823","nonce":760479844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6817","nonce":760479844}]},"public_addr":"172.21.15.161:6805/760479844","cluster_addr":"172.21.15.161:6811/760479844","heartbeat_back_addr":"172.21.15.161:6823/760479844","heartbeat_front_addr":"172.21.15.161:6817/760479844","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.756279+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.840443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.708058+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.848897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.721708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.855016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T00:44:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 10 librados: watch_flush enter 2024-04-19T00:44:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 10 librados: watch_flush exit 2024-04-19T00:44:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 20 client.4247.objecter shutdown clearing up homeless session... 2024-04-19T00:44:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 10 client.4247.objecter successfully canceled tick 2024-04-19T00:44:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fed70038d80 msgr2=0x7fed7003b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fed70038d80 0x7fed7003b240 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed88009f50 tx=0x7fed88009190 comp rx=0 tx=0).stop 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 10 monclient: shutdown 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 msgr2=0x7fed8c1d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 0x7fed8c1d0ed0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00c420 tx=0x7fed7c00c8e0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 shutdown_connections 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fed8c139370 0x7fed8c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fed70038d80 0x7fed7003b240 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fed8c138a50 0x7fed8c1d0ed0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 --2- 172.21.15.43:0/3790024587 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fed8c0a5bb0 0x7fed8c1d0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 >> 172.21.15.43:0/3790024587 conn(0x7fed8c0a7cb0 msgr2=0x7fed8c133a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 shutdown_connections 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 -- 172.21.15.43:0/3790024587 wait complete. 2024-04-19T00:44:34.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.788+0000 7fed932bc640 1 librados: shutdown 2024-04-19T00:44:34.808 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-19T00:44:34.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T00:44:34.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient: get_monmap_and_config 2024-04-19T00:44:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient: build_initial_monmap 2024-04-19T00:44:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient: monmap: 2024-04-19T00:44:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.926 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:34.926582+0000 2024-04-19T00:44:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:34.926582+0000 2024-04-19T00:44:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:34.927 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:34.928 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:34.929 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 Processor -- start 2024-04-19T00:44:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 -- start start 2024-04-19T00:44:34.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient: init 2024-04-19T00:44:34.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:34.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient: _add_conns ranks=[2,8,10,9,6,7,0,5,3,1,4] 2024-04-19T00:44:34.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa7c06c240 0x7faa7c0a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): picked mon.noname-b con 0x7faa7c06c240 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 0x7faa7c0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): picked mon.noname-e con 0x7faa7c0a2f90 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa7c0a4f70 0x7faa7c0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): picked mon.noname-f con 0x7faa7c0a4f70 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7faa7c06c620 con 0x7faa7c0a4f70 2024-04-19T00:44:34.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faa7c1374f0 con 0x7faa7c06c240 2024-04-19T00:44:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7faa7c0722d0 con 0x7faa7c0a2f90 2024-04-19T00:44:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:34.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa811d3640 10 monclient(hunting): authenticate will time out at 945.053496s 2024-04-19T00:44:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa7c06c240 0x7faa7c0a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 0x7faa7c0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 0x7faa7c0a4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59422/0 (socket says 172.21.15.43:59422) 2024-04-19T00:44:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 -- 172.21.15.43:0/517165285 learned_addr learned my addr 172.21.15.43:0/517165285 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa7c06c240 0x7faa7c0a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35232/0 (socket says 172.21.15.43:35232) 2024-04-19T00:44:34.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): get_auth_request con 0x7faa7c0a2f90 auth_method 0 2024-04-19T00:44:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): get_auth_request con 0x7faa7c06c240 auth_method 0 2024-04-19T00:44:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7b59e640 1 --2- 172.21.15.43:0/517165285 >> v2:172.21.15.161:3300/0 conn(0x7faa7c0a4f70 0x7faa7c0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7b59e640 10 monclient(hunting): get_auth_request con 0x7faa7c0a4f70 auth_method 0 2024-04-19T00:44:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7b59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-04-19T00:44:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 -- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa7c06c240 msgr2=0x7faa7c0a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 --2- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa7c06c240 0x7faa7c0a2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 -- 172.21.15.43:0/517165285 >> v2:172.21.15.161:3300/0 conn(0x7faa7c0a4f70 msgr2=0x7faa7c0a5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 --2- 172.21.15.43:0/517165285 >> v2:172.21.15.161:3300/0 conn(0x7faa7c0a4f70 0x7faa7c0a5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient: _finish_hunting 0 2024-04-19T00:44:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 monclient: found mon.noname-e 2024-04-19T00:44:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.924+0000 7faa7a59c640 1 -- 172.21.15.43:0/517165285 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c072f50 con 0x7faa7c0a2f90 2024-04-19T00:44:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 --2- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 0x7faa7c0a4a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa70009870 tx=0x7faa7001c420 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c163c551f4d314ea server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 1 -- 172.21.15.43:0/517165285 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa700b0070 con 0x7faa7c0a2f90 2024-04-19T00:44:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: dump: 2024-04-19T00:44:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: _finish_auth 0 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 1 -- 172.21.15.43:0/517165285 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa700aa810 con 0x7faa7c0a2f90 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 1 -- 172.21.15.43:0/517165285 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa700a9300 con 0x7faa7c0a2f90 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 5 monclient: authenticate success, global_id 4256 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 4 set_mon_vals no callback set 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa79d9b640 10 monclient: dump: 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: shutdown 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 msgr2=0x7faa7c0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 0x7faa7c0a4a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7faa70009870 tx=0x7faa7001c420 comp rx=0 tx=0).stop 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- 172.21.15.43:0/517165285 shutdown_connections 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- 172.21.15.43:0/517165285 >> v2:172.21.15.161:3300/0 conn(0x7faa7c0a4f70 0x7faa7c0a5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faa7c0a2f90 0x7faa7c0a4a30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- 172.21.15.43:0/517165285 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faa7c06c240 0x7faa7c0a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- 172.21.15.43:0/517165285 >> 172.21.15.43:0/517165285 conn(0x7faa7c0a62e0 msgr2=0x7faa7c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- 172.21.15.43:0/517165285 shutdown_connections 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- 172.21.15.43:0/517165285 wait complete. 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: build_initial_monmap 2024-04-19T00:44:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: monmap: 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 librados: starting msgr at 2024-04-19T00:44:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 librados: starting objecter 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 Processor -- start 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- start start 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 librados: setting wanted keys 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 librados: calling monclient init 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: init 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient: _add_conns ranks=[0,6,7,3,2,4,8,5,1] 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 0x7faa7c1d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): picked mon.noname-a con 0x7faa7c06c240 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 0x7faa7c1d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): picked mon.noname-d con 0x7faa7c0a2f90 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faa7c0a4f70 0x7faa7c1d1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): picked mon.noname-i con 0x7faa7c0a4f70 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faa7c071d00 con 0x7faa7c06c240 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7faa7c071b80 con 0x7faa7c0a2f90 2024-04-19T00:44:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7faa7c140510 con 0x7faa7c0a4f70 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa811d3640 10 monclient(hunting): authenticate will time out at 945.059246s 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 0x7faa7c1d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 0x7faa7c1d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 0x7faa7c1d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54144/0 (socket says 172.21.15.43:54144) 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 0x7faa7c1d0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42158/0 (socket says 172.21.15.43:42158) 2024-04-19T00:44:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 1 -- 172.21.15.43:0/1537997376 learned_addr learned my addr 172.21.15.43:0/1537997376 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): get_auth_request con 0x7faa7c06c240 auth_method 0 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): get_auth_request con 0x7faa7c0a2f90 auth_method 0 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient(hunting): handle_auth_done global_id 4209 payload 824 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 -- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faa7c0a4f70 msgr2=0x7faa7c1d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faa7c0a4f70 0x7faa7c1d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 -- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 msgr2=0x7faa7c1d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 0x7faa7c1d05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient: _finish_hunting 0 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 monclient: found mon.noname-d 2024-04-19T00:44:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 -- 172.21.15.43:0/1537997376 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa7c06e3a0 con 0x7faa7c0a2f90 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7ad9d640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 0x7faa7c1d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa7a59c640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 0x7faa7c1d0b00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa700ab000 tx=0x7faa700191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.928+0000 7faa637fe640 10 client.?.objecter ms_handle_connect 0x7faa7c0a2f90 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa700b9070 con 0x7faa7c0a2f90 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: dump: 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa700044b0 con 0x7faa7c0a2f90 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faa700b0040 con 0x7faa7c0a2f90 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa7959a640 4 set_mon_vals no callback set 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 monclient: dump: 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 5 monclient: authenticate success, global_id 4209 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 monclient: _renew_subs 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7c0724f0 con 0x7faa7c0a2f90 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 monclient: _renew_subs 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7c06e0c0 con 0x7faa7c0a2f90 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 1 librados: init done 2024-04-19T00:44:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7faa700aad10 con 0x7faa7c0a2f90 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa7c13c5c0 con 0x7faa7c0a2f90 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faa68038d80 0x7faa6803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(14..14 src has 1..14) ==== 4346+0+0 (secure 0 0 0) 0x7faa700e1f50 con 0x7faa7c0a2f90 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 client.4209.objecter ms_dispatch 0x7faa7c000ba0 osd_map(14..14 src has 1..14) 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 3 client.4209.objecter handle_osd_map got epochs [14,14] > 0 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 3 client.4209.objecter handle_osd_map decoding full epoch 14 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 20 client.4209.objecter dump_active .. 0 homeless 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa7ad9d640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faa68038d80 0x7faa6803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa7ad9d640 10 monclient: get_auth_request con 0x7faa68038d80 auth_method 0 2024-04-19T00:44:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa7ad9d640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faa68038d80 0x7faa6803b240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009f70 tx=0x7faa6c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.932+0000 7faa637fe640 10 client.4209.objecter ms_handle_connect 0x7faa68038d80 2024-04-19T00:44:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.936+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faa70016130 con 0x7faa7c0a2f90 2024-04-19T00:44:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.936+0000 7faa637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:34.936+0000 7faa637fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa811d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:35.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa811d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:35.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa811d3640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa7c13c160 con 0x7faa7c0a2f90 2024-04-19T00:44:35.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa637fe640 1 -- 172.21.15.43:0/1537997376 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12229 (secure 0 0 0) 0x7faa700abc00 con 0x7faa7c0a2f90 2024-04-19T00:44:35.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:35.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.116+0000 7faa637fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:35.121 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:35.121 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":15,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T00:44:35.054176+0000","last_up_change":"2024-04-19T00:44:32.017099+0000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T00:44:32.605528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T00:44:32.926773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6804","nonce":110940973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6809","nonce":110940973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6821","nonce":110940973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6815","nonce":110940973}]},"public_addr":"172.21.15.43:6804/110940973","cluster_addr":"172.21.15.43:6809/110940973","heartbeat_back_addr":"172.21.15.43:6821/110940973","heartbeat_front_addr":"172.21.15.43:6815/110940973","state":["exists","up"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6802","nonce":1380723486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6808","nonce":1380723486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6820","nonce":1380723486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6814","nonce":1380723486}]},"public_addr":"172.21.15.43:6802/1380723486","cluster_addr":"172.21.15.43:6808/1380723486","heartbeat_back_addr":"172.21.15.43:6820/1380723486","heartbeat_front_addr":"172.21.15.43:6814/1380723486","state":["exists","up"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6806","nonce":1290935720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6813","nonce":1290935720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6823","nonce":1290935720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6819","nonce":1290935720}]},"public_addr":"172.21.15.43:6806/1290935720","cluster_addr":"172.21.15.43:6813/1290935720","heartbeat_back_addr":"172.21.15.43:6823/1290935720","heartbeat_front_addr":"172.21.15.43:6819/1290935720","state":["exists","up"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6801","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6804","nonce":2204255832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6810","nonce":2204255832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6821","nonce":2204255832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6815","nonce":2204255832}]},"public_addr":"172.21.15.161:6804/2204255832","cluster_addr":"172.21.15.161:6810/2204255832","heartbeat_back_addr":"172.21.15.161:6821/2204255832","heartbeat_front_addr":"172.21.15.161:6815/2204255832","state":["exists","up"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6803","nonce":3653492594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6809","nonce":3653492594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6822","nonce":3653492594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6816","nonce":3653492594}]},"public_addr":"172.21.15.161:6803/3653492594","cluster_addr":"172.21.15.161:6809/3653492594","heartbeat_back_addr":"172.21.15.161:6822/3653492594","heartbeat_front_addr":"172.21.15.161:6816/3653492594","state":["exists","up"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6805","nonce":760479844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6811","nonce":760479844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6823","nonce":760479844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6817","nonce":760479844}]},"public_addr":"172.21.15.161:6805/760479844","cluster_addr":"172.21.15.161:6811/760479844","heartbeat_back_addr":"172.21.15.161:6823/760479844","heartbeat_front_addr":"172.21.15.161:6817/760479844","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.756279+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.840443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.708058+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.848897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.721708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.855016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T00:44:35.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 10 librados: watch_flush enter 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 10 librados: watch_flush exit 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 20 client.4209.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 10 client.4209.objecter successfully canceled tick 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faa68038d80 msgr2=0x7faa6803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faa68038d80 0x7faa6803b240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009f70 tx=0x7faa6c009190 comp rx=0 tx=0).stop 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 10 monclient: shutdown 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 msgr2=0x7faa7c1d0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 0x7faa7c1d0b00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7faa700ab000 tx=0x7faa700191c0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 shutdown_connections 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faa68038d80 0x7faa6803b240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faa7c0a4f70 0x7faa7c1d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faa7c0a2f90 0x7faa7c1d0b00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 --2- 172.21.15.43:0/1537997376 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faa7c06c240 0x7faa7c1d05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 >> 172.21.15.43:0/1537997376 conn(0x7faa7c0a62e0 msgr2=0x7faa7c133100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 shutdown_connections 2024-04-19T00:44:35.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 -- 172.21.15.43:0/1537997376 wait complete. 2024-04-19T00:44:35.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.120+0000 7faa811d3640 1 librados: shutdown 2024-04-19T00:44:35.140 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-19T00:44:35.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-19T00:44:35.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-19T00:44:35.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-19T00:44:35.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-19T00:44:35.142 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-19T00:44:35.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient: monmap: 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.263311+0000 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.263311+0000 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.262 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.263 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient: monmap: 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.263267+0000 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.263267+0000 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.264 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 Processor -- start 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 -- start start 2024-04-19T00:44:35.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient: init 2024-04-19T00:44:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 1 Processor -- start 2024-04-19T00:44:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 1 -- start start 2024-04-19T00:44:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient: init 2024-04-19T00:44:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient: _add_conns ranks=[3,5,2,6,4,8,10,9,1,7,0] 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient: _add_conns ranks=[0,7,1,3,9,6,10,2,5,8,4] 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49540a1f20 0x7f49540a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): picked mon.noname-g con 0x7f49540a1f20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f49540a1270 0x7f49540a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): picked mon.noname-h con 0x7f49540a1270 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f495413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa53c138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c139540 0x7fa53c13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient(hunting): picked mon.noname-i con 0x7fa53c139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa53c0a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa53c0a0d40 con 0x7fa53c138a50 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa53c0a0a60 con 0x7fa53c0a5bb0 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa53c0a1020 con 0x7fa53c139540 2024-04-19T00:44:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa5423ed640 10 monclient(hunting): authenticate will time out at 945.390426s 2024-04-19T00:44:35.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f495413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f495413fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35242/0 (socket says 172.21.15.43:35242) 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 -- 172.21.15.43:0/1764043549 learned_addr learned my addr 172.21.15.43:0/1764043549 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): picked mon.noname-b con 0x7f49540a4430 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f49540a0920 con 0x7f49540a4430 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f495406bf40 con 0x7f49540a1f20 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f495409fff0 con 0x7f49540a1270 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4959bf9640 10 monclient(hunting): authenticate will time out at 945.390711s 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): get_auth_request con 0x7f49540a4430 auth_method 0 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53bfff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa53c138a50 0x7fa53c138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44750/0 (socket says 172.21.15.43:44750) 2024-04-19T00:44:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53bfff640 1 -- 172.21.15.43:0/2193729337 learned_addr learned my addr 172.21.15.43:0/2193729337 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:35.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f49540a1270 0x7f49540a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): get_auth_request con 0x7f49540a1270 auth_method 0 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 1 --2- 172.21.15.43:0/2193729337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c139540 0x7fa53c13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): get_auth_request con 0x7fa53c139540 auth_method 0 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f49537fe640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49540a1f20 0x7f49540a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f49537fe640 10 monclient(hunting): get_auth_request con 0x7f49540a1f20 auth_method 0 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f49537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f49537fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f49537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4952ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 -- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f49540a1270 msgr2=0x7f49540a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f49540a1270 0x7f49540a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 -- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49540a1f20 msgr2=0x7f49540a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49540a1f20 0x7f49540a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 monclient: found mon.noname-b 2024-04-19T00:44:35.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7f4953fff640 1 -- 172.21.15.43:0/1764043549 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49540a02d0 con 0x7f49540a4430 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108698685 0 0) 0x7fa53c0a0d40 con 0x7fa53c138a50 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): _negotiate 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): my global_id is 4257 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa524003530 con 0x7fa53c138a50 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230130768 0 0) 0x7fa53c0a0a60 con 0x7fa53c0a5bb0 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): _negotiate 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient(hunting): my global_id is 4249 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa53c0a0d40 con 0x7fa53c0a5bb0 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 1 -- 172.21.15.43:0/2193729337 >> v1:172.21.15.161:6789/0 conn(0x7fa53c0a5bb0 legacy=0x7fa53c140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 1 -- 172.21.15.43:0/2193729337 >> v1:172.21.15.43:6789/0 conn(0x7fa53c138a50 legacy=0x7fa53c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3268685073 0 0) 0x7fa524003530 con 0x7fa53c138a50 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 1 monclient: found mon.noname-i 2024-04-19T00:44:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53b7fe640 1 -- 172.21.15.43:0/2193729337 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c0a17b0 con 0x7fa53c139540 2024-04-19T00:44:35.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.260+0000 7fa53affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4953fff640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f495413fba0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f494c009e80 tx=0x7f494c01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bc8e7f836cb9f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 1 -- 172.21.15.43:0/1764043549 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f494c0090d0 con 0x7f49540a4430 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: dump: 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.279 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 1 -- 172.21.15.43:0/1764043549 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494c0a6510 con 0x7f49540a4430 2024-04-19T00:44:35.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 1 -- 172.21.15.43:0/1764043549 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f494c009270 con 0x7f49540a4430 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f49527fc640 10 monclient: dump: 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.281 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 5 monclient: authenticate success, global_id 4250 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 4 set_mon_vals no callback set 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 10 monclient: shutdown 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 msgr2=0x7f495413fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f495413fba0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f494c009e80 tx=0x7f494c01c910 comp rx=0 tx=0).stop 2024-04-19T00:44:35.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 shutdown_connections 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f495413fba0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f49540a1270 0x7f49540a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1764043549 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f49540a1f20 0x7f49540a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 >> 172.21.15.43:0/1764043549 conn(0x7f49540a85a0 msgr2=0x7f49540a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 shutdown_connections 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 -- 172.21.15.43:0/1764043549 wait complete. 2024-04-19T00:44:35.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53b7fe640 1 --2- 172.21.15.43:0/2193729337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c139540 0x7fa53c13dfe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0097f0 tx=0x7fa52c01c2c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=12d138f2009ea0e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa52c0b0070 con 0x7fa53c139540 2024-04-19T00:44:35.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 10 monclient: monmap: 2024-04-19T00:44:35.284 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.285 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: dump: 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.286 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa5423ed640 5 monclient: authenticate success, global_id 4282 2024-04-19T00:44:35.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa5423ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:35.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 librados: starting msgr at 2024-04-19T00:44:35.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 librados: starting objecter 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 Processor -- start 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 -- start start 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 librados: setting wanted keys 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 1 librados: calling monclient init 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7f4959bf9640 10 monclient: init 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _add_conns ranks=[1,4,2,3,6,7,0,8,5] 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49540a1270 0x7f495413a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): picked mon.noname-f con 0x7f49540a1270 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f49540a1f20 0x7f4954122a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): picked mon.noname-c con 0x7f49540a1f20 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f4954122f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): picked mon.noname-b con 0x7f49540a4430 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49540a2480 con 0x7f49540a1270 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f49540a2300 con 0x7f49540a4430 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4954142900 con 0x7f49540a1f20 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient(hunting): authenticate will time out at 945.396440s 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f4954122f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f4954122f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35254/0 (socket says 172.21.15.43:35254) 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 -- 172.21.15.43:0/1496805217 learned_addr learned my addr 172.21.15.43:0/1496805217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): get_auth_request con 0x7f49540a4430 auth_method 0 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49540a1270 0x7f495413a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c0aa3a0 con 0x7fa53c139540 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 1 -- 172.21.15.43:0/2193729337 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa530001970 con 0x7fa53c139540 2024-04-19T00:44:35.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa53affd640 10 monclient: dump: 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.292 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa5423ed640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.264+0000 7fa5423ed640 4 set_mon_vals no callback set 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient: shutdown 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- 172.21.15.43:0/2193729337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c139540 msgr2=0x7fa53c13dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 --2- 172.21.15.43:0/2193729337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c139540 0x7fa53c13dfe0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0097f0 tx=0x7fa52c01c2c0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- 172.21.15.43:0/2193729337 shutdown_connections 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 --2- 172.21.15.43:0/2193729337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c139540 0x7fa53c13dfe0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- 172.21.15.43:0/2193729337 >> 172.21.15.43:0/2193729337 conn(0x7fa53c0a7cb0 msgr2=0x7fa53c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): get_auth_request con 0x7f49540a1270 auth_method 0 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f49540a1f20 0x7f4954122a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): get_auth_request con 0x7f49540a1f20 auth_method 0 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- 172.21.15.43:0/2193729337 shutdown_connections 2024-04-19T00:44:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 -- 172.21.15.43:0/1496805217 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f49540a1f20 msgr2=0x7f4954122a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f49540a1f20 0x7f4954122a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 -- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49540a1270 msgr2=0x7f495413a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49540a1270 0x7f495413a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 monclient: found mon.noname-b 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 -- 172.21.15.43:0/1496805217 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49541d1660 con 0x7f49540a4430 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4953fff640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f4954122f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f494c009040 tx=0x7f494c0196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 client.?.objecter ms_handle_connect 0x7f49540a4430 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f494c0be070 con 0x7f49540a4430 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: dump: 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.297 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 5 monclient: authenticate success, global_id 4259 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _renew_subs 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49541262c0 con 0x7f49540a4430 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494c003bb0 con 0x7f49540a4430 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f494c009270 con 0x7f49540a4430 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- 172.21.15.43:0/2193729337 wait complete. 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient: monmap: 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.300 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 librados: starting msgr at 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 librados: starting objecter 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 Processor -- start 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- start start 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 librados: setting wanted keys 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 librados: calling monclient init 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient: init 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient: _add_conns ranks=[7,4,8,0,1,5,6,2,3] 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c0a5bb0 0x7fa53c12cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): picked mon.noname-i con 0x7fa53c0a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa53c138a50 0x7fa53c12d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): picked mon.noname-c con 0x7fa53c138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa53c139540 0x7fa53c09be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: dump: 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.302 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _renew_subs 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f495409f6c0 con 0x7f49540a4430 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4951ffb640 4 set_mon_vals no callback set 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 librados: init done 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 client.4259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 15 client.4259.objecter _session_command_op_assign -1 1 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 20 client.4259.objecter _get_session osd=-1 returning homeless 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f495409ef70 con 0x7f49540a4430 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f494c004040 con 0x7f49540a4430 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4940038da0 0x7f494003b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f494c0b5030 con 0x7f49540a4430 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 client.4259.objecter ms_dispatch 0x7f4954000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 3 client.4259.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 3 client.4259.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 client.4259.objecter checking command 1 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7f494003ec10 0x7f4940041030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 client.4259.objecter _get_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 15 client.4259.objecter _session_command_op_remove -1 1 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 client.4259.objecter _get_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 client.4259.objecter get_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 15 client.4259.objecter _session_command_op_assign 4 1 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 4 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 client.4259.objecter _send_command 1 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 --> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49400416e0 con 0x7f494003ec10 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 20 client.4259.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f494c0e5480 con 0x7f49540a4430 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f493bfff640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T00:44:35.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7f494003ec10 0x7f4940041030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4940038da0 0x7f494003b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f4952ffd640 10 monclient: get_auth_request con 0x7f494003ec10 auth_method 0 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7f49537fe640 10 monclient: get_auth_request con 0x7f4940038da0 auth_method 0 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): picked mon.noname-e con 0x7fa53c139540 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa53c144ca0 con 0x7fa53c138a50 2024-04-19T00:44:35.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa53c0a0780 con 0x7fa53c0a5bb0 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa53c144e20 con 0x7fa53c139540 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa5423ed640 10 monclient(hunting): authenticate will time out at 945.399656s 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa53b7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa53c138a50 0x7fa53c12d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.268+0000 7fa54098a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa53c139540 0x7fa53c09be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53bfff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c0a5bb0 0x7fa53c12cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa53c138a50 0x7fa53c12d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48498/0 (socket says 172.21.15.43:48498) 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 -- 172.21.15.43:0/484828064 learned_addr learned my addr 172.21.15.43:0/484828064 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa54098a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa53c139540 0x7fa53c09be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59436/0 (socket says 172.21.15.43:59436) 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): get_auth_request con 0x7fa53c138a50 auth_method 0 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f4952ffd640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7f494003ec10 0x7f4940041030 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 10 client.4259.objecter ms_handle_connect 0x7f494003ec10 2024-04-19T00:44:35.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 10 client.4259.objecter ms_handle_connect 0x7f4940038da0 2024-04-19T00:44:35.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f49537fe640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4940038da0 0x7f494003b260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49440079e0 tx=0x7f4944008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient(hunting): handle_auth_done global_id 4234 payload 824 2024-04-19T00:44:35.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 -- 172.21.15.43:0/484828064 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa53c139540 msgr2=0x7fa53c09be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa53c139540 0x7fa53c09be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 -- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c0a5bb0 msgr2=0x7fa53c12cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c0a5bb0 0x7fa53c12cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 monclient: found mon.noname-c 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 -- 172.21.15.43:0/484828064 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53c09d6c0 con 0x7fa53c138a50 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa54098a640 10 monclient: get_auth_request con 0x7fa53c139540 auth_method 0 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53bfff640 10 monclient: get_auth_request con 0x7fa53c0a5bb0 auth_method 0 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa54098a640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa53c139540 0x7fa53c09be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53bfff640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa53c0a5bb0 0x7fa53c12cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53b7fe640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa53c138a50 0x7fa53c12d080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0ae040 tx=0x7fa52c019060 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 client.?.objecter ms_handle_connect 0x7fa53c138a50 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa52c0b0040 con 0x7fa53c138a50 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: dump: 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.316 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa52c003b50 con 0x7fa53c138a50 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa52c003d00 con 0x7fa53c138a50 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa53a7fc640 4 set_mon_vals no callback set 2024-04-19T00:44:35.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa538ff9640 10 monclient: dump: 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.318 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 5 monclient: authenticate success, global_id 4234 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 monclient: _renew_subs 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa53c134a40 con 0x7fa53c138a50 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 monclient: _renew_subs 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa53c134fc0 con 0x7fa53c138a50 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 1 librados: init done 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 client.4234.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 15 client.4234.objecter _session_command_op_assign -1 1 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 20 client.4234.objecter _get_session osd=-1 returning homeless 2024-04-19T00:44:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa53c09e200 con 0x7fa53c138a50 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7fa52c0036e0 con 0x7fa53c138a50 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa524039560 0x7fa52403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fa52c0e2700 con 0x7fa53c138a50 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 10 client.4234.objecter ms_dispatch 0x7fa53c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 3 client.4234.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 3 client.4234.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 10 client.4234.objecter checking command 1 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fa52403f360 0x7fa524041780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 20 client.4234.objecter _get_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 15 client.4234.objecter _session_command_op_remove -1 1 2024-04-19T00:44:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 20 client.4234.objecter _get_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 20 client.4234.objecter get_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 15 client.4234.objecter _session_command_op_assign 3 1 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 4 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 10 client.4234.objecter _send_command 1 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 --> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa52c0a9ec0 con 0x7fa52403f360 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 20 client.4234.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== osd.4 v2:172.21.15.161:6800/3653492594 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f49400416e0 con 0x7f494003ec10 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 10 client.4259.objecter ms_dispatch 0x7f4954000ba0 command_reply(tid 1: 0 ) 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 10 client.4259.objecter _finish_command 1 = system:0 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 4 2024-04-19T00:44:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.272+0000 7f493bfff640 15 client.4259.objecter _session_command_op_remove 4 1 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fa52c0e9050 con 0x7fa53c138a50 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa53bfff640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa524039560 0x7fa52403ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa54098a640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fa52403f360 0x7fa524041780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa53bfff640 10 monclient: get_auth_request con 0x7fa524039560 auth_method 0 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa53bfff640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa524039560 0x7fa52403ba20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa530005890 tx=0x7fa530009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.276+0000 7fa538ff9640 10 client.4234.objecter ms_handle_connect 0x7fa524039560 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa54098a640 10 monclient: get_auth_request con 0x7fa52403f360 auth_method 0 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa538ff9640 10 client.4234.objecter ms_handle_connect 0x7fa52403f360 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa54098a640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fa52403f360 0x7fa524041780 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== osd.3 v2:172.21.15.161:6801/2204255832 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa52c0a9ec0 con 0x7fa52403f360 2024-04-19T00:44:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa538ff9640 10 client.4234.objecter ms_dispatch 0x7fa53c000ba0 command_reply(tid 1: 0 ) 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa538ff9640 10 client.4234.objecter _finish_command 1 = system:0 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa538ff9640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 4 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.280+0000 7fa538ff9640 15 client.4234.objecter _session_command_op_remove 3 1 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 client.4234.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 15 client.4234.objecter _session_command_op_assign -1 2 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter _get_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter _get_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 15 client.4234.objecter _session_command_op_remove -1 2 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter get_session s=0x7fa52403ec90 osd=3 3 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 15 client.4234.objecter _session_command_op_assign 3 2 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 4 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 client.4234.objecter _send_command 2 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 --> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa53c12d5c0 con 0x7fa52403f360 2024-04-19T00:44:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa538ff9640 1 -- 172.21.15.43:0/484828064 <== osd.3 v2:172.21.15.161:6801/2204255832 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa53c12d5c0 con 0x7fa52403f360 2024-04-19T00:44:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa538ff9640 10 client.4234.objecter ms_dispatch 0x7fa53c000ba0 command_reply(tid 2: 0 ) 2024-04-19T00:44:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa538ff9640 10 client.4234.objecter _finish_command 2 = system:0 2024-04-19T00:44:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa538ff9640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 4 2024-04-19T00:44:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa538ff9640 15 client.4234.objecter _session_command_op_remove 3 2 2024-04-19T00:44:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 librados: watch_flush enter 2024-04-19T00:44:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 librados: watch_flush exit 2024-04-19T00:44:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 client.4234.objecter close_session for osd.3 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fa52403f360 msgr2=0x7fa524041780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fa52403f360 0x7fa524041780 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter put_session s=0x7fa52403ec90 osd=3 1 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 client.4234.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 client.4234.objecter successfully canceled tick 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa524039560 msgr2=0x7fa52403ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa524039560 0x7fa52403ba20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa530005890 tx=0x7fa530009040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 10 monclient: shutdown 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa53c138a50 msgr2=0x7fa53c12d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 --2- 172.21.15.43:0/484828064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa53c138a50 0x7fa53c12d080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa52c0ae040 tx=0x7fa52c019060 comp rx=0 tx=0).stop 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa54098a640 1 -- 172.21.15.43:0/484828064 reap_dead start 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 shutdown_connections 2024-04-19T00:44:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 >> 172.21.15.43:0/484828064 conn(0x7fa53c0a7cb0 msgr2=0x7fa53c0a8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 shutdown_connections 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 -- 172.21.15.43:0/484828064 wait complete. 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.296+0000 7fa5423ed640 1 librados: shutdown 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 client.4259.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 15 client.4259.objecter _session_command_op_assign -1 2 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter _get_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter _get_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 15 client.4259.objecter _session_command_op_remove -1 2 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter get_session s=0x7f494003e540 osd=4 3 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 15 client.4259.objecter _session_command_op_assign 4 2 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 4 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 client.4259.objecter _send_command 2 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 --> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4954125f40 con 0x7f494003ec10 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f493bfff640 1 -- 172.21.15.43:0/1496805217 <== osd.4 v2:172.21.15.161:6800/3653492594 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4954125f40 con 0x7f494003ec10 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f493bfff640 10 client.4259.objecter ms_dispatch 0x7f4954000ba0 command_reply(tid 2: 0 ) 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f493bfff640 10 client.4259.objecter _finish_command 2 = system:0 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f493bfff640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 4 2024-04-19T00:44:35.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f493bfff640 15 client.4259.objecter _session_command_op_remove 4 2 2024-04-19T00:44:35.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 librados: watch_flush enter 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 librados: watch_flush exit 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 client.4259.objecter close_session for osd.4 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7f494003ec10 msgr2=0x7f4940041030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7f494003ec10 0x7f4940041030 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter put_session s=0x7f494003e540 osd=4 1 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 client.4259.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 client.4259.objecter successfully canceled tick 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4940038da0 msgr2=0x7f494003b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4940038da0 0x7f494003b260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49440079e0 tx=0x7f4944008040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 10 monclient: shutdown 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 msgr2=0x7f4954122f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 --2- 172.21.15.43:0/1496805217 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f49540a4430 0x7f4954122f40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f494c009040 tx=0x7f494c0196b0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4953fff640 1 -- 172.21.15.43:0/1496805217 reap_dead start 2024-04-19T00:44:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 shutdown_connections 2024-04-19T00:44:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 >> 172.21.15.43:0/1496805217 conn(0x7f49540a85a0 msgr2=0x7f49540a7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 shutdown_connections 2024-04-19T00:44:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 -- 172.21.15.43:0/1496805217 wait complete. 2024-04-19T00:44:35.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.304+0000 7f4959bf9640 1 librados: shutdown 2024-04-19T00:44:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient: monmap: 2024-04-19T00:44:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.312066+0000 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.312066+0000 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 Processor -- start 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 -- start start 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient: init 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient: _add_conns ranks=[5,10,4,6,7,0,9,1,8,3,2] 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f280a1fc0 0x7f6f280a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): picked mon.noname-h con 0x7f6f280a1fc0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6f280a1280 0x7f6f280a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): picked mon.noname-f con 0x7f6f280a1280 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 0x7f6f2813fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): picked mon.noname-c con 0x7f6f280a43c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6f2806c1a0 con 0x7f6f280a1280 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f6f280a08c0 con 0x7f6f280a43c0 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6f280a0300 con 0x7f6f280a1fc0 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2cd1b640 10 monclient(hunting): authenticate will time out at 945.438981s 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 0x7f6f2813fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 0x7f6f2813fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48506/0 (socket says 172.21.15.43:48506) 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 -- 172.21.15.43:0/3524765891 learned_addr learned my addr 172.21.15.43:0/3524765891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): get_auth_request con 0x7f6f280a43c0 auth_method 0 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 1 --2- 172.21.15.43:0/3524765891 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f280a1fc0 0x7f6f280a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 1 --2- 172.21.15.43:0/3524765891 >> v2:172.21.15.161:3300/0 conn(0x7f6f280a1280 0x7f6f280a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): get_auth_request con 0x7f6f280a1fc0 auth_method 0 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): get_auth_request con 0x7f6f280a1280 auth_method 0 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient(hunting): handle_auth_done global_id 4243 payload 274 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 -- 172.21.15.43:0/3524765891 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f280a1fc0 msgr2=0x7f6f280a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 --2- 172.21.15.43:0/3524765891 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f280a1fc0 0x7f6f280a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 -- 172.21.15.43:0/3524765891 >> v2:172.21.15.161:3300/0 conn(0x7f6f280a1280 msgr2=0x7f6f280a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 --2- 172.21.15.43:0/3524765891 >> v2:172.21.15.161:3300/0 conn(0x7f6f280a1280 0x7f6f280a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 monclient: found mon.noname-c 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.308+0000 7f6f26d9d640 1 -- 172.21.15.43:0/3524765891 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f280a05e0 con 0x7f6f280a43c0 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f26d9d640 1 --2- 172.21.15.43:0/3524765891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 0x7f6f2813fb30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009e50 tx=0x7f6f2001c8c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e66c59d30d4cae7c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 1 -- 172.21.15.43:0/3524765891 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f200090d0 con 0x7f6f280a43c0 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: dump: 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 1 -- 172.21.15.43:0/3524765891 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f200a6980 con 0x7f6f280a43c0 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 1 -- 172.21.15.43:0/3524765891 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f200af480 con 0x7f6f280a43c0 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2559a640 10 monclient: dump: 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2cd1b640 5 monclient: authenticate success, global_id 4243 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2cd1b640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.312+0000 7f6f2cd1b640 4 set_mon_vals no callback set 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient: shutdown 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/3524765891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 msgr2=0x7f6f2813fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/3524765891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 0x7f6f2813fb30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6f20009e50 tx=0x7f6f2001c8c0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/3524765891 shutdown_connections 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/3524765891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6f280a43c0 0x7f6f2813fb30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/3524765891 >> v2:172.21.15.161:3300/0 conn(0x7f6f280a1280 0x7f6f280a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/3524765891 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6f280a1fc0 0x7f6f280a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/3524765891 >> 172.21.15.43:0/3524765891 conn(0x7f6f280a85c0 msgr2=0x7f6f280a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/3524765891 shutdown_connections 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/3524765891 wait complete. 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient: monmap: 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 librados: starting msgr at 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 librados: starting objecter 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 Processor -- start 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- start start 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 librados: setting wanted keys 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 librados: calling monclient init 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient: init 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient: _add_conns ranks=[2,3,1,5,0,7,4,6,8] 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6f280a1280 0x7f6f281d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): picked mon.noname-b con 0x7f6f280a1280 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f280a1fc0 0x7f6f2813f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): picked mon.noname-g con 0x7f6f280a1fc0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f280a43c0 0x7f6f2813f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): picked mon.noname-f con 0x7f6f280a43c0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6f280a1b70 con 0x7f6f280a43c0 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6f280a19f0 con 0x7f6f280a1280 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6f28142890 con 0x7f6f280a1fc0 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2cd1b640 10 monclient(hunting): authenticate will time out at 945.447355s 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6f280a1280 0x7f6f281d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6f280a1280 0x7f6f281d4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35260/0 (socket says 172.21.15.43:35260) 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 1 -- 172.21.15.43:0/997081818 learned_addr learned my addr 172.21.15.43:0/997081818 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): get_auth_request con 0x7f6f280a1280 auth_method 0 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f25d9b640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f280a1fc0 0x7f6f2813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f25d9b640 10 monclient(hunting): get_auth_request con 0x7f6f280a1fc0 auth_method 0 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f25d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f25d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.316+0000 7f6f25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f25d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f25d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f25d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 -- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f280a1fc0 msgr2=0x7f6f2813f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6f280a1fc0 0x7f6f2813f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 -- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f280a43c0 msgr2=0x7f6f2813f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6f280a43c0 0x7f6f2813f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 monclient: found mon.noname-b 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 -- 172.21.15.43:0/997081818 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f2813c580 con 0x7f6f280a1280 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2659c640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6f280a1280 0x7f6f281d4e40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800e930 tx=0x7f6f1800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.?.objecter ms_handle_connect 0x7f6f280a1280 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f18017070 con 0x7f6f280a1280 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: dump: 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f18013470 con 0x7f6f280a1280 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6f18013a90 con 0x7f6f280a1280 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 5 monclient: authenticate success, global_id 4268 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 monclient: _renew_subs 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f2813bec0 con 0x7f6f280a1280 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f24d99640 4 set_mon_vals no callback set 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 monclient: _renew_subs 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f2813d750 con 0x7f6f280a1280 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 1 librados: init done 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: dump: 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.355 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 client.4268.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f6f1801a070 con 0x7f6f280a1280 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 15 client.4268.objecter _session_command_op_assign -1 1 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 20 client.4268.objecter _get_session osd=-1 returning homeless 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f08038d80 0x7f6f0803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f6f1800f040 con 0x7f6f280a1280 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f2813fcc0 con 0x7f6f280a1280 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.4268.objecter ms_dispatch 0x7f6f28000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 3 client.4268.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 3 client.4268.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.4268.objecter checking command 1 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7f6f0803eb80 0x7f6f08040fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 client.4268.objecter _get_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 15 client.4268.objecter _session_command_op_remove -1 1 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 client.4268.objecter _get_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 client.4268.objecter get_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 15 client.4268.objecter _session_command_op_assign 5 1 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 4 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.4268.objecter _send_command 1 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 --> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f18022e50 con 0x7f6f0803eb80 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 20 client.4268.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f6f1801b3b0 con 0x7f6f280a1280 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f25d9b640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f08038d80 0x7f6f0803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f25d9b640 10 monclient: get_auth_request con 0x7f6f08038d80 auth_method 0 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f26d9d640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7f6f0803eb80 0x7f6f08040fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f26d9d640 10 monclient: get_auth_request con 0x7f6f0803eb80 auth_method 0 2024-04-19T00:44:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f25d9b640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f08038d80 0x7f6f0803b240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6f2809e8a0 tx=0x7f6f1c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.4268.objecter ms_handle_connect 0x7f6f08038d80 2024-04-19T00:44:35.359 INFO:teuthology.orchestra.run.smithi043.stdout:51539607556 2024-04-19T00:44:35.360 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T00:44:35.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f16ffd640 10 client.4268.objecter ms_handle_connect 0x7f6f0803eb80 2024-04-19T00:44:35.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.320+0000 7f6f26d9d640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7f6f0803eb80 0x7f6f08040fa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient: monmap: 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.327103+0000 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.327103+0000 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.362 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 Processor -- start 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 -- start start 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient: init 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient: _add_conns ranks=[0,8,7,10,1,3,2,4,5,6,9] 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ef80a2a80 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ef8136ed0 0x7f0ef813bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): picked mon.noname-e con 0x7f0ef8136ed0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef81372f0 0x7f0ef813e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): picked mon.noname-i con 0x7f0ef81372f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef8072fd0 con 0x7f0ef80a2a80 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== osd.5 v2:172.21.15.161:6802/760479844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f18022e50 con 0x7f6f0803eb80 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f6f16ffd640 10 client.4268.objecter ms_dispatch 0x7f6f28000ba0 command_reply(tid 1: 0 ) 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f6f16ffd640 10 client.4268.objecter _finish_command 1 = system:0 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f6f16ffd640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 4 2024-04-19T00:44:35.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f6f16ffd640 15 client.4268.objecter _session_command_op_remove 5 1 2024-04-19T00:44:35.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef7fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0ef80a2a80 0x7f0ef80a2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44752/0 (socket says 172.21.15.43:44752) 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef7fff640 1 -- 172.21.15.43:0/538741564 learned_addr learned my addr 172.21.15.43:0/538741564 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ef8136ed0 0x7f0ef813bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0ef80a3570 con 0x7f0ef81372f0 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0ef80732b0 con 0x7f0ef8136ed0 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efe491640 10 monclient(hunting): authenticate will time out at 945.454353s 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef81372f0 0x7f0ef813e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request con 0x7f0ef8136ed0 auth_method 0 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): get_auth_request con 0x7f0ef81372f0 auth_method 0 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef6ffd640 1 -- 172.21.15.43:0/538741564 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 521846495 0 0) 0x7f0ef8072fd0 con 0x7f0ef80a2a80 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0efca2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef6ffd640 10 monclient(hunting): _negotiate 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef6ffd640 10 monclient(hunting): my global_id is 4266 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef6ffd640 1 -- 172.21.15.43:0/538741564 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ee0003530 con 0x7f0ef80a2a80 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 1 -- 172.21.15.43:0/538741564 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef81372f0 msgr2=0x7f0ef813e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef81372f0 0x7f0ef813e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 1 -- 172.21.15.43:0/538741564 >> v1:172.21.15.43:6789/0 conn(0x7f0ef80a2a80 legacy=0x7f0ef80a2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.324+0000 7f0ef77fe640 1 -- 172.21.15.43:0/538741564 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef8073590 con 0x7f0ef8136ed0 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient: monmap: 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.330966+0000 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.330966+0000 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.369 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 Processor -- start 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 -- start start 2024-04-19T00:44:35.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient: init 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef77fe640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ef8136ed0 0x7f0ef813bbe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0097f0 tx=0x7f0eec01c360 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c7f2b5907f14cadb server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient: _add_conns ranks=[1,5,8,3,6,7,2,10,9,0,4] 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f484c0a1fc0 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1280 0x7f484c0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): picked mon.noname-h con 0x7f484c0a1280 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 0x7f484c140f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): picked mon.noname-e con 0x7f484c13eb30 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f484c0a05e0 con 0x7f484c0a1fc0 2024-04-19T00:44:35.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f484c06c1a0 con 0x7f484c0a1280 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f484c0a08c0 con 0x7f484c13eb30 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4853bb9640 10 monclient(hunting): authenticate will time out at 945.457970s 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient: monmap: 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.372 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.332394+0000 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.332394+0000 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.373 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 1 -- 172.21.15.43:0/538741564 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0eec0b0070 con 0x7f0ef8136ed0 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: dump: 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.374 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 1 -- 172.21.15.43:0/538741564 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec0aa810 con 0x7f0ef8136ed0 2024-04-19T00:44:35.376 INFO:teuthology.orchestra.run.smithi043.stdout:51539607557 2024-04-19T00:44:35.376 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T00:44:35.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 0x7f484c140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 5 monclient: authenticate success, global_id 4274 2024-04-19T00:44:35.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 0x7f484c140f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59460/0 (socket says 172.21.15.43:59460) 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 -- 172.21.15.43:0/729890769 learned_addr learned my addr 172.21.15.43:0/729890769 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): get_auth_request con 0x7f484c13eb30 auth_method 0 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4851154640 1 --2- 172.21.15.43:0/729890769 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1280 0x7f484c0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 1 -- 172.21.15.43:0/538741564 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0eec0a9300 con 0x7f0ef8136ed0 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0ef6ffd640 10 monclient: dump: 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.379 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4851154640 10 monclient(hunting): get_auth_request con 0x7f484c0a1280 auth_method 0 2024-04-19T00:44:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4851154640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4851154640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4851154640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 4 set_mon_vals no callback set 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 -- 172.21.15.43:0/729890769 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1280 msgr2=0x7f484c0a4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 --2- 172.21.15.43:0/729890769 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1280 0x7f484c0a4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 -- 172.21.15.43:0/729890769 >> v1:172.21.15.161:6789/0 conn(0x7f484c0a1fc0 legacy=0x7f484c0a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f4852156640 1 -- 172.21.15.43:0/729890769 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c0a0020 con 0x7f484c13eb30 2024-04-19T00:44:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 Processor -- start 2024-04-19T00:44:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 -- start start 2024-04-19T00:44:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient: init 2024-04-19T00:44:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 10 monclient: shutdown 2024-04-19T00:44:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ef8136ed0 msgr2=0x7f0ef813bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ef8136ed0 0x7f0ef813bbe0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0097f0 tx=0x7f0eec01c360 comp rx=0 tx=0).stop 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 shutdown_connections 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef81372f0 0x7f0ef813e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 --2- 172.21.15.43:0/538741564 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ef8136ed0 0x7f0ef813bbe0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 >> 172.21.15.43:0/538741564 conn(0x7f0ef80a6410 msgr2=0x7f0ef806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 shutdown_connections 2024-04-19T00:44:35.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f0efe491640 1 -- 172.21.15.43:0/538741564 wait complete. 2024-04-19T00:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient: _add_conns ranks=[2,7,0,5,6,4,3,1,10,8,9] 2024-04-19T00:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 0x7f3f04136e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): picked mon.noname-b con 0x7f3f040a3050 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3f040a3430 0x7f3f040a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): picked mon.noname-i con 0x7f3f040a3430 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f0409a0d0 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f04076730 con 0x7f3f0409a0d0 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3f04076480 con 0x7f3f040a3050 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3f0409dd20 con 0x7f3f040a3430 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f09870640 10 monclient(hunting): authenticate will time out at 945.459851s 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 0x7f3f04136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 0x7f3f04136e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35270/0 (socket says 172.21.15.43:35270) 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 1 -- 172.21.15.43:0/1258176859 learned_addr learned my addr 172.21.15.43:0/1258176859 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 10 monclient(hunting): get_auth_request con 0x7f3f040a3050 auth_method 0 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.328+0000 7f3f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f027fc640 1 --2- 172.21.15.43:0/1258176859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3f040a3430 0x7f3f040a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: monmap: 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.388 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f027fc640 10 monclient(hunting): get_auth_request con 0x7f3f040a3430 auth_method 0 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f027fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f027fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 1 -- 172.21.15.43:0/1258176859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3f040a3430 msgr2=0x7f3f040a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 1 --2- 172.21.15.43:0/1258176859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3f040a3430 0x7f3f040a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 1 -- 172.21.15.43:0/1258176859 >> v1:172.21.15.43:6789/0 conn(0x7f3f0409a0d0 legacy=0x7f3f0409a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 1 monclient: found mon.noname-b 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 1 -- 172.21.15.43:0/1258176859 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0409ec70 con 0x7f3f040a3050 2024-04-19T00:44:35.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4852156640 1 --2- 172.21.15.43:0/729890769 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 0x7f484c140f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4844009e80 tx=0x7f484401c8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e1929e9a3b8e6008 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 librados: starting msgr at 2024-04-19T00:44:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 librados: starting objecter 2024-04-19T00:44:35.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 Processor -- start 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 -- start start 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 librados: setting wanted keys 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 librados: calling monclient init 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: init 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: _add_conns ranks=[3,7,6,0,2,8,1,5,4] 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ef80a2a80 0x7f0ef811f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): picked mon.noname-g con 0x7f0ef80a2a80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef8136ed0 0x7f0ef809e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): picked mon.noname-i con 0x7f0ef8136ed0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ef81372f0 0x7f0ef8135a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): picked mon.noname-d con 0x7f0ef81372f0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef8141220 con 0x7f0ef80a2a80 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0ef81410a0 con 0x7f0ef81372f0 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0ef81413a0 con 0x7f0ef8136ed0 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient(hunting): authenticate will time out at 945.461342s 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ef81372f0 0x7f0ef8135a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ef81372f0 0x7f0ef8135a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42164/0 (socket says 172.21.15.43:42164) 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 -- 172.21.15.43:0/145073627 learned_addr learned my addr 172.21.15.43:0/145073627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): get_auth_request con 0x7f0ef81372f0 auth_method 0 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef77fe640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef8136ed0 0x7f0ef809e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request con 0x7f0ef8136ed0 auth_method 0 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef7fff640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ef80a2a80 0x7f0ef811f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient(hunting): handle_auth_done global_id 4218 payload 824 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 -- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef8136ed0 msgr2=0x7f0ef809e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ef8136ed0 0x7f0ef809e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 -- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ef80a2a80 msgr2=0x7f0ef811f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ef80a2a80 0x7f0ef811f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 monclient: found mon.noname-d 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 -- 172.21.15.43:0/145073627 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef809f4b0 con 0x7f0ef81372f0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef7fff640 10 monclient: get_auth_request con 0x7f0ef80a2a80 auth_method 0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef7fff640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ef80a2a80 0x7f0ef811f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efca2e640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ef81372f0 0x7f0ef8135a30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009cb0 tx=0x7f0ef000c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f0ef81372f0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ef0039070 con 0x7f0ef81372f0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: dump: 2024-04-19T00:44:35.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 5 monclient: authenticate success, global_id 4218 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: _renew_subs 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 -- 172.21.15.43:0/145073627 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef8121b40 con 0x7f0ef81372f0 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef000cd30 con 0x7f0ef81372f0 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ef0035860 con 0x7f0ef81372f0 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: dump: 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.398 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef67fc640 4 set_mon_vals no callback set 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: _renew_subs 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 -- 172.21.15.43:0/145073627 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef8120770 con 0x7f0ef81372f0 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0efe491640 1 librados: init done 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 1 -- 172.21.15.43:0/729890769 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48440090d0 con 0x7f484c13eb30 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: dump: 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.399 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 1 -- 172.21.15.43:0/729890769 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48440a6970 con 0x7f484c13eb30 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 1 -- 172.21.15.43:0/729890769 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f48440af480 con 0x7f484c13eb30 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4850953640 10 monclient: dump: 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.401 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f02ffd640 1 --2- 172.21.15.43:0/1258176859 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 0x7f3f04136e40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40098a0 tx=0x7f3ef401c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b833e34acc51ded1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f0ef003c070 con 0x7f0ef81372f0 2024-04-19T00:44:35.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 1 -- 172.21.15.43:0/1258176859 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ef40b2070 con 0x7f3f040a3050 2024-04-19T00:44:35.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:35.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: dump: 2024-04-19T00:44:35.403 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.404 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 1 -- 172.21.15.43:0/1258176859 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef40a9070 con 0x7f3f040a3050 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 1 -- 172.21.15.43:0/1258176859 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ef40abc40 con 0x7f3f040a3050 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ede7fc640 10 client.4218.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ede7fc640 15 client.4218.objecter _session_command_op_assign -1 1 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ede7fc640 20 client.4218.objecter _get_session osd=-1 returning homeless 2024-04-19T00:44:35.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ede7fc640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ede7fc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f01ffb640 10 monclient: dump: 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.406 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ed0002920 con 0x7f0ef81372f0 2024-04-19T00:44:35.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f09870640 5 monclient: authenticate success, global_id 4277 2024-04-19T00:44:35.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f09870640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f3f09870640 4 set_mon_vals no callback set 2024-04-19T00:44:35.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038d80 0x7f0ee403b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4853bb9640 5 monclient: authenticate success, global_id 4283 2024-04-19T00:44:35.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4853bb9640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f4853bb9640 4 set_mon_vals no callback set 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f0ef003d7b0 con 0x7f0ef81372f0 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 client.4218.objecter ms_dispatch 0x7f0ef8000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 3 client.4218.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 3 client.4218.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 client.4218.objecter checking command 1 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7f0ee403ebf0 0x7f0ee4041010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 client.4218.objecter _get_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 15 client.4218.objecter _session_command_op_remove -1 1 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 client.4218.objecter _get_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 client.4218.objecter get_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 15 client.4218.objecter _session_command_op_assign 2 1 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 4 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 client.4218.objecter _send_command 1 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 --> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ef0036e50 con 0x7f0ee403ebf0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 20 client.4218.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f0ef0070060 con 0x7f0ef81372f0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef4ff9640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef7fff640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038d80 0x7f0ee403b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.332+0000 7f0ef7fff640 10 monclient: get_auth_request con 0x7f0ee4038d80 auth_method 0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef77fe640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7f0ee403ebf0 0x7f0ee4041010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef77fe640 10 monclient: get_auth_request con 0x7f0ee403ebf0 auth_method 0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 10 client.4218.objecter ms_handle_connect 0x7f0ee403ebf0 2024-04-19T00:44:35.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef77fe640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7f0ee403ebf0 0x7f0ee4041010 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient: shutdown 2024-04-19T00:44:35.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- 172.21.15.43:0/729890769 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 msgr2=0x7f484c140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 --2- 172.21.15.43:0/729890769 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 0x7f484c140f20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4844009e80 tx=0x7f484401c8b0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 10 monclient: shutdown 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 1 -- 172.21.15.43:0/1258176859 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 msgr2=0x7f3f04136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 1 --2- 172.21.15.43:0/1258176859 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 0x7f3f04136e40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40098a0 tx=0x7f3ef401c4f0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- 172.21.15.43:0/729890769 shutdown_connections 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 --2- 172.21.15.43:0/729890769 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f484c13eb30 0x7f484c140f20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 --2- 172.21.15.43:0/729890769 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1280 0x7f484c0a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- 172.21.15.43:0/729890769 >> 172.21.15.43:0/729890769 conn(0x7f484c0a85c0 msgr2=0x7f484c0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 1 -- 172.21.15.43:0/1258176859 shutdown_connections 2024-04-19T00:44:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== osd.2 v2:172.21.15.43:6803/1290935720 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ef0036e50 con 0x7f0ee403ebf0 2024-04-19T00:44:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 10 client.4218.objecter ms_dispatch 0x7f0ef8000ba0 command_reply(tid 1: 0 ) 2024-04-19T00:44:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 10 client.4218.objecter _finish_command 1 = system:0 2024-04-19T00:44:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 4 2024-04-19T00:44:35.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 15 client.4218.objecter _session_command_op_remove 2 1 2024-04-19T00:44:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- 172.21.15.43:0/729890769 shutdown_connections 2024-04-19T00:44:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- 172.21.15.43:0/729890769 wait complete. 2024-04-19T00:44:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 1 --2- 172.21.15.43:0/1258176859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3f040a3430 0x7f3f040a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 1 --2- 172.21.15.43:0/1258176859 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3f040a3050 0x7f3f04136e40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f3f09870640 1 -- 172.21.15.43:0/1258176859 >> 172.21.15.43:0/1258176859 conn(0x7f3f040a4140 msgr2=0x7f3f040a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef4ff9640 10 client.4218.objecter ms_handle_connect 0x7f0ee4038d80 2024-04-19T00:44:35.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f0ef7fff640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038d80 0x7f0ee403b240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8001230 tx=0x7f0ee8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient: monmap: 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.419 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 librados: starting msgr at 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 librados: starting objecter 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 Processor -- start 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- start start 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 librados: setting wanted keys 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 librados: calling monclient init 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient: init 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient: _add_conns ranks=[2,5,1,4,7,3,8,0,6] 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f484c0a1280 0x7f484c135d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): picked mon.noname-b con 0x7f484c0a1280 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1fc0 0x7f484c1362c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): picked mon.noname-h con 0x7f484c0a1fc0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f484c13eb30 0x7f484c070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): picked mon.noname-f con 0x7f484c13eb30 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f484c0a4590 con 0x7f484c13eb30 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f484c0a4710 con 0x7f484c0a1280 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f484c0a1280 0x7f484c135d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f484c143c80 con 0x7f484c0a1fc0 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4853bb9640 10 monclient(hunting): authenticate will time out at 945.467414s 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f484c0a1280 0x7f484c135d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35280/0 (socket says 172.21.15.43:35280) 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 -- 172.21.15.43:0/2359979440 learned_addr learned my addr 172.21.15.43:0/2359979440 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): get_auth_request con 0x7f484c0a1280 auth_method 0 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4852156640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f484c13eb30 0x7f484c070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4852156640 10 monclient(hunting): get_auth_request con 0x7f484c13eb30 auth_method 0 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4852156640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4852156640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4852156640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 -- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1fc0 msgr2=0x7f484c1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f484c0a1fc0 0x7f484c1362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 -- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f484c13eb30 msgr2=0x7f484c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f484c13eb30 0x7f484c070020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 monclient: found mon.noname-b 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.336+0000 7f4851955640 1 -- 172.21.15.43:0/2359979440 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484c072110 con 0x7f484c0a1280 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4851955640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f484c0a1280 0x7f484c135d80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f484800c450 tx=0x7f484800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.?.objecter ms_handle_connect 0x7f484c0a1280 2024-04-19T00:44:35.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 -- 172.21.15.43:0/1258176859 shutdown_connections 2024-04-19T00:44:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 -- 172.21.15.43:0/1258176859 wait complete. 2024-04-19T00:44:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4848012aa0 con 0x7f484c0a1280 2024-04-19T00:44:35.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: dump: 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.425 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4848017070 con 0x7f484c0a1280 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4848012c80 con 0x7f484c0a1280 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 5 monclient: authenticate success, global_id 4286 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 monclient: _renew_subs 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484c13bbf0 con 0x7f484c0a1280 2024-04-19T00:44:35.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4843fff640 4 set_mon_vals no callback set 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 monclient: _renew_subs 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f484c13c1b0 con 0x7f484c0a1280 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 1 librados: init done 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 15 client.4286.objecter _session_command_op_assign -1 1 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 20 client.4286.objecter _get_session osd=-1 returning homeless 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f484c136800 con 0x7f484c0a1280 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: dump: 2024-04-19T00:44:35.427 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.428 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient: monmap: 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.429 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f4848012630 con 0x7f484c0a1280 2024-04-19T00:44:35.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4830038d80 0x7f483003b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 librados: starting msgr at 2024-04-19T00:44:35.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 librados: starting objecter 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f484804aad0 con 0x7f484c0a1280 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.4286.objecter ms_dispatch 0x7f484c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 3 client.4286.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 3 client.4286.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.4286.objecter checking command 1 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7f483003ebf0 0x7f4830041010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 client.4286.objecter _get_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 15 client.4286.objecter _session_command_op_remove -1 1 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 client.4286.objecter _get_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 client.4286.objecter get_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 15 client.4286.objecter _session_command_op_assign 0 1 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 4 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.4286.objecter _send_command 1 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 --> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48300416c0 con 0x7f483003ebf0 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 20 client.4286.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f484804e050 con 0x7f484c0a1280 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4851154640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4830038d80 0x7f483003b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4851154640 10 monclient: get_auth_request con 0x7f4830038d80 auth_method 0 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4852156640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7f483003ebf0 0x7f4830041010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4852156640 10 monclient: get_auth_request con 0x7f483003ebf0 auth_method 0 2024-04-19T00:44:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 Processor -- start 2024-04-19T00:44:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 -- start start 2024-04-19T00:44:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 librados: setting wanted keys 2024-04-19T00:44:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 librados: calling monclient init 2024-04-19T00:44:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient: init 2024-04-19T00:44:35.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.4286.objecter ms_handle_connect 0x7f483003ebf0 2024-04-19T00:44:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4852156640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7f483003ebf0 0x7f4830041010 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f48427fc640 10 client.4286.objecter ms_handle_connect 0x7f4830038d80 2024-04-19T00:44:35.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f4851154640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4830038d80 0x7f483003b240 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f484c0716f0 tx=0x7f483c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient: _add_conns ranks=[5,8,1,4,2,3,6,0,7] 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3f0409a0d0 0x7f3f041ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): picked mon.noname-h con 0x7f3f0409a0d0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3f040a3050 0x7f3f04135900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): picked mon.noname-e con 0x7f3f040a3050 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f040a3430 0x7f3f04135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): picked mon.noname-f con 0x7f3f040a3430 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f027fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3f040a3050 0x7f3f04135900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f027fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3f040a3050 0x7f3f04135900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59472/0 (socket says 172.21.15.43:59472) 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f027fc640 1 -- 172.21.15.43:0/2466915261 learned_addr learned my addr 172.21.15.43:0/2466915261 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f040a3d50 con 0x7f3f040a3430 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f02ffd640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3f0409a0d0 0x7f3f041ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f3f041416f0 con 0x7f3f0409a0d0 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f3f040a1ab0 con 0x7f3f040a3050 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.340+0000 7f3f09870640 10 monclient(hunting): authenticate will time out at 945.471990s 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== osd.0 v2:172.21.15.43:6801/110940973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f48300416c0 con 0x7f483003ebf0 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f48427fc640 10 client.4286.objecter ms_dispatch 0x7f484c000ba0 command_reply(tid 1: 0 ) 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f48427fc640 10 client.4286.objecter _finish_command 1 = system:0 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f48427fc640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 4 2024-04-19T00:44:35.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f48427fc640 15 client.4286.objecter _session_command_op_remove 0 1 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f040a3430 0x7f3f04135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): get_auth_request con 0x7f3f040a3050 auth_method 0 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): get_auth_request con 0x7f3f0409a0d0 auth_method 0 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): get_auth_request con 0x7f3f040a3430 auth_method 0 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 -- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3f0409a0d0 msgr2=0x7f3f041ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3f0409a0d0 0x7f3f041ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 -- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f040a3430 msgr2=0x7f3f04135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f040a3430 0x7f3f04135e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0413d280 con 0x7f3f040a3050 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f027fc640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3f040a3050 0x7f3f04135900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800d5f0 tx=0x7f3ef800dab0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 client.?.objecter ms_handle_connect 0x7f3f040a3050 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ef8012a60 con 0x7f3f040a3050 2024-04-19T00:44:35.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: dump: 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.444 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ef8019070 con 0x7f3f040a3050 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ef8012c20 con 0x7f3f040a3050 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 monclient: dump: 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.445 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.446 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f017fa640 4 set_mon_vals no callback set 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 5 monclient: authenticate success, global_id 4292 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 10 monclient: _renew_subs 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0412d0a0 con 0x7f3f040a3050 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 10 monclient: _renew_subs 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0413d960 con 0x7f3f040a3050 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f09870640 1 librados: init done 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 4) ==== 44268+0+0 (secure 0 0 0) 0x7f3ef80125f0 con 0x7f3f040a3050 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3ef0038db0 0x7f3ef003b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f3ef800f040 con 0x7f3f040a3050 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 client.4292.objecter ms_dispatch 0x7f3f04000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 3 client.4292.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 3 client.4292.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 20 client.4292.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 10 client.4292.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 15 client.4292.objecter _session_command_op_assign -1 1 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f3edc002ff0 0x7f3edc0054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 20 client.4292.objecter _get_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 20 client.4292.objecter _get_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 15 client.4292.objecter _session_command_op_remove -1 1 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 20 client.4292.objecter get_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 15 client.4292.objecter _session_command_op_assign 1 1 2024-04-19T00:44:35.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 4 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 10 client.4292.objecter _send_command 1 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3edc0084a0 con 0x7f3edc002ff0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3ef0038db0 0x7f3ef003b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 10 monclient: get_auth_request con 0x7f3ef0038db0 auth_method 0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3eebfff640 10 client.4292.objecter ms_handle_connect 0x7f3ef0038db0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.344+0000 7f3f02ffd640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3ef0038db0 0x7f3ef003b270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40af000 tx=0x7f3ef40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3f037fe640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f3edc002ff0 0x7f3edc0054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3f037fe640 10 monclient: get_auth_request con 0x7f3edc002ff0 auth_method 0 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 10 client.4268.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-19T00:44:35.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 15 client.4268.objecter _session_command_op_assign -1 2 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 20 client.4268.objecter _get_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 20 client.4268.objecter _get_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 15 client.4268.objecter _session_command_op_remove -1 2 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 20 client.4268.objecter get_session s=0x7f6f0803e4b0 osd=5 3 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 15 client.4268.objecter _session_command_op_assign 5 2 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 4 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 10 client.4268.objecter _send_command 2 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 --> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f28068240 con 0x7f6f0803eb80 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3eebfff640 10 client.4292.objecter ms_handle_connect 0x7f3edc002ff0 2024-04-19T00:44:35.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3f037fe640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f3edc002ff0 0x7f3edc0054b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f16ffd640 1 -- 172.21.15.43:0/997081818 <== osd.5 v2:172.21.15.161:6802/760479844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6f28068240 con 0x7f6f0803eb80 2024-04-19T00:44:35.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f16ffd640 10 client.4268.objecter ms_dispatch 0x7f6f28000ba0 command_reply(tid 2: 0 ) 2024-04-19T00:44:35.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f16ffd640 10 client.4268.objecter _finish_command 2 = system:0 2024-04-19T00:44:35.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f16ffd640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 4 2024-04-19T00:44:35.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f16ffd640 15 client.4268.objecter _session_command_op_remove 5 2 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 10 librados: watch_flush enter 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 10 librados: watch_flush exit 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== osd.1 v2:172.21.15.43:6800/1380723486 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3edc0084a0 con 0x7f3edc002ff0 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3eebfff640 10 client.4292.objecter ms_dispatch 0x7f3f04000ba0 command_reply(tid 1: 0 ) 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3eebfff640 10 client.4292.objecter _finish_command 1 = system:0 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3eebfff640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 4 2024-04-19T00:44:35.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f3eebfff640 15 client.4292.objecter _session_command_op_remove 1 1 2024-04-19T00:44:35.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.348+0000 7f6f2cd1b640 10 client.4268.objecter close_session for osd.5 2024-04-19T00:44:35.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7f6f0803eb80 msgr2=0x7f6f08040fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7f6f0803eb80 0x7f6f08040fa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 20 client.4268.objecter put_session s=0x7f6f0803e4b0 osd=5 1 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 20 client.4268.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 10 client.4268.objecter successfully canceled tick 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f08038d80 msgr2=0x7f6f0803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6f08038d80 0x7f6f0803b240 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6f2809e8a0 tx=0x7f6f1c009190 comp rx=0 tx=0).stop 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 10 monclient: shutdown 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6f280a1280 msgr2=0x7f6f281d4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 --2- 172.21.15.43:0/997081818 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6f280a1280 0x7f6f281d4e40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800e930 tx=0x7f6f1800edf0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f26d9d640 1 -- 172.21.15.43:0/997081818 reap_dead start 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 shutdown_connections 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 >> 172.21.15.43:0/997081818 conn(0x7f6f280a85c0 msgr2=0x7f6f2809be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 shutdown_connections 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 -- 172.21.15.43:0/997081818 wait complete. 2024-04-19T00:44:35.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f6f2cd1b640 1 librados: shutdown 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 client.4218.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 15 client.4218.objecter _session_command_op_assign -1 2 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter _get_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter _get_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 15 client.4218.objecter _session_command_op_remove -1 2 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter get_session s=0x7f0ee403e520 osd=2 3 2024-04-19T00:44:35.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 15 client.4218.objecter _session_command_op_assign 2 2 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 4 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 client.4218.objecter _send_command 2 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 --> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ed0001230 con 0x7f0ee403ebf0 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ef4ff9640 1 -- 172.21.15.43:0/145073627 <== osd.2 v2:172.21.15.43:6803/1290935720 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ed0001230 con 0x7f0ee403ebf0 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ef4ff9640 10 client.4218.objecter ms_dispatch 0x7f0ef8000ba0 command_reply(tid 2: 0 ) 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ef4ff9640 10 client.4218.objecter _finish_command 2 = system:0 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ef4ff9640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 4 2024-04-19T00:44:35.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ef4ff9640 15 client.4218.objecter _session_command_op_remove 2 2 2024-04-19T00:44:35.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 librados: watch_flush enter 2024-04-19T00:44:35.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 librados: watch_flush exit 2024-04-19T00:44:35.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 client.4218.objecter close_session for osd.2 2024-04-19T00:44:35.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7f0ee403ebf0 msgr2=0x7f0ee4041010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7f0ee403ebf0 0x7f0ee4041010 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter put_session s=0x7f0ee403e520 osd=2 1 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 client.4218.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 client.4218.objecter successfully canceled tick 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038d80 msgr2=0x7f0ee403b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038d80 0x7f0ee403b240 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8001230 tx=0x7f0ee8009040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 10 monclient: shutdown 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ef81372f0 msgr2=0x7f0ef8135a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 --2- 172.21.15.43:0/145073627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ef81372f0 0x7f0ef8135a30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009cb0 tx=0x7f0ef000c490 comp rx=0 tx=0).stop 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0efca2e640 1 -- 172.21.15.43:0/145073627 reap_dead start 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 shutdown_connections 2024-04-19T00:44:35.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 >> 172.21.15.43:0/145073627 conn(0x7f0ef80a6410 msgr2=0x7f0ef813bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 shutdown_connections 2024-04-19T00:44:35.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 -- 172.21.15.43:0/145073627 wait complete. 2024-04-19T00:44:35.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.352+0000 7f0ede7fc640 1 librados: shutdown 2024-04-19T00:44:35.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 15 client.4286.objecter _session_command_op_assign -1 2 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter _get_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter _get_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 15 client.4286.objecter _session_command_op_remove -1 2 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter get_session s=0x7f483003e520 osd=0 3 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 15 client.4286.objecter _session_command_op_assign 0 2 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 4 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 client.4286.objecter _send_command 2 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 --> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f484c076500 con 0x7f483003ebf0 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f48427fc640 1 -- 172.21.15.43:0/2359979440 <== osd.0 v2:172.21.15.43:6801/110940973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f484c076500 con 0x7f483003ebf0 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f48427fc640 10 client.4286.objecter ms_dispatch 0x7f484c000ba0 command_reply(tid 2: 0 ) 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f48427fc640 10 client.4286.objecter _finish_command 2 = system:0 2024-04-19T00:44:35.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f48427fc640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 4 2024-04-19T00:44:35.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f48427fc640 15 client.4286.objecter _session_command_op_remove 0 2 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 librados: watch_flush enter 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 librados: watch_flush exit 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 client.4286.objecter close_session for osd.0 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7f483003ebf0 msgr2=0x7f4830041010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7f483003ebf0 0x7f4830041010 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter put_session s=0x7f483003e520 osd=0 1 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 client.4286.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 client.4286.objecter successfully canceled tick 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4830038d80 msgr2=0x7f483003b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4830038d80 0x7f483003b240 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f484c0716f0 tx=0x7f483c008040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 10 monclient: shutdown 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f484c0a1280 msgr2=0x7f484c135d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 --2- 172.21.15.43:0/2359979440 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f484c0a1280 0x7f484c135d80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f484800c450 tx=0x7f484800c910 comp rx=0 tx=0).stop 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4852156640 1 -- 172.21.15.43:0/2359979440 reap_dead start 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 shutdown_connections 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 >> 172.21.15.43:0/2359979440 conn(0x7f484c0a85c0 msgr2=0x7f484c13a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 shutdown_connections 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 -- 172.21.15.43:0/2359979440 wait complete. 2024-04-19T00:44:35.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.360+0000 7f4853bb9640 1 librados: shutdown 2024-04-19T00:44:35.464 INFO:teuthology.orchestra.run.smithi043.stdout:47244640260 2024-04-19T00:44:35.464 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T00:44:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 client.4292.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-19T00:44:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 15 client.4292.objecter _session_command_op_assign -1 2 2024-04-19T00:44:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter _get_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter _get_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 15 client.4292.objecter _session_command_op_remove -1 2 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter get_session s=0x7f3edc0028e0 osd=1 3 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 15 client.4292.objecter _session_command_op_assign 1 2 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 4 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 client.4292.objecter _send_command 2 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3edc001220 con 0x7f3edc002ff0 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3eebfff640 1 -- 172.21.15.43:0/2466915261 <== osd.1 v2:172.21.15.43:6800/1380723486 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3edc001220 con 0x7f3edc002ff0 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3eebfff640 10 client.4292.objecter ms_dispatch 0x7f3f04000ba0 command_reply(tid 2: 0 ) 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3eebfff640 10 client.4292.objecter _finish_command 2 = system:0 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3eebfff640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 4 2024-04-19T00:44:35.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3eebfff640 15 client.4292.objecter _session_command_op_remove 1 2 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 librados: watch_flush enter 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 librados: watch_flush exit 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 client.4292.objecter close_session for osd.1 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f3edc002ff0 msgr2=0x7f3edc0054b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f3edc002ff0 0x7f3edc0054b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter put_session s=0x7f3edc0028e0 osd=1 1 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 client.4292.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 client.4292.objecter successfully canceled tick 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3ef0038db0 msgr2=0x7f3ef003b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3ef0038db0 0x7f3ef003b270 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3ef40af000 tx=0x7f3ef40a7000 comp rx=0 tx=0).stop 2024-04-19T00:44:35.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 10 monclient: shutdown 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3f040a3050 msgr2=0x7f3f04135900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 --2- 172.21.15.43:0/2466915261 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3f040a3050 0x7f3f04135900 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800d5f0 tx=0x7f3ef800dab0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3f037fe640 1 -- 172.21.15.43:0/2466915261 reap_dead start 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 shutdown_connections 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 >> 172.21.15.43:0/2466915261 conn(0x7f3f040a4140 msgr2=0x7f3f040a62c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 shutdown_connections 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 -- 172.21.15.43:0/2466915261 wait complete. 2024-04-19T00:44:35.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.372+0000 7f3ee9ffb640 1 librados: shutdown 2024-04-19T00:44:35.470 INFO:teuthology.orchestra.run.smithi043.stdout:51539607557 2024-04-19T00:44:35.470 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T00:44:35.471 INFO:teuthology.orchestra.run.smithi043.stdout:47244640258 2024-04-19T00:44:35.471 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T00:44:35.472 INFO:teuthology.orchestra.run.smithi043.stdout:47244640258 2024-04-19T00:44:35.472 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.596+0000 7fb9d5711640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.596+0000 7fb9d5711640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.596+0000 7fb9d5711640 10 monclient: monmap: 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.602343+0000 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.602343+0000 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.601 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.602 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 Processor -- start 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 -- start start 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient: init 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient: _add_conns ranks=[6,0,9,2,8,7,4,5,3,1,10] 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb9d0138a50 0x7fb9d0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): picked mon.noname-d con 0x7fb9d0138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9d00a5bb0 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb9d0139370 0x7fb9d0138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): picked mon.noname-a con 0x7fb9d0139370 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9c7fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb9d00a5bb0 0x7fb9d0137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44760/0 (socket says 172.21.15.43:44760) 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9c7fff640 1 -- 172.21.15.43:0/2047335584 learned_addr learned my addr 172.21.15.43:0/2047335584 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 1 --2- 172.21.15.43:0/2047335584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb9d0138a50 0x7fb9d0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9d00a0d40 con 0x7fb9d00a5bb0 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d00a1020 con 0x7fb9d0139370 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb9d00a0780 con 0x7fb9d0138a50 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9d5711640 10 monclient(hunting): authenticate will time out at 945.730067s 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): get_auth_request con 0x7fb9d0138a50 auth_method 0 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 1 --2- 172.21.15.43:0/2047335584 >> v2:172.21.15.43:3300/0 conn(0x7fb9d0139370 0x7fb9d0138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): get_auth_request con 0x7fb9d0139370 auth_method 0 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ce7fc640 1 -- 172.21.15.43:0/2047335584 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085197715 0 0) 0x7fb9d00a0d40 con 0x7fb9d00a5bb0 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ce7fc640 10 monclient(hunting): _negotiate 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ce7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ce7fc640 10 monclient(hunting): my global_id is 4275 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ce7fc640 1 -- 172.21.15.43:0/2047335584 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9bc003530 con 0x7fb9d00a5bb0 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-04-19T00:44:35.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 1 -- 172.21.15.43:0/2047335584 >> v2:172.21.15.43:3300/0 conn(0x7fb9d0139370 msgr2=0x7fb9d0138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 1 --2- 172.21.15.43:0/2047335584 >> v2:172.21.15.43:3300/0 conn(0x7fb9d0139370 0x7fb9d0138070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 1 -- 172.21.15.43:0/2047335584 >> v1:172.21.15.43:6789/0 conn(0x7fb9d00a5bb0 legacy=0x7fb9d0137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 1 monclient: found mon.noname-d 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.600+0000 7fb9ceffd640 1 -- 172.21.15.43:0/2047335584 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d00a0a60 con 0x7fb9d0138a50 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ceffd640 1 --2- 172.21.15.43:0/2047335584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb9d0138a50 0x7fb9d0138e30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097c0 tx=0x7fb9c001c360 comp rx=0 tx=0).ready entity=mon.6 client_cookie=10bfe1b487f6ee82 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 1 -- 172.21.15.43:0/2047335584 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb9c00b0070 con 0x7fb9d0138a50 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: dump: 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.608 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 1 -- 172.21.15.43:0/2047335584 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c00aa810 con 0x7fb9d0138a50 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 5 monclient: authenticate success, global_id 4227 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 4 set_mon_vals no callback set 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 1 -- 172.21.15.43:0/2047335584 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb9c00aae30 con 0x7fb9d0138a50 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9ce7fc640 10 monclient: dump: 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.610 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 10 monclient: shutdown 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb9d0138a50 msgr2=0x7fb9d0138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 --2- 172.21.15.43:0/2047335584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb9d0138a50 0x7fb9d0138e30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb9c00097c0 tx=0x7fb9c001c360 comp rx=0 tx=0).stop 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 shutdown_connections 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 --2- 172.21.15.43:0/2047335584 >> v2:172.21.15.43:3300/0 conn(0x7fb9d0139370 0x7fb9d0138070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 --2- 172.21.15.43:0/2047335584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb9d0138a50 0x7fb9d0138e30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 >> 172.21.15.43:0/2047335584 conn(0x7fb9d00a7cb0 msgr2=0x7fb9d00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 shutdown_connections 2024-04-19T00:44:35.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.604+0000 7fb9d5711640 1 -- 172.21.15.43:0/2047335584 wait complete. 2024-04-19T00:44:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient: monmap: 2024-04-19T00:44:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.618 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.619 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 librados: starting msgr at 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 librados: starting objecter 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 Processor -- start 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 -- start start 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 librados: setting wanted keys 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 librados: calling monclient init 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient: init 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient: _add_conns ranks=[8,4,5,2,7,3,6,1,0] 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 0x7fb9d0139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): picked mon.noname-e con 0x7fb9d00a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 0x7fb9d0139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 0x7fb9d0139b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59486/0 (socket says 172.21.15.43:59486) 2024-04-19T00:44:35.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 1 -- 172.21.15.43:0/2473475023 learned_addr learned my addr 172.21.15.43:0/2473475023 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb9d0138a50 0x7fb9d013a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): picked mon.noname-c con 0x7fb9d0138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb9d0139370 0x7fb9d013a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): picked mon.noname-h con 0x7fb9d0139370 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fb9d0138730 con 0x7fb9d0138a50 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb9d01385b0 con 0x7fb9d0139370 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb9d0138a50 0x7fb9d013a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb9d0143820 con 0x7fb9d00a5bb0 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9d5711640 10 monclient(hunting): authenticate will time out at 945.738701s 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): get_auth_request con 0x7fb9d00a5bb0 auth_method 0 2024-04-19T00:44:35.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): get_auth_request con 0x7fb9d0138a50 auth_method 0 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 -- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 msgr2=0x7fb9d0139b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 0x7fb9d0139b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 -- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb9d0139370 msgr2=0x7fb9d013a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb9d0139370 0x7fb9d013a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 monclient: found mon.noname-c 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d0137bd0 con 0x7fb9d0138a50 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9ceffd640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 0x7fb9d0139b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c7fff640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb9d0138a50 0x7fb9d013a0a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8007fd0 tx=0x7fb9b800c890 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 client.?.objecter ms_handle_connect 0x7fb9d0138a50 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb9b8017070 con 0x7fb9d0138a50 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 monclient: dump: 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.624 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.608+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b80070d0 con 0x7fb9d0138a50 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 5 monclient: authenticate success, global_id 4252 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 10 monclient: _renew_subs 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d013bbf0 con 0x7fb9d0138a50 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 10 monclient: _renew_subs 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d0137200 con 0x7fb9d0138a50 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9d5711640 1 librados: init done 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb9b8007e30 con 0x7fb9d0138a50 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9cdffb640 4 set_mon_vals no callback set 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c57fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b4005090 con 0x7fb9d0138a50 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 10 monclient: dump: 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.626 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb9b8005af0 con 0x7fb9d0138a50 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb9b0038e90 0x7fb9b003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fb9b8048550 con 0x7fb9d0138a50 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 10 client.4252.objecter ms_dispatch 0x7fb9d0000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 3 client.4252.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 3 client.4252.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9c77fe640 20 client.4252.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9ceffd640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb9b0038e90 0x7fb9b003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.612+0000 7fb9ceffd640 10 monclient: get_auth_request con 0x7fb9b0038e90 auth_method 0 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.616+0000 7fb9ceffd640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb9b0038e90 0x7fb9b003b350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009710 tx=0x7fb9c00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.616+0000 7fb9c77fe640 10 client.4252.objecter ms_handle_connect 0x7fb9b0038e90 2024-04-19T00:44:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.624+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb9b8011d50 con 0x7fb9d0138a50 2024-04-19T00:44:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.624+0000 7fb9c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.624+0000 7fb9c77fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.624+0000 7fdeb3310640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.624+0000 7fdeb3310640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.624+0000 7fdeb3310640 10 monclient: monmap: 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.630255+0000 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.630255+0000 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.630 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient: monmap: 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.631 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.630796+0000 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.630796+0000 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.632 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.633 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 Processor -- start 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 -- start start 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient: init 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient: _add_conns ranks=[7,4,3,1,9,6,2,0,5,10,8] 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac138a50 0x7fdeac138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): picked mon.noname-i con 0x7fdeac138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 0x7fdeac139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): picked mon.noname-c con 0x7fdeac0a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdeac137960 0x7fdeac137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): picked mon.noname-g con 0x7fdeac137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdeac0a1020 con 0x7fdeac137960 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdeac1398b0 con 0x7fdeac0a5bb0 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdeac0a0780 con 0x7fdeac138a50 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb3310640 10 monclient(hunting): authenticate will time out at 945.757251s 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 0x7fdeac139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 0x7fdeac139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48512/0 (socket says 172.21.15.43:48512) 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 -- 172.21.15.43:0/3071014167 learned_addr learned my addr 172.21.15.43:0/3071014167 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): get_auth_request con 0x7fdeac0a5bb0 auth_method 0 2024-04-19T00:44:35.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb18ad640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdeac137960 0x7fdeac137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient(hunting): handle_auth_done global_id 4261 payload 274 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 -- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac138a50 msgr2=0x7fdeac138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb10ac640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac138a50 0x7fdeac138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac138a50 0x7fdeac138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 -- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdeac137960 msgr2=0x7fdeac137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdeac137960 0x7fdeac137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 monclient: found mon.noname-c 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb08ab640 1 -- 172.21.15.43:0/3071014167 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac0a0d40 con 0x7fdeac0a5bb0 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb18ad640 10 monclient: get_auth_request con 0x7fdeac137960 auth_method 0 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb18ad640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdeac137960 0x7fdeac137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb10ac640 10 monclient: get_auth_request con 0x7fdeac138a50 auth_method 0 2024-04-19T00:44:35.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fdeb10ac640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac138a50 0x7fdeac138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:35.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 1 Processor -- start 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 1 -- start start 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient: init 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient: _add_conns ranks=[6,4,1,9,7,8,5,10,0,2,3] 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 0x7fc124138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient(hunting): picked mon.noname-d con 0x7fc124138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc1240a5bb0 0x7fc124139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient(hunting): picked mon.noname-c con 0x7fc1240a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc124137960 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc129ff6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 0x7fc124138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc129ff6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 0x7fc124138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42176/0 (socket says 172.21.15.43:42176) 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc129ff6640 1 -- 172.21.15.43:0/1033675971 learned_addr learned my addr 172.21.15.43:0/1033675971 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc1297f5640 1 --2- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc1240a5bb0 0x7fc124139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb08ab640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 0x7fdeac139370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0098a0 tx=0x7fde9c01c410 comp rx=0 tx=0).ready entity=mon.4 client_cookie=dcd94e0c09c4267e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 1 -- 172.21.15.43:0/3071014167 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde9c0b0070 con 0x7fdeac0a5bb0 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: dump: 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.644 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 1 -- 172.21.15.43:0/3071014167 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde9c0044a0 con 0x7fdeac0a5bb0 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 5 monclient: authenticate success, global_id 4261 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 4 set_mon_vals no callback set 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 1 -- 172.21.15.43:0/3071014167 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde9c003a90 con 0x7fdeac0a5bb0 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdea3fff640 10 monclient: dump: 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.646 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: shutdown 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 -- 172.21.15.43:0/3071014167 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 msgr2=0x7fdeac139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 0x7fdeac139370 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fde9c0098a0 tx=0x7fde9c01c410 comp rx=0 tx=0).stop 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 -- 172.21.15.43:0/3071014167 shutdown_connections 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdeac137960 0x7fdeac137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdeac0a5bb0 0x7fdeac139370 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 --2- 172.21.15.43:0/3071014167 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac138a50 0x7fdeac138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 -- 172.21.15.43:0/3071014167 >> 172.21.15.43:0/3071014167 conn(0x7fdeac0a7cb0 msgr2=0x7fdeac0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 -- 172.21.15.43:0/3071014167 shutdown_connections 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 -- 172.21.15.43:0/3071014167 wait complete. 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: monmap: 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.648 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 librados: starting msgr at 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 librados: starting objecter 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 Processor -- start 2024-04-19T00:44:35.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 -- start start 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 librados: setting wanted keys 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 librados: calling monclient init 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: init 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient: _add_conns ranks=[0,7,8,4,3,6,2,1,5] 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac0a5bb0 0x7fdeac1d0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 10 monclient(hunting): picked mon.noname-a con 0x7fdeac0a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.632+0000 7fdeb3310640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac137960 0x7fdeac1369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): picked mon.noname-i con 0x7fdeac137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 0x7fdeac13c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): picked mon.noname-e con 0x7fdeac138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdeac0a2100 con 0x7fdeac0a5bb0 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdeac0a1f80 con 0x7fdeac137960 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdeac142810 con 0x7fdeac138a50 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient(hunting): authenticate will time out at 945.764053s 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 0x7fdeac13c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 0x7fdeac13c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59498/0 (socket says 172.21.15.43:59498) 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 -- 172.21.15.43:0/886947788 learned_addr learned my addr 172.21.15.43:0/886947788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): get_auth_request con 0x7fdeac138a50 auth_method 0 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac0a5bb0 0x7fdeac1d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): get_auth_request con 0x7fdeac0a5bb0 auth_method 0 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb08ab640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac137960 0x7fdeac1369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 -- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac137960 msgr2=0x7fdeac1369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac137960 0x7fdeac1369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 -- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac0a5bb0 msgr2=0x7fdeac1d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac0a5bb0 0x7fdeac1d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 -- 172.21.15.43:0/886947788 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeac1d0e50 con 0x7fdeac138a50 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb08ab640 10 monclient: get_auth_request con 0x7fdeac137960 auth_method 0 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb08ab640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac137960 0x7fdeac1369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb18ad640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 0x7fdeac13c840 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdea400ced0 tx=0x7fdea400a710 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 client.?.objecter ms_handle_connect 0x7fdeac138a50 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdea400dab0 con 0x7fdeac138a50 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: dump: 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.654 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea400dc90 con 0x7fdeac138a50 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 5 monclient: authenticate success, global_id 4310 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient: _renew_subs 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdeac09ee60 con 0x7fdeac138a50 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdea400de40 con 0x7fdeac138a50 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 10 monclient: dump: 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.656 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient: _renew_subs 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdeac13e390 con 0x7fdeac138a50 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 librados: init done 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea37fe640 4 set_mon_vals no callback set 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdeac0008e0 con 0x7fdeac138a50 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdea401c020 con 0x7fdeac138a50 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdea1ffb640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde90038e40 0x7fde9003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde90038e40 0x7fde9003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 10 monclient: get_auth_request con 0x7fde90038e40 auth_method 0 2024-04-19T00:44:35.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.636+0000 7fdeb10ac640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde90038e40 0x7fde9003b300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdea8006fd0 tx=0x7fdea8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.628+0000 7fc12c25a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1240a0d40 con 0x7fc124137960 2024-04-19T00:44:35.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc12c25a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc1240a1020 con 0x7fc1240a5bb0 2024-04-19T00:44:35.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc12c25a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fc1241398b0 con 0x7fc124138a50 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc12c25a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc12c25a640 10 monclient(hunting): authenticate will time out at 945.768277s 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): get_auth_request con 0x7fc124138a50 auth_method 0 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): get_auth_request con 0x7fc1240a5bb0 auth_method 0 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc1297f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 1 -- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc1240a5bb0 msgr2=0x7fc124139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc128ff4640 1 -- 172.21.15.43:0/1033675971 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349976939 0 0) 0x7fc1240a0d40 con 0x7fc124137960 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 1 --2- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc1240a5bb0 0x7fc124139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 1 -- 172.21.15.43:0/1033675971 >> v1:172.21.15.161:6789/0 conn(0x7fc124137960 legacy=0x7fc124137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 1 monclient: found mon.noname-d 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 1 -- 172.21.15.43:0/1033675971 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1240a0a60 con 0x7fc124138a50 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc128ff4640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:44:35.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.640+0000 7fc129ff6640 1 --2- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 0x7fc124138e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc1200098a0 tx=0x7fc12001c460 comp rx=0 tx=0).ready entity=mon.6 client_cookie=761f8e6dcb033735 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 10 client.4310.objecter ms_handle_connect 0x7fde90038e40 2024-04-19T00:44:35.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fdea400d820 con 0x7fdeac138a50 2024-04-19T00:44:35.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 10 client.4310.objecter ms_dispatch 0x7fdeac000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 3 client.4310.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 3 client.4310.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 20 client.4310.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 1 -- 172.21.15.43:0/1033675971 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1200b0070 con 0x7fc124138a50 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: dump: 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.664 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 1 -- 172.21.15.43:0/1033675971 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1200aa810 con 0x7fc124138a50 2024-04-19T00:44:35.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 1 -- 172.21.15.43:0/1033675971 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1200a9300 con 0x7fc124138a50 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc128ff4640 10 monclient: dump: 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.666 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 5 monclient: authenticate success, global_id 4236 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 4 set_mon_vals no callback set 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 10 monclient: shutdown 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 msgr2=0x7fc124138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 --2- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 0x7fc124138e30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc1200098a0 tx=0x7fc12001c460 comp rx=0 tx=0).stop 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 shutdown_connections 2024-04-19T00:44:35.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 --2- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc1240a5bb0 0x7fc124139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 --2- 172.21.15.43:0/1033675971 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc124138a50 0x7fc124138e30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 >> 172.21.15.43:0/1033675971 conn(0x7fc1240a7cb0 msgr2=0x7fc1240a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 shutdown_connections 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 -- 172.21.15.43:0/1033675971 wait complete. 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdea4015960 con 0x7fdeac138a50 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fdea1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 10 monclient: monmap: 2024-04-19T00:44:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.673 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 librados: starting msgr at 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 librados: starting objecter 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 Processor -- start 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 -- start start 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 librados: setting wanted keys 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 1 librados: calling monclient init 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.644+0000 7fc12c25a640 10 monclient: init 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient: _add_conns ranks=[8,5,3,6,4,7,0,1,2] 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 0x7fc124134180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): picked mon.noname-e con 0x7fc1240a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc124137960 0x7fc1241346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 0x7fc124134180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 0x7fc124134180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59514/0 (socket says 172.21.15.43:59514) 2024-04-19T00:44:35.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 -- 172.21.15.43:0/3342913925 learned_addr learned my addr 172.21.15.43:0/3342913925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): picked mon.noname-h con 0x7fc124137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc124138a50 0x7fc124134c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): picked mon.noname-g con 0x7fc124138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc124143bb0 con 0x7fc124138a50 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc124143a30 con 0x7fc124137960 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fc124143d30 con 0x7fc1240a5bb0 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient(hunting): authenticate will time out at 945.776670s 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): get_auth_request con 0x7fc1240a5bb0 auth_method 0 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc1297f5640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc124137960 0x7fc1241346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12a7f7640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc124138a50 0x7fc124134c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc1297f5640 10 monclient(hunting): get_auth_request con 0x7fc124137960 auth_method 0 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc1297f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc1297f5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc1297f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12a7f7640 10 monclient(hunting): get_auth_request con 0x7fc124138a50 auth_method 0 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12a7f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12a7f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12a7f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 -- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc124137960 msgr2=0x7fc1241346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc124137960 0x7fc1241346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 -- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc124138a50 msgr2=0x7fc124134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc124138a50 0x7fc124134c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc124137600 con 0x7fc1240a5bb0 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc129ff6640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 0x7fc124134180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc1200097c0 tx=0x7fc120019200 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 client.?.objecter ms_handle_connect 0x7fc1240a5bb0 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1200b9020 con 0x7fc1240a5bb0 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: dump: 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.679 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc120004040 con 0x7fc1240a5bb0 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11bfff640 4 set_mon_vals no callback set 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 5 monclient: authenticate success, global_id 4319 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc1200b0040 con 0x7fc1240a5bb0 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient: _renew_subs 2024-04-19T00:44:35.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc124136f00 con 0x7fc1240a5bb0 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc11a7fc640 10 monclient: dump: 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.681 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient: _renew_subs 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fc12409fc90 con 0x7fc1240a5bb0 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.648+0000 7fc12c25a640 1 librados: init done 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc107fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc107fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc107fff640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc100005090 con 0x7fc1240a5bb0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc1200bc020 con 0x7fc1240a5bb0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc108038d70 0x7fc10803b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fc1200e2df0 con 0x7fc1240a5bb0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 10 client.4319.objecter ms_dispatch 0x7fc124000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 3 client.4319.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 3 client.4319.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 20 client.4319.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc1297f5640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc108038d70 0x7fc10803b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc1297f5640 10 monclient: get_auth_request con 0x7fc108038d70 auth_method 0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc11a7fc640 10 client.4319.objecter ms_handle_connect 0x7fc108038d70 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.652+0000 7fc1297f5640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc108038d70 0x7fc10803b230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc12406e8c0 tx=0x7fc114008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.660+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc1200c3030 con 0x7fc1240a5bb0 2024-04-19T00:44:35.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.660+0000 7fc11a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.660+0000 7fc11a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.660+0000 7fc86e479640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.660+0000 7fc86e479640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.660+0000 7fc86e479640 10 monclient: monmap: 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.666046+0000 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.666046+0000 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.689 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 Processor -- start 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 -- start start 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient: init 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient: _add_conns ranks=[7,10,5,2,6,8,9,0,1,3,4] 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 0x7fc868138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): picked mon.noname-i con 0x7fc868138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc8680a5bb0 0x7fc868139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): picked mon.noname-f con 0x7fc8680a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc868137960 0x7fc868137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): picked mon.noname-h con 0x7fc868137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc8680a1020 con 0x7fc8680a5bb0 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc8681398b0 con 0x7fc868137960 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc8680a0780 con 0x7fc868138a50 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86e479640 10 monclient(hunting): authenticate will time out at 945.793055s 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 0x7fc868138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc868137960 0x7fc868137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 0x7fc868138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:52136/0 (socket says 172.21.15.43:52136) 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 -- 172.21.15.43:0/209061360 learned_addr learned my addr 172.21.15.43:0/209061360 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): get_auth_request con 0x7fc868138a50 auth_method 0 2024-04-19T00:44:35.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): get_auth_request con 0x7fc868137960 auth_method 0 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc8680a5bb0 0x7fc868139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): get_auth_request con 0x7fc8680a5bb0 auth_method 0 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc8677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 -- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc868137960 msgr2=0x7fc868137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 --2- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc868137960 0x7fc868137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 -- 172.21.15.43:0/209061360 >> v2:172.21.15.161:3300/0 conn(0x7fc8680a5bb0 msgr2=0x7fc868139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 --2- 172.21.15.43:0/209061360 >> v2:172.21.15.161:3300/0 conn(0x7fc8680a5bb0 0x7fc868139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 monclient: found mon.noname-i 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:35.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc867fff640 1 -- 172.21.15.43:0/209061360 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8680a0d40 con 0x7fc868138a50 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.664+0000 7fc86ca16640 1 --2- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc868137960 0x7fc868137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc867fff640 1 --2- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 0x7fc868138e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc858009870 tx=0x7fc85801c3f0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=e5fbcf27d5ced5f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 1 -- 172.21.15.43:0/209061360 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8580b0070 con 0x7fc868138a50 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: dump: 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.694 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 1 -- 172.21.15.43:0/209061360 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8580aa3a0 con 0x7fc868138a50 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 1 -- 172.21.15.43:0/209061360 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8580aaa40 con 0x7fc868138a50 2024-04-19T00:44:35.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 5 monclient: authenticate success, global_id 4318 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 4 set_mon_vals no callback set 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc866ffd640 10 monclient: dump: 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.696 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: shutdown 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 msgr2=0x7fc868138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 0x7fc868138e30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc858009870 tx=0x7fc85801c3f0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- 172.21.15.43:0/209061360 shutdown_connections 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc868137960 0x7fc868137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- 172.21.15.43:0/209061360 >> v2:172.21.15.161:3300/0 conn(0x7fc8680a5bb0 0x7fc868139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- 172.21.15.43:0/209061360 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc868138a50 0x7fc868138e30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- 172.21.15.43:0/209061360 >> 172.21.15.43:0/209061360 conn(0x7fc8680a7cb0 msgr2=0x7fc8680a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- 172.21.15.43:0/209061360 shutdown_connections 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- 172.21.15.43:0/209061360 wait complete. 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: monmap: 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.698 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 librados: starting msgr at 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 librados: starting objecter 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 Processor -- start 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- start start 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 librados: setting wanted keys 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 librados: calling monclient init 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: init 2024-04-19T00:44:35.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient: _add_conns ranks=[5,3,2,8,6,7,1,0,4] 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 0x7fc8680a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient(hunting): picked mon.noname-h con 0x7fc8680a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc868137960 0x7fc8680a4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient(hunting): picked mon.noname-g con 0x7fc868137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc868138a50 0x7fc8681354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient(hunting): picked mon.noname-b con 0x7fc868138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680a2100 con 0x7fc868138a50 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680a1f80 con 0x7fc868137960 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.668+0000 7fc86e479640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc868142810 con 0x7fc8680a5bb0 2024-04-19T00:44:35.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 0x7fc8680a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 0x7fc8680a4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:42874/0 (socket says 172.21.15.43:42874) 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 -- 172.21.15.43:0/3775468797 learned_addr learned my addr 172.21.15.43:0/3775468797 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient(hunting): authenticate will time out at 945.800194s 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc868137960 0x7fc8680a4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc868138a50 0x7fc8681354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): get_auth_request con 0x7fc8680a5bb0 auth_method 0 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): get_auth_request con 0x7fc868138a50 auth_method 0 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): get_auth_request con 0x7fc868137960 auth_method 0 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86ca16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient: monmap: 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.675111+0000 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.675111+0000 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.703 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.704 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 -- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc868137960 msgr2=0x7fc8680a4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc868137960 0x7fc8680a4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 -- 172.21.15.43:0/3775468797 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc868138a50 msgr2=0x7fc8681354e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc868138a50 0x7fc8681354e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 monclient: found mon.noname-h 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 -- 172.21.15.43:0/3775468797 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8680a25b0 con 0x7fc8680a5bb0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc867fff640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 0x7fc8680a4640 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc858009870 tx=0x7fc858019190 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 client.?.objecter ms_handle_connect 0x7fc8680a5bb0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8580b0040 con 0x7fc8680a5bb0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 Processor -- start 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: dump: 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 -- start start 2024-04-19T00:44:35.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient: init 2024-04-19T00:44:35.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 5 monclient: authenticate success, global_id 4271 2024-04-19T00:44:35.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient: _renew_subs 2024-04-19T00:44:35.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:44:35.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8680a2870 con 0x7fc8680a5bb0 2024-04-19T00:44:35.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8580aa550 con 0x7fc8680a5bb0 2024-04-19T00:44:35.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient: _add_conns ranks=[8,2,7,6,10,0,9,3,1,5,4] 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f69900a0a40 0x7f69900a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): picked mon.noname-e con 0x7f69900a0a40 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6990131f40 0x7f6990135060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): picked mon.noname-b con 0x7f6990131f40 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6990132380 0x7f69901355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): picked mon.noname-i con 0x7f6990132380 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f699009d6f0 con 0x7f6990131f40 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f699009d9d0 con 0x7f6990132380 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f699009dcb0 con 0x7f69900a0a40 2024-04-19T00:44:35.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f69962e0640 10 monclient(hunting): authenticate will time out at 945.802422s 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6990131f40 0x7f6990135060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6990131f40 0x7f6990135060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35300/0 (socket says 172.21.15.43:35300) 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 1 -- 172.21.15.43:0/1229431003 learned_addr learned my addr 172.21.15.43:0/1229431003 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 10 monclient(hunting): get_auth_request con 0x7f6990131f40 auth_method 0 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8667fc640 4 set_mon_vals no callback set 2024-04-19T00:44:35.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6990132380 0x7f69901355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient: _renew_subs 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8680a50c0 con 0x7fc8680a5bb0 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 1 librados: init done 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc8580aab70 con 0x7fc8680a5bb0 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 monclient: dump: 2024-04-19T00:44:35.714 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.715 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc858003a50 con 0x7fc8680a5bb0 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc854038da0 0x7fc85403b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f69900a0a40 0x7f69900a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): get_auth_request con 0x7f6990132380 auth_method 0 2024-04-19T00:44:35.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): get_auth_request con 0x7f69900a0a40 auth_method 0 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f699487d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fc8580bfa50 con 0x7fc8680a5bb0 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 10 client.4271.objecter ms_dispatch 0x7fc868000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 3 client.4271.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 3 client.4271.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc864ff9640 20 client.4271.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 -- 172.21.15.43:0/1229431003 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6990132380 msgr2=0x7f69901355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6990132380 0x7f69901355a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 -- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6990131f40 msgr2=0x7f6990135060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6990131f40 0x7f6990135060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7f698ffff640 1 -- 172.21.15.43:0/1229431003 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69900a4bd0 con 0x7f69900a0a40 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc8677fe640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc854038da0 0x7fc85403b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.672+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8680a2a40 con 0x7fc8680a5bb0 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7fc8677fe640 10 monclient: get_auth_request con 0x7fc854038da0 auth_method 0 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient: get_monmap_and_config 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient: monmap: 2024-04-19T00:44:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:35.678604+0000 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:35.678604+0000 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:35.722 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 Processor -- start 2024-04-19T00:44:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7fc8677fe640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc854038da0 0x7fc85403b260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc86809de00 tx=0x7fc85c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7fc864ff9640 10 client.4271.objecter ms_handle_connect 0x7fc854038da0 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 -- start start 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient: init 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient: _add_conns ranks=[2,9,1,0,6,3,8,10,4,7,5] 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 0x7f0460138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f046006d800 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f046006dbe0 0x7f046006e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f046006dbe0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0460135400 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04600a5310 con 0x7f0460135400 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f04600a4ff0 con 0x7f046006dbe0 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0460070f70 con 0x7f046006d800 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045ffff640 10 monclient(hunting): authenticate will time out at 945.805443s 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 0x7f0460138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 0x7f0460138fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35302/0 (socket says 172.21.15.43:35302) 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 -- 172.21.15.43:0/449743861 learned_addr learned my addr 172.21.15.43:0/449743861 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): get_auth_request con 0x7f046006d800 auth_method 0 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f0457fff640 1 --2- 172.21.15.43:0/449743861 >> v2:172.21.15.43:3300/0 conn(0x7f046006dbe0 0x7f046006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f0457fff640 10 monclient(hunting): get_auth_request con 0x7f046006dbe0 auth_method 0 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f0457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f0457fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f0457fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 -- 172.21.15.43:0/449743861 >> v2:172.21.15.43:3300/0 conn(0x7f046006dbe0 msgr2=0x7f046006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 --2- 172.21.15.43:0/449743861 >> v2:172.21.15.43:3300/0 conn(0x7f046006dbe0 0x7f046006e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 -- 172.21.15.43:0/449743861 >> v1:172.21.15.161:6789/0 conn(0x7f0460135400 legacy=0x7f04601358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 monclient: found mon.noname-b 2024-04-19T00:44:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 -- 172.21.15.43:0/449743861 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0460071250 con 0x7f046006d800 2024-04-19T00:44:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f0457fff640 1 --2- 172.21.15.43:0/449743861 >> v2:172.21.15.43:3300/0 conn(0x7f046006dbe0 0x7f046006e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698ffff640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f69900a0a40 0x7f69900a0e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6980009dc0 tx=0x7f698001c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1fa645a63bb4b7bf server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 1 -- 172.21.15.43:0/1229431003 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69800a6940 con 0x7f69900a0a40 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: dump: 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 1 -- 172.21.15.43:0/1229431003 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69800a6b00 con 0x7f69900a0a40 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f69962e0640 5 monclient: authenticate success, global_id 4328 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f69962e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f69962e0640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f69962e0640 4 set_mon_vals no callback set 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 1 -- 172.21.15.43:0/1229431003 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f69800a6cf0 con 0x7f69900a0a40 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f698effd640 10 monclient: dump: 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.676+0000 7f045effd640 1 --2- 172.21.15.43:0/449743861 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 0x7f0460138fc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0450009870 tx=0x7f0450024cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ac38bc0ad0c4106 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 1 -- 172.21.15.43:0/449743861 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04500ba070 con 0x7f046006d800 2024-04-19T00:44:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient: shutdown 2024-04-19T00:44:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f69900a0a40 msgr2=0x7f69900a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f69900a0a40 0x7f69900a0e20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6980009dc0 tx=0x7f698001c850 comp rx=0 tx=0).stop 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: dump: 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 5 monclient: authenticate success, global_id 4313 2024-04-19T00:44:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 1 -- 172.21.15.43:0/449743861 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04500add00 con 0x7f046006d800 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 4 set_mon_vals no callback set 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 1 -- 172.21.15.43:0/449743861 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04500b37b0 con 0x7f046006d800 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045e7fc640 10 monclient: dump: 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.741 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- 172.21.15.43:0/1229431003 shutdown_connections 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6990132380 0x7f69901355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6990131f40 0x7f6990135060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- 172.21.15.43:0/1229431003 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f69900a0a40 0x7f69900a0e20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- 172.21.15.43:0/1229431003 >> 172.21.15.43:0/1229431003 conn(0x7f69900a3df0 msgr2=0x7f69900a4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- 172.21.15.43:0/1229431003 shutdown_connections 2024-04-19T00:44:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- 172.21.15.43:0/1229431003 wait complete. 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: shutdown 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/449743861 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 msgr2=0x7f0460138fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 --2- 172.21.15.43:0/449743861 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 0x7f0460138fc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0450009870 tx=0x7f0450024cc0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/449743861 shutdown_connections 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 --2- 172.21.15.43:0/449743861 >> v2:172.21.15.43:3300/0 conn(0x7f046006dbe0 0x7f046006e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 --2- 172.21.15.43:0/449743861 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046006d800 0x7f0460138fc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/449743861 >> 172.21.15.43:0/449743861 conn(0x7f04600a9690 msgr2=0x7f04600a56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/449743861 shutdown_connections 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/449743861 wait complete. 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient: monmap: 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: build_initial_monmap 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: monmap: 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 librados: starting msgr at 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 librados: starting msgr at 2024-04-19T00:44:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 librados: starting objecter 2024-04-19T00:44:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 librados: starting objecter 2024-04-19T00:44:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 Processor -- start 2024-04-19T00:44:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 Processor -- start 2024-04-19T00:44:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- start start 2024-04-19T00:44:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 librados: setting wanted keys 2024-04-19T00:44:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 librados: calling monclient init 2024-04-19T00:44:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: init 2024-04-19T00:44:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- start start 2024-04-19T00:44:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 librados: setting wanted keys 2024-04-19T00:44:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 librados: calling monclient init 2024-04-19T00:44:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient: init 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: _add_conns ranks=[4,8,7,3,5,6,2,0,1] 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 0x7f04601303f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f046006d800 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046006dbe0 0x7f0460130930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): picked mon.noname-e con 0x7f046006dbe0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0460135400 0x7f04600a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): picked mon.noname-i con 0x7f0460135400 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f046009e290 con 0x7f046006d800 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f046009e410 con 0x7f0460135400 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f046009e590 con 0x7f046006dbe0 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient(hunting): authenticate will time out at 945.810503s 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 0x7f04601303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 0x7f04601303f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48530/0 (socket says 172.21.15.43:48530) 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 -- 172.21.15.43:0/896936776 learned_addr learned my addr 172.21.15.43:0/896936776 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): get_auth_request con 0x7f046006d800 auth_method 0 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f0457fff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046006dbe0 0x7f0460130930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f0457fff640 10 monclient(hunting): get_auth_request con 0x7f046006dbe0 auth_method 0 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f0457fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f0457fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f0457fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045f7fe640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0460135400 0x7f04600a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045f7fe640 10 monclient(hunting): get_auth_request con 0x7f0460135400 auth_method 0 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-04-19T00:44:35.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 -- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046006dbe0 msgr2=0x7f0460130930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046006dbe0 0x7f0460130930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 -- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0460135400 msgr2=0x7f04600a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0460135400 0x7f04600a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 monclient: found mon.noname-c 2024-04-19T00:44:35.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 -- 172.21.15.43:0/896936776 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046006f1e0 con 0x7f046006d800 2024-04-19T00:44:35.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient: _add_conns ranks=[4,3,8,7,5,6,1,0,2] 2024-04-19T00:44:35.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f69900a0a40 0x7f6990130b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): picked mon.noname-c con 0x7f69900a0a40 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:35.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6990131f40 0x7f69901310b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): picked mon.noname-g con 0x7f6990131f40 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:35.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f0457fff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046006dbe0 0x7f0460130930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:35.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045effd640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 0x7f04601303f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04500b8040 tx=0x7f0450021a60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 client.?.objecter ms_handle_connect 0x7f046006d800 2024-04-19T00:44:35.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6990132380 0x7f69901315f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): picked mon.noname-e con 0x7f6990132380 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f699013e5b0 con 0x7f6990131f40 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6990131f40 0x7f69901310b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f6990135ef0 con 0x7f69900a0a40 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f699013e730 con 0x7f6990132380 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6990131f40 0x7f69901310b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:36538/0 (socket says 172.21.15.43:36538) 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 1 -- 172.21.15.43:0/2969481431 learned_addr learned my addr 172.21.15.43:0/2969481431 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698ffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f69900a0a40 0x7f6990130b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6990132380 0x7f69901315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04500c5070 con 0x7f046006d800 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: dump: 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.759 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f69962e0640 10 monclient(hunting): authenticate will time out at 945.811643s 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0450003a90 con 0x7f046006d800 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04500ba070 con 0x7f046006d800 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045dffb640 4 set_mon_vals no callback set 2024-04-19T00:44:35.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f04577fe640 10 monclient: dump: 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.762 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 5 monclient: authenticate success, global_id 4279 2024-04-19T00:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: _renew_subs 2024-04-19T00:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046006ec90 con 0x7f046006d800 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): get_auth_request con 0x7f6990131f40 auth_method 0 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): get_auth_request con 0x7f6990132380 auth_method 0 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698ffff640 10 monclient(hunting): get_auth_request con 0x7f69900a0a40 auth_method 0 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f699487d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f698f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 -- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f69900a0a40 msgr2=0x7f6990130b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f69900a0a40 0x7f6990130b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 -- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6990131f40 msgr2=0x7f69901310b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6990131f40 0x7f69901310b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 monclient: found mon.noname-e 2024-04-19T00:44:35.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 -- 172.21.15.43:0/2969481431 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6990139200 con 0x7f6990132380 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: _renew_subs 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0460120300 con 0x7f046006d800 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.680+0000 7f045ffff640 1 librados: init done 2024-04-19T00:44:35.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f045ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f045ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f045ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f046006a560 con 0x7f046006d800 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f699487d640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6990132380 0x7f69901315f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69900695f0 tx=0x7f6988009bd0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 client.?.objecter ms_handle_connect 0x7f6990132380 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f698800fd00 con 0x7f6990132380 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:35.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: dump: 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.769 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6988012460 con 0x7f6990132380 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f698800f890 con 0x7f6990132380 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 monclient: dump: 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:35.770 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 5 monclient: authenticate success, global_id 4346 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 monclient: _renew_subs 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f699013a990 con 0x7f6990132380 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0450003c40 con 0x7f046006d800 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0440038e40 0x7f044003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f04500eb600 con 0x7f046006d800 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 10 client.4279.objecter ms_dispatch 0x7f0460000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 3 client.4279.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 3 client.4279.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f04577fe640 20 client.4279.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f0457fff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0440038e40 0x7f044003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f0457fff640 10 monclient: get_auth_request con 0x7f0440038e40 auth_method 0 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 monclient: _renew_subs 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f699009cc10 con 0x7f6990132380 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 1 librados: init done 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6990138f50 con 0x7f6990132380 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6988012610 con 0x7f6990132380 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f697c03d230 0x7f697c03f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f6988048fc0 con 0x7f6990132380 2024-04-19T00:44:35.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 10 client.4346.objecter ms_dispatch 0x7f6990000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 3 client.4346.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 3 client.4346.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698cff9640 20 client.4346.objecter dump_active .. 0 homeless 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698ffff640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f697c03d230 0x7f697c03f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.684+0000 7f698ffff640 10 monclient: get_auth_request con 0x7f697c03d230 auth_method 0 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.688+0000 7f698e7fc640 4 set_mon_vals no callback set 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.688+0000 7f698cff9640 10 client.4346.objecter ms_handle_connect 0x7f697c03d230 2024-04-19T00:44:35.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.688+0000 7f698ffff640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f697c03d230 0x7f697c03f6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f69800af000 tx=0x7f69800a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.688+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f045001e940 con 0x7f046006d800 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.688+0000 7f04577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.688+0000 7f04577fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.692+0000 7f04577fe640 10 client.4279.objecter ms_handle_connect 0x7f0440038e40 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.692+0000 7f0457fff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0440038e40 0x7f044003b300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04600a1a50 tx=0x7f0448008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.696+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6988017030 con 0x7f6990132380 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.696+0000 7f698cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.696+0000 7f698cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.780+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc858016130 con 0x7fc8680a5bb0 2024-04-19T00:44:35.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.780+0000 7fc864ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:35.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.780+0000 7fc864ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.828+0000 7fb9c57fa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T00:44:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.828+0000 7fb9c57fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T00:44:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.828+0000 7fb9c57fa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.828+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb9b4005380 con 0x7fb9d0138a50 2024-04-19T00:44:35.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c77fe640 1 -- 172.21.15.43:0/2473475023 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb9b8005de0 con 0x7fb9d0138a50 2024-04-19T00:44:35.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T00:44:35.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c77fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:35.834 INFO:teuthology.orchestra.run.smithi043.stdout:51539607555 2024-04-19T00:44:35.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 10 librados: watch_flush enter 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 10 librados: watch_flush exit 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 20 client.4252.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 10 client.4252.objecter successfully canceled tick 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb9b0038e90 msgr2=0x7fb9b003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb9b0038e90 0x7fb9b003b350 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0009710 tx=0x7fb9c00a6000 comp rx=0 tx=0).stop 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 10 monclient: shutdown 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb9d0138a50 msgr2=0x7fb9d013a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb9d0138a50 0x7fb9d013a0a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8007fd0 tx=0x7fb9b800c890 comp rx=0 tx=0).stop 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 shutdown_connections 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb9d0139370 0x7fb9d013a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb9d0138a50 0x7fb9d013a0a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb9b0038e90 0x7fb9b003b350 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 --2- 172.21.15.43:0/2473475023 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb9d00a5bb0 0x7fb9d0139b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 >> 172.21.15.43:0/2473475023 conn(0x7fb9d00a7cb0 msgr2=0x7fb9d00a90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 shutdown_connections 2024-04-19T00:44:35.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 -- 172.21.15.43:0/2473475023 wait complete. 2024-04-19T00:44:35.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.832+0000 7fb9c57fa640 1 librados: shutdown 2024-04-19T00:44:35.853 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.3 2024-04-19T00:44:35.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc107fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T00:44:35.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc107fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T00:44:35.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc107fff640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fc100005380 con 0x7fc1240a5bb0 2024-04-19T00:44:35.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc11a7fc640 1 -- 172.21.15.43:0/3342913925 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc124137600 con 0x7fc1240a5bb0 2024-04-19T00:44:35.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc11a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T00:44:35.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.860+0000 7fc11a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:35.864 INFO:teuthology.orchestra.run.smithi043.stdout:51539607556 2024-04-19T00:44:35.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 10 librados: watch_flush enter 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 10 librados: watch_flush exit 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 20 client.4319.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 10 client.4319.objecter successfully canceled tick 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc108038d70 msgr2=0x7fc10803b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc108038d70 0x7fc10803b230 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc12406e8c0 tx=0x7fc114008040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 10 monclient: shutdown 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 msgr2=0x7fc124134180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 0x7fc124134180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc1200097c0 tx=0x7fc120019200 comp rx=0 tx=0).stop 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 shutdown_connections 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc124138a50 0x7fc124134c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc124137960 0x7fc1241346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc108038d70 0x7fc10803b230 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 --2- 172.21.15.43:0/3342913925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc1240a5bb0 0x7fc124134180 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 >> 172.21.15.43:0/3342913925 conn(0x7fc1240a7cb0 msgr2=0x7fc1240a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 shutdown_connections 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 -- 172.21.15.43:0/3342913925 wait complete. 2024-04-19T00:44:35.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.864+0000 7fc107fff640 1 librados: shutdown 2024-04-19T00:44:35.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.4 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fdeac121d00 con 0x7fdeac138a50 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdea1ffb640 1 -- 172.21.15.43:0/886947788 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdea4015440 con 0x7fdeac138a50 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdea1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T00:44:35.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdea1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:35.899 INFO:teuthology.orchestra.run.smithi043.stdout:47244640257 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 librados: watch_flush enter 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 librados: watch_flush exit 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 20 client.4310.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 client.4310.objecter successfully canceled tick 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde90038e40 msgr2=0x7fde9003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde90038e40 0x7fde9003b300 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdea8006fd0 tx=0x7fdea8008040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 10 monclient: shutdown 2024-04-19T00:44:35.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 msgr2=0x7fdeac13c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 0x7fdeac13c840 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdea400ced0 tx=0x7fdea400a710 comp rx=0 tx=0).stop 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 shutdown_connections 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde90038e40 0x7fde9003b300 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdeac138a50 0x7fdeac13c840 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdeac137960 0x7fdeac1369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 --2- 172.21.15.43:0/886947788 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdeac0a5bb0 0x7fdeac1d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 >> 172.21.15.43:0/886947788 conn(0x7fdeac0a7cb0 msgr2=0x7fdeac0a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 shutdown_connections 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 -- 172.21.15.43:0/886947788 wait complete. 2024-04-19T00:44:35.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.896+0000 7fdeb3310640 1 librados: shutdown 2024-04-19T00:44:35.916 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.1 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f045ffff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f045ffff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f045ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f0460076190 con 0x7f046006d800 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f04577fe640 1 -- 172.21.15.43:0/896936776 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f04500bec40 con 0x7f046006d800 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f04577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T00:44:35.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.968+0000 7f04577fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:35.973 INFO:teuthology.orchestra.run.smithi043.stdout:47244640259 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 10 librados: watch_flush enter 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 10 librados: watch_flush exit 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 20 client.4279.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 10 client.4279.objecter successfully canceled tick 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0440038e40 msgr2=0x7f044003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0440038e40 0x7f044003b300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f04600a1a50 tx=0x7f0448008040 comp rx=0 tx=0).stop 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 10 monclient: shutdown 2024-04-19T00:44:35.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 msgr2=0x7f04601303f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 0x7f04601303f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04500b8040 tx=0x7f0450021a60 comp rx=0 tx=0).stop 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 shutdown_connections 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0440038e40 0x7f044003b300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046006dbe0 0x7f0460130930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0460135400 0x7f04600a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 --2- 172.21.15.43:0/896936776 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f046006d800 0x7f04601303f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 >> 172.21.15.43:0/896936776 conn(0x7f04600a9690 msgr2=0x7f04601336f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 shutdown_connections 2024-04-19T00:44:35.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 -- 172.21.15.43:0/896936776 wait complete. 2024-04-19T00:44:35.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.972+0000 7f045ffff640 1 librados: shutdown 2024-04-19T00:44:35.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T00:44:35.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T00:44:35.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:35.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f69900768c0 con 0x7f6990132380 2024-04-19T00:44:35.982 INFO:teuthology.orchestra.run.smithi043.stdout:51539607556 2024-04-19T00:44:35.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f698cff9640 1 -- 172.21.15.43:0/2969481431 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6988012900 con 0x7f6990132380 2024-04-19T00:44:35.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f698cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T00:44:35.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f698cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:35.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 librados: watch_flush enter 2024-04-19T00:44:35.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 librados: watch_flush exit 2024-04-19T00:44:35.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 20 client.4346.objecter shutdown clearing up homeless session... 2024-04-19T00:44:35.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 client.4346.objecter successfully canceled tick 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f697c03d230 msgr2=0x7f697c03f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f697c03d230 0x7f697c03f6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f69800af000 tx=0x7f69800a7000 comp rx=0 tx=0).stop 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 10 monclient: shutdown 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6990132380 msgr2=0x7f69901315f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6990132380 0x7f69901315f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69900695f0 tx=0x7f6988009bd0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 shutdown_connections 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f697c03d230 0x7f697c03f6f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6990132380 0x7f69901315f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6990131f40 0x7f69901310b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 --2- 172.21.15.43:0/2969481431 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f69900a0a40 0x7f6990130b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 >> 172.21.15.43:0/2969481431 conn(0x7f69900a3df0 msgr2=0x7f69900a01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 shutdown_connections 2024-04-19T00:44:35.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 -- 172.21.15.43:0/2969481431 wait complete. 2024-04-19T00:44:35.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:35.976+0000 7f69962e0640 1 librados: shutdown 2024-04-19T00:44:35.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2024-04-19T00:44:35.996 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.5 2024-04-19T00:44:36.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.000+0000 7fc86e479640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T00:44:36.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.000+0000 7fc86e479640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T00:44:36.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.000+0000 7fc86e479640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:44:36.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.000+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc868076190 con 0x7fc8680a5bb0 2024-04-19T00:44:36.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc864ff9640 1 -- 172.21.15.43:0/3775468797 <== mon.5 v2:172.21.15.161:3302/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc8580aad30 con 0x7fc8680a5bb0 2024-04-19T00:44:36.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc864ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T00:44:36.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc864ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:36.006 INFO:teuthology.orchestra.run.smithi043.stdout:47244640257 2024-04-19T00:44:36.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 10 librados: watch_flush enter 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 10 librados: watch_flush exit 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 20 client.4271.objecter shutdown clearing up homeless session... 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 10 client.4271.objecter successfully canceled tick 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc854038da0 msgr2=0x7fc85403b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc854038da0 0x7fc85403b260 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc86809de00 tx=0x7fc85c009190 comp rx=0 tx=0).stop 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 10 monclient: shutdown 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 msgr2=0x7fc8680a4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 0x7fc8680a4640 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc858009870 tx=0x7fc858019190 comp rx=0 tx=0).stop 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 shutdown_connections 2024-04-19T00:44:36.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc854038da0 0x7fc85403b260 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:36.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc868138a50 0x7fc8681354e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:36.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc868137960 0x7fc8680a4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:36.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 --2- 172.21.15.43:0/3775468797 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc8680a5bb0 0x7fc8680a4640 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:36.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 >> 172.21.15.43:0/3775468797 conn(0x7fc8680a7cb0 msgr2=0x7fc86806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:36.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 shutdown_connections 2024-04-19T00:44:36.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 -- 172.21.15.43:0/3775468797 wait complete. 2024-04-19T00:44:36.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.004+0000 7fc86e479640 1 librados: shutdown 2024-04-19T00:44:36.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640257 for osd.0 2024-04-19T00:44:36.854 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-19T00:44:36.882 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-19T00:44:36.917 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-19T00:44:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient: get_monmap_and_config 2024-04-19T00:44:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient: build_initial_monmap 2024-04-19T00:44:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient: monmap: 2024-04-19T00:44:36.989 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:36.991286+0000 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:36.991286+0000 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:36.990 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:36.991 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:36.991 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:36.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 Processor -- start 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 -- start start 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient: init 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient: _add_conns ranks=[0,10,4,8,5,7,9,1,6,3,2] 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f993c0a4b10 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f993c0a4ef0 0x7f993c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): picked mon.noname-f con 0x7f993c0a4ef0 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f993c13e170 0x7f993c140560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): picked mon.noname-c con 0x7f993c13e170 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f993c1398f0 con 0x7f993c0a4b10 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:36.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f993c06e750 con 0x7f993c0a4ef0 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f993c06f250 con 0x7f993c13e170 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f994468a640 10 monclient(hunting): authenticate will time out at 947.118354s 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942426640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f993c0a4b10 0x7f993c1391e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44770/0 (socket says 172.21.15.43:44770) 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942426640 1 -- 172.21.15.43:0/2236440671 learned_addr learned my addr 172.21.15.43:0/2236440671 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 1 --2- 172.21.15.43:0/2236440671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f993c13e170 0x7f993c140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): get_auth_request con 0x7f993c13e170 auth_method 0 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:36.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941c25640 1 --2- 172.21.15.43:0/2236440671 >> v2:172.21.15.161:3300/0 conn(0x7f993c0a4ef0 0x7f993c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941c25640 10 monclient(hunting): get_auth_request con 0x7f993c0a4ef0 auth_method 0 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941c25640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941c25640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941424640 1 -- 172.21.15.43:0/2236440671 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828457434 0 0) 0x7f993c1398f0 con 0x7f993c0a4b10 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941424640 10 monclient(hunting): _negotiate 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941424640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941424640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941424640 10 monclient(hunting): my global_id is 4311 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941424640 1 -- 172.21.15.43:0/2236440671 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9928003530 con 0x7f993c0a4b10 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 1 -- 172.21.15.43:0/2236440671 >> v2:172.21.15.161:3300/0 conn(0x7f993c0a4ef0 msgr2=0x7f993c0a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:36.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 1 --2- 172.21.15.43:0/2236440671 >> v2:172.21.15.161:3300/0 conn(0x7f993c0a4ef0 0x7f993c0a5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 1 -- 172.21.15.43:0/2236440671 >> v1:172.21.15.43:6789/0 conn(0x7f993c0a4b10 legacy=0x7f993c1391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient: _finish_hunting 0 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 1 monclient: found mon.noname-c 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9942c27640 1 -- 172.21.15.43:0/2236440671 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c06f530 con 0x7f993c13e170 2024-04-19T00:44:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.988+0000 7f9941c25640 1 --2- 172.21.15.43:0/2236440671 >> v2:172.21.15.161:3300/0 conn(0x7f993c0a4ef0 0x7f993c0a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:36.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-19T00:44:36.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-19T00:44:37.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9942c27640 1 --2- 172.21.15.43:0/2236440671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f993c13e170 0x7f993c140560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9934009d90 tx=0x7f993401c830 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f7f38abdcc692565 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 1 -- 172.21.15.43:0/2236440671 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99340b2070 con 0x7f993c13e170 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: dump: 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.004 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 1 -- 172.21.15.43:0/2236440671 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99340a5a70 con 0x7f993c13e170 2024-04-19T00:44:37.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 5 monclient: authenticate success, global_id 4297 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 4 set_mon_vals no callback set 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 1 -- 172.21.15.43:0/2236440671 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f99340a5c60 con 0x7f993c13e170 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f9941424640 10 monclient: dump: 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.006 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 10 monclient: shutdown 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 -- 172.21.15.43:0/2236440671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f993c13e170 msgr2=0x7f993c140560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 --2- 172.21.15.43:0/2236440671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f993c13e170 0x7f993c140560 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9934009d90 tx=0x7f993401c830 comp rx=0 tx=0).stop 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 -- 172.21.15.43:0/2236440671 shutdown_connections 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 --2- 172.21.15.43:0/2236440671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f993c13e170 0x7f993c140560 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 --2- 172.21.15.43:0/2236440671 >> v2:172.21.15.161:3300/0 conn(0x7f993c0a4ef0 0x7f993c0a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 -- 172.21.15.43:0/2236440671 >> 172.21.15.43:0/2236440671 conn(0x7f993c0a98c0 msgr2=0x7f993c0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 -- 172.21.15.43:0/2236440671 shutdown_connections 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.992+0000 7f994468a640 1 -- 172.21.15.43:0/2236440671 wait complete. 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: monmap: 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.008 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 librados: starting msgr at 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 librados: starting objecter 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 Processor -- start 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- start start 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 librados: setting wanted keys 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 librados: calling monclient init 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: init 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _add_conns ranks=[7,6,3,2,5,0,1,8,4] 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f993c0a4b10 0x7f993c12daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): picked mon.noname-i con 0x7f993c0a4b10 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 0x7f993c12e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): picked mon.noname-d con 0x7f993c0a4ef0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:37.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f993c13e170 0x7f993c0a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): picked mon.noname-g con 0x7f993c13e170 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f993c143650 con 0x7f993c13e170 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f993c1434d0 con 0x7f993c0a4ef0 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f993c1437d0 con 0x7f993c0a4b10 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient(hunting): authenticate will time out at 947.125188s 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 0x7f993c12e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 0x7f993c12e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42178/0 (socket says 172.21.15.43:42178) 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 -- 172.21.15.43:0/1056119682 learned_addr learned my addr 172.21.15.43:0/1056119682 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): get_auth_request con 0x7f993c0a4ef0 auth_method 0 2024-04-19T00:44:37.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9942426640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f993c0a4b10 0x7f993c12daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9942c27640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f993c13e170 0x7f993c0a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient(hunting): handle_auth_done global_id 4245 payload 824 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 -- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f993c0a4b10 msgr2=0x7f993c12daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f993c0a4b10 0x7f993c12daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 -- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f993c13e170 msgr2=0x7f993c0a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f993c13e170 0x7f993c0a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 monclient: found mon.noname-d 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 -- 172.21.15.43:0/1056119682 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c06e4d0 con 0x7f993c0a4ef0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9942c27640 10 monclient: get_auth_request con 0x7f993c13e170 auth_method 0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9942c27640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f993c13e170 0x7f993c0a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9941c25640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 0x7f993c12e030 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f992c00e810 tx=0x7f992c00ecd0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 client.?.objecter ms_handle_connect 0x7f993c0a4ef0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f992c017070 con 0x7f993c0a4ef0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: dump: 2024-04-19T00:44:37.013 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.014 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f992c00cb40 con 0x7f993c0a4ef0 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 5 monclient: authenticate success, global_id 4245 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _renew_subs 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993c06e920 con 0x7f993c0a4ef0 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f992c00f040 con 0x7f993c0a4ef0 2024-04-19T00:44:37.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 monclient: dump: 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.016 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9940c23640 4 set_mon_vals no callback set 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _renew_subs 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f993c13ba00 con 0x7f993c0a4ef0 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 librados: init done 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f992c01a070 con 0x7f993c0a4ef0 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f993c069630 con 0x7f993c0a4ef0 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9920038e10 0x7f992003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f992c048890 con 0x7f993c0a4ef0 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 10 client.4245.objecter ms_dispatch 0x7f993c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 3 client.4245.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 3 client.4245.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9932ffd640 20 client.4245.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9942426640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9920038e10 0x7f992003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:36.996+0000 7f9942426640 10 monclient: get_auth_request con 0x7f9920038e10 auth_method 0 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.000+0000 7f9932ffd640 10 client.4245.objecter ms_handle_connect 0x7f9920038e10 2024-04-19T00:44:37.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.000+0000 7f9942426640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9920038e10 0x7f992003b2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9938007fd0 tx=0x7f9938009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.004+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f992c00cda0 con 0x7f993c0a4ef0 2024-04-19T00:44:37.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.004+0000 7f9932ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.004+0000 7f9932ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient: monmap: 2024-04-19T00:44:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.022 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.021140+0000 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.021140+0000 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.023 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 Processor -- start 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 -- start start 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient: init 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient: _add_conns ranks=[9,4,3,1,2,8,10,0,6,7,5] 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdddc0729f0 0x7fdddc0a3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): picked mon.noname-a con 0x7fdddc0729f0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 0x7fdddc1368b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): picked mon.noname-c con 0x7fdddc072dd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0731f0 0x7fdddc136df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): picked mon.noname-g con 0x7fdddc0731f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fdddc0a0100 con 0x7fdddc0729f0 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdddc0a11a0 con 0x7fdddc0731f0 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdddc0a1480 con 0x7fdddc072dd0 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.016+0000 7fdde37d9640 10 monclient(hunting): authenticate will time out at 947.147928s 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 0x7fdddc1368b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdddc0729f0 0x7fdddc0a3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 0x7fdddc1368b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48554/0 (socket says 172.21.15.43:48554) 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 -- 172.21.15.43:0/496588132 learned_addr learned my addr 172.21.15.43:0/496588132 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): get_auth_request con 0x7fdddc072dd0 auth_method 0 2024-04-19T00:44:37.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): get_auth_request con 0x7fdddc0729f0 auth_method 0 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 1 --2- 172.21.15.43:0/496588132 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0731f0 0x7fdddc136df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): get_auth_request con 0x7fdddc0731f0 auth_method 0 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 -- 172.21.15.43:0/496588132 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0731f0 msgr2=0x7fdddc136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 --2- 172.21.15.43:0/496588132 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0731f0 0x7fdddc136df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 -- 172.21.15.43:0/496588132 >> v2:172.21.15.43:3300/0 conn(0x7fdddc0729f0 msgr2=0x7fdddc0a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 --2- 172.21.15.43:0/496588132 >> v2:172.21.15.43:3300/0 conn(0x7fdddc0729f0 0x7fdddc0a3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 monclient: found mon.noname-c 2024-04-19T00:44:37.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:37.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 -- 172.21.15.43:0/496588132 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdddc0a38b0 con 0x7fdddc072dd0 2024-04-19T00:44:37.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde1575640 1 --2- 172.21.15.43:0/496588132 >> v2:172.21.15.43:3300/0 conn(0x7fdddc0729f0 0x7fdddc0a3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:37.028 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde0d74640 1 --2- 172.21.15.43:0/496588132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 0x7fdddc1368b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009870 tx=0x7fddcc01c420 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1f0c11892676cfbf server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 1 -- 172.21.15.43:0/496588132 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fddcc0b0070 con 0x7fdddc072dd0 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: dump: 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.033 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.034 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 1 -- 172.21.15.43:0/496588132 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddcc0aa810 con 0x7fdddc072dd0 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 5 monclient: authenticate success, global_id 4306 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 1 -- 172.21.15.43:0/496588132 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fddcc0a9300 con 0x7fdddc072dd0 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 4 set_mon_vals no callback set 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:37.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fddd3fff640 10 monclient: dump: 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.036 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 10 monclient: shutdown 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 -- 172.21.15.43:0/496588132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 msgr2=0x7fdddc1368b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 --2- 172.21.15.43:0/496588132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 0x7fdddc1368b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009870 tx=0x7fddcc01c420 comp rx=0 tx=0).stop 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 -- 172.21.15.43:0/496588132 shutdown_connections 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 --2- 172.21.15.43:0/496588132 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0731f0 0x7fdddc136df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 --2- 172.21.15.43:0/496588132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdddc072dd0 0x7fdddc1368b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 --2- 172.21.15.43:0/496588132 >> v2:172.21.15.43:3300/0 conn(0x7fdddc0729f0 0x7fdddc0a3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.020+0000 7fdde37d9640 1 -- 172.21.15.43:0/496588132 >> 172.21.15.43:0/496588132 conn(0x7fdddc0a4460 msgr2=0x7fdddc0a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/496588132 shutdown_connections 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/496588132 wait complete. 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: monmap: 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.037 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.038 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 librados: starting msgr at 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 librados: starting objecter 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 Processor -- start 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- start start 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 librados: setting wanted keys 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 librados: calling monclient init 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: init 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: _add_conns ranks=[3,8,7,1,2,0,6,4,5] 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0729f0 0x7fdddc135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): picked mon.noname-g con 0x7fdddc0729f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 0x7fdddc1d09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 0x7fdddc1d09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 0x7fdddc1d09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59528/0 (socket says 172.21.15.43:59528) 2024-04-19T00:44:37.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): picked mon.noname-e con 0x7fdddc072dd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 -- 172.21.15.43:0/4079392547 learned_addr learned my addr 172.21.15.43:0/4079392547 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdddc0731f0 0x7fdddc1d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): picked mon.noname-i con 0x7fdddc0731f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdddc1427e0 con 0x7fdddc0729f0 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdddc137740 con 0x7fdddc0731f0 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdddc142960 con 0x7fdddc072dd0 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient(hunting): authenticate will time out at 947.154673s 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): get_auth_request con 0x7fdddc072dd0 auth_method 0 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0729f0 0x7fdddc135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): get_auth_request con 0x7fdddc0729f0 auth_method 0 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1d76640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdddc0731f0 0x7fdddc1d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1575640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1d76640 10 monclient(hunting): get_auth_request con 0x7fdddc0731f0 auth_method 0 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1d76640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde1d76640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 -- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdddc0731f0 msgr2=0x7fdddc1d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdddc0731f0 0x7fdddc1d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 -- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0729f0 msgr2=0x7fdddc135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0729f0 0x7fdddc135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 monclient: found mon.noname-e 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdddc0a6630 con 0x7fdddc072dd0 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde0d74640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 0x7fdddc1d09f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0ae040 tx=0x7fddcc0191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 client.?.objecter ms_handle_connect 0x7fdddc072dd0 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fddcc0b9070 con 0x7fdddc072dd0 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: dump: 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.043 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddcc0044b0 con 0x7fdddc072dd0 2024-04-19T00:44:37.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 5 monclient: authenticate success, global_id 4355 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: _renew_subs 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdddc134830 con 0x7fdddc072dd0 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fddcc0b0040 con 0x7fdddc072dd0 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd1ffb640 10 monclient: dump: 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.045 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: _renew_subs 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdddc134df0 con 0x7fdddc072dd0 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fdde37d9640 1 librados: init done 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.024+0000 7fddd37fe640 4 set_mon_vals no callback set 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde37d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde37d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde37d9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdddc0008e0 con 0x7fdddc072dd0 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fddcc004040 con 0x7fdddc072dd0 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fddc0038e10 0x7fddc003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fddcc0e2060 con 0x7fdddc072dd0 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 10 client.4355.objecter ms_dispatch 0x7fdddc000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 3 client.4355.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 3 client.4355.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 20 client.4355.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde1575640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fddc0038e10 0x7fddc003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde1575640 10 monclient: get_auth_request con 0x7fddc0038e10 auth_method 0 2024-04-19T00:44:37.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fdde1575640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fddc0038e10 0x7fddc003b2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009580 tx=0x7fddd8009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.028+0000 7fddd1ffb640 10 client.4355.objecter ms_handle_connect 0x7fddc0038e10 2024-04-19T00:44:37.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.036+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fddcc016130 con 0x7fdddc072dd0 2024-04-19T00:44:37.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.036+0000 7fddd1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.036+0000 7fddd1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.104+0000 7f0470c26640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.104+0000 7f0470c26640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.104+0000 7f0470c26640 10 monclient: monmap: 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.110156+0000 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.110156+0000 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.109 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.110 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.110 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 Processor -- start 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 -- start start 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient: init 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient: _add_conns ranks=[5,8,9,7,0,1,3,2,4,6,10] 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f046c138a50 0x7f046c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): picked mon.noname-h con 0x7f046c138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046c0a5bb0 0x7f046c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): picked mon.noname-e con 0x7f046c0a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f046c137960 0x7f046c137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): picked mon.noname-a con 0x7f046c137960 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f046c0a1020 con 0x7f046c137960 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f046c1398b0 con 0x7f046c138a50 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f046c0a0780 con 0x7f046c0a5bb0 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0470c26640 10 monclient(hunting): authenticate will time out at 947.238280s 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f046c137960 0x7f046c137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f046c138a50 0x7f046c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f046c137960 0x7f046c137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54178/0 (socket says 172.21.15.43:54178) 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 -- 172.21.15.43:0/1999484677 learned_addr learned my addr 172.21.15.43:0/1999484677 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): get_auth_request con 0x7f046c137960 auth_method 0 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): get_auth_request con 0x7f046c138a50 auth_method 0 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046c0a5bb0 0x7f046c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): get_auth_request con 0x7f046c0a5bb0 auth_method 0 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f0469d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient(hunting): handle_auth_done global_id 4329 payload 274 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 -- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046c0a5bb0 msgr2=0x7f046c139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046c0a5bb0 0x7f046c139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 -- 172.21.15.43:0/1999484677 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f046c138a50 msgr2=0x7f046c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f046c138a50 0x7f046c138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 monclient: found mon.noname-a 2024-04-19T00:44:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.108+0000 7f046ad9d640 1 -- 172.21.15.43:0/1999484677 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c0a0d40 con 0x7f046c137960 2024-04-19T00:44:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046ad9d640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c137960 0x7f046c137dc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0464009e50 tx=0x7f046401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147e0fd76e8c3177 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 1 -- 172.21.15.43:0/1999484677 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04640090d0 con 0x7f046c137960 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: dump: 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.121 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f0470c26640 5 monclient: authenticate success, global_id 4329 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f0470c26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 1 -- 172.21.15.43:0/1999484677 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04640a5b90 con 0x7f046c137960 2024-04-19T00:44:37.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f0470c26640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f0470c26640 4 set_mon_vals no callback set 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 1 -- 172.21.15.43:0/1999484677 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04640a5d80 con 0x7f046c137960 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.112+0000 7f046959a640 10 monclient: dump: 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.123 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient: shutdown 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c137960 msgr2=0x7f046c137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c137960 0x7f046c137dc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0464009e50 tx=0x7f046401c8e0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- 172.21.15.43:0/1999484677 shutdown_connections 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f046c137960 0x7f046c137dc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f046c0a5bb0 0x7f046c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- 172.21.15.43:0/1999484677 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f046c138a50 0x7f046c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- 172.21.15.43:0/1999484677 >> 172.21.15.43:0/1999484677 conn(0x7f046c0a7cb0 msgr2=0x7f046c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- 172.21.15.43:0/1999484677 shutdown_connections 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- 172.21.15.43:0/1999484677 wait complete. 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient: monmap: 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.125 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 librados: starting msgr at 2024-04-19T00:44:37.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 librados: starting objecter 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 Processor -- start 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- start start 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 librados: setting wanted keys 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 librados: calling monclient init 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient: init 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient: _add_conns ranks=[1,7,2,5,0,3,8,4,6] 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f046c0a5bb0 0x7f046c1d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): picked mon.noname-f con 0x7f046c0a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f046c137960 0x7f046c136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): picked mon.noname-i con 0x7f046c137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 0x7f046c13c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): picked mon.noname-b con 0x7f046c138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:37.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f046c0a2100 con 0x7f046c0a5bb0 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f046c0a1f80 con 0x7f046c138a50 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f046c142810 con 0x7f046c137960 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0470c26640 10 monclient(hunting): authenticate will time out at 947.247041s 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 0x7f046c13c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 0x7f046c13c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35304/0 (socket says 172.21.15.43:35304) 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 -- 172.21.15.43:0/2217413709 learned_addr learned my addr 172.21.15.43:0/2217413709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): get_auth_request con 0x7f046c138a50 auth_method 0 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f046c0a5bb0 0x7f046c1d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): get_auth_request con 0x7f046c0a5bb0 auth_method 0 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0469d9b640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f046c137960 0x7f046c136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0469d9b640 10 monclient(hunting): get_auth_request con 0x7f046c137960 auth_method 0 2024-04-19T00:44:37.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0469d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0469d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f0469d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 -- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f046c137960 msgr2=0x7f046c136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f046c137960 0x7f046c136a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 -- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f046c0a5bb0 msgr2=0x7f046c1d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f046c0a5bb0 0x7f046c1d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 monclient: found mon.noname-b 2024-04-19T00:44:37.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 -- 172.21.15.43:0/2217413709 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046c13e510 con 0x7f046c138a50 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f046ad9d640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 0x7f046c13c840 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f046401c710 tx=0x7f0464019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f045affd640 10 client.?.objecter ms_handle_connect 0x7f046c138a50 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.116+0000 7f045affd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f04640bc020 con 0x7f046c138a50 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: dump: 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.131 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 5 monclient: authenticate success, global_id 4322 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 10 monclient: _renew_subs 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 1 -- 172.21.15.43:0/2217413709 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046c09ee60 con 0x7f046c138a50 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04640039a0 con 0x7f046c138a50 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 10 monclient: _renew_subs 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0464004040 con 0x7f046c138a50 2024-04-19T00:44:37.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 1 -- 172.21.15.43:0/2217413709 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f046c137720 con 0x7f046c138a50 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0470c26640 1 librados: init done 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0468d99640 4 set_mon_vals no callback set 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 monclient: dump: 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.135 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0458ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0458ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0458ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0444005090 con 0x7f046c138a50 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f04640bf020 con 0x7f046c138a50 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f044c038e10 0x7f044c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f04640adcf0 con 0x7f046c138a50 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 client.4322.objecter ms_dispatch 0x7f046c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 3 client.4322.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 3 client.4322.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 20 client.4322.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f046a59c640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f044c038e10 0x7f044c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f046a59c640 10 monclient: get_auth_request con 0x7f044c038e10 auth_method 0 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f045affd640 10 client.4322.objecter ms_handle_connect 0x7f044c038e10 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.120+0000 7f046a59c640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f044c038e10 0x7f044c03b2d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f045c009a40 tx=0x7f045c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.124+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f04640c6030 con 0x7f046c138a50 2024-04-19T00:44:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.124+0000 7f045affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.124+0000 7f045affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.128+0000 7f703517c640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.128+0000 7f703517c640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.128+0000 7f703517c640 10 monclient: monmap: 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.133953+0000 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.133953+0000 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.145 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 Processor -- start 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 -- start start 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient: init 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient: _add_conns ranks=[2,3,9,8,4,10,1,0,7,6,5] 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7030138a50 0x7f7030138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): picked mon.noname-b con 0x7f7030138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f7030139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): picked mon.noname-g con 0x7f70300a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7030137960 0x7f7030137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): picked mon.noname-a con 0x7f7030137960 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f70300a1020 con 0x7f7030137960 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f70301398b0 con 0x7f7030138a50 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70300a0780 con 0x7f70300a5bb0 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f703517c640 10 monclient(hunting): authenticate will time out at 947.260924s 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7030138a50 0x7f7030138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7030138a50 0x7f7030138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35310/0 (socket says 172.21.15.43:35310) 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 1 -- 172.21.15.43:0/4114498710 learned_addr learned my addr 172.21.15.43:0/4114498710 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): get_auth_request con 0x7f7030138a50 auth_method 0 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 --2- 172.21.15.43:0/4114498710 >> v2:172.21.15.43:3300/0 conn(0x7f7030137960 0x7f7030137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): get_auth_request con 0x7f7030137960 auth_method 0 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702e59c640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f7030139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702e59c640 10 monclient(hunting): get_auth_request con 0x7f70300a5bb0 auth_method 0 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 -- 172.21.15.43:0/4114498710 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 msgr2=0x7f7030139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f7030139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 -- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7030138a50 msgr2=0x7f7030138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7030138a50 0x7f7030138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 monclient: found mon.noname-a 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.132+0000 7f702f59e640 1 -- 172.21.15.43:0/4114498710 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70300a0d40 con 0x7f7030137960 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702f59e640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030137960 0x7f7030137dc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7028009d90 tx=0x7f702801c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9501a43ce9871505 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 1 -- 172.21.15.43:0/4114498710 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70280b2070 con 0x7f7030137960 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: dump: 2024-04-19T00:44:37.150 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.151 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 1 -- 172.21.15.43:0/4114498710 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70280a5ad0 con 0x7f7030137960 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 5 monclient: authenticate success, global_id 4338 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 1 -- 172.21.15.43:0/4114498710 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70280a5d00 con 0x7f7030137960 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f702dd9b640 10 monclient: dump: 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.152 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 4 set_mon_vals no callback set 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: shutdown 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030137960 msgr2=0x7f7030137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030137960 0x7f7030137dc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7028009d90 tx=0x7f702801c800 comp rx=0 tx=0).stop 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- 172.21.15.43:0/4114498710 shutdown_connections 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030137960 0x7f7030137dc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f7030139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- 172.21.15.43:0/4114498710 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7030138a50 0x7f7030138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- 172.21.15.43:0/4114498710 >> 172.21.15.43:0/4114498710 conn(0x7f70300a7cb0 msgr2=0x7f70300a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- 172.21.15.43:0/4114498710 shutdown_connections 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- 172.21.15.43:0/4114498710 wait complete. 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: monmap: 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.158 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.159 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 librados: starting msgr at 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 librados: starting objecter 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 Processor -- start 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- start start 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 librados: setting wanted keys 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 librados: calling monclient init 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: init 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient: _add_conns ranks=[3,7,0,1,2,6,5,4,8] 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f70301d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): picked mon.noname-g con 0x7f70300a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7030137960 0x7f7030136c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): picked mon.noname-i con 0x7f7030137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 0x7f703013ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): picked mon.noname-a con 0x7f7030138a50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f70300a2100 con 0x7f7030138a50 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70300a1f80 con 0x7f70300a5bb0 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7030142810 con 0x7f7030137960 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.136+0000 7f703517c640 10 monclient(hunting): authenticate will time out at 947.267952s 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 0x7f703013ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 0x7f703013ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54202/0 (socket says 172.21.15.43:54202) 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 -- 172.21.15.43:0/849879552 learned_addr learned my addr 172.21.15.43:0/849879552 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): get_auth_request con 0x7f7030138a50 auth_method 0 2024-04-19T00:44:37.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7030137960 0x7f7030136c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f70301d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): get_auth_request con 0x7f7030137960 auth_method 0 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): get_auth_request con 0x7f70300a5bb0 auth_method 0 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 -- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7030137960 msgr2=0x7f7030136c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7030137960 0x7f7030136c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 -- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 msgr2=0x7f70301d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f70301d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 monclient: found mon.noname-a 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 -- 172.21.15.43:0/849879552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703009fc90 con 0x7f7030138a50 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702f59e640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 0x7f703013ca50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f702801d0a0 tx=0x7f70280195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 client.?.objecter ms_handle_connect 0x7f7030138a50 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70280b2050 con 0x7f7030138a50 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: dump: 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.164 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70280be070 con 0x7f7030138a50 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7028003bb0 con 0x7f7030138a50 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 monclient: dump: 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.166 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 5 monclient: authenticate success, global_id 4347 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 monclient: _renew_subs 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703009ee60 con 0x7f7030138a50 2024-04-19T00:44:37.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 monclient: _renew_subs 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7030121d30 con 0x7f7030138a50 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 1 librados: init done 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702d59a640 4 set_mon_vals no callback set 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7028004040 con 0x7f7030138a50 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f701c038e10 0x7f701c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70300008e0 con 0x7f7030138a50 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f70280e3d80 con 0x7f7030138a50 2024-04-19T00:44:37.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 10 client.4347.objecter ms_dispatch 0x7f7030000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 3 client.4347.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 3 client.4347.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f70177fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.140+0000 7f702ed9d640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f701c038e10 0x7f701c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.148+0000 7f702ed9d640 10 monclient: get_auth_request con 0x7f701c038e10 auth_method 0 2024-04-19T00:44:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.148+0000 7f702ed9d640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f701c038e10 0x7f701c03b2d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7020009a30 tx=0x7f7020009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.148+0000 7f70177fe640 10 client.4347.objecter ms_handle_connect 0x7f701c038e10 2024-04-19T00:44:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.152+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7028016540 con 0x7f7030138a50 2024-04-19T00:44:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.152+0000 7f70177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.152+0000 7f70177fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient: monmap: 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.163079+0000 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.163079+0000 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.179 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 Processor -- start 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 -- start start 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient: init 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient: _add_conns ranks=[9,4,2,0,10,5,7,1,8,3,6] 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19780a02b0 0x7f19780a0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): picked mon.noname-a con 0x7f19780a02b0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:37.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 0x7f19780a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): picked mon.noname-c con 0x7f197809fe80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 0x7f1978138e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 0x7f19780a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 0x7f19780a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48570/0 (socket says 172.21.15.43:48570) 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 0x7f1978138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19780a02b0 0x7f19780a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 0x7f1978138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35322/0 (socket says 172.21.15.43:35322) 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f19780a02b0 0x7f19780a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54212/0 (socket says 172.21.15.43:54212) 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): picked mon.noname-b con 0x7f19780a1d70 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 1 -- 172.21.15.43:0/2613738876 learned_addr learned my addr 172.21.15.43:0/2613738876 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f19780a2df0 con 0x7f19780a02b0 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f19780a51f0 con 0x7f19780a1d70 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f19780a2b10 con 0x7f197809fe80 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f197d4e9640 10 monclient(hunting): authenticate will time out at 947.290933s 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): get_auth_request con 0x7f197809fe80 auth_method 0 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): get_auth_request con 0x7f19780a1d70 auth_method 0 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): get_auth_request con 0x7f19780a02b0 auth_method 0 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f1976ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 -- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 msgr2=0x7f19780a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 --2- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 0x7f19780a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 -- 172.21.15.43:0/2613738876 >> v2:172.21.15.43:3300/0 conn(0x7f19780a02b0 msgr2=0x7f19780a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 --2- 172.21.15.43:0/2613738876 >> v2:172.21.15.43:3300/0 conn(0x7f19780a02b0 0x7f19780a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 monclient: found mon.noname-b 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19777fe640 1 -- 172.21.15.43:0/2613738876 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19780a2830 con 0x7f19780a1d70 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.160+0000 7f19767fc640 1 --2- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 0x7f19780a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f19777fe640 1 --2- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 0x7f1978138e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1970009dc0 tx=0x7f197001c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb2be350100578e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 1 -- 172.21.15.43:0/2613738876 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19700aa8c0 con 0x7f19780a1d70 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: dump: 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.185 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 1 -- 172.21.15.43:0/2613738876 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19700af070 con 0x7f19780a1d70 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 1 -- 172.21.15.43:0/2613738876 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f19700a93b0 con 0x7f19780a1d70 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f197d4e9640 5 monclient: authenticate success, global_id 4340 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f197d4e9640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f197d4e9640 4 set_mon_vals no callback set 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.164+0000 7f1975ffb640 10 monclient: dump: 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.187 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient: shutdown 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 msgr2=0x7f1978138e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 0x7f1978138e70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1970009dc0 tx=0x7f197001c550 comp rx=0 tx=0).stop 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 shutdown_connections 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f19780a1d70 0x7f1978138e70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2613738876 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f197809fe80 0x7f19780a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2613738876 >> v2:172.21.15.43:3300/0 conn(0x7f19780a02b0 0x7f19780a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 >> 172.21.15.43:0/2613738876 conn(0x7f19780a8000 msgr2=0x7f19780a8410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 shutdown_connections 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- 172.21.15.43:0/2613738876 wait complete. 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient: monmap: 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.189 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 librados: starting msgr at 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 librados: starting objecter 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 Processor -- start 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- start start 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 librados: setting wanted keys 2024-04-19T00:44:37.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 librados: calling monclient init 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient: init 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient: _add_conns ranks=[7,5,8,3,1,2,6,0,4] 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f197809fe80 0x7f19781d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): picked mon.noname-i con 0x7f197809fe80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f19780a02b0 0x7f197813f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): picked mon.noname-h con 0x7f19780a02b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f19780a1d70 0x7f197813f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): picked mon.noname-e con 0x7f19780a1d70 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f1978142d50 con 0x7f19780a02b0 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1978142bd0 con 0x7f197809fe80 2024-04-19T00:44:37.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f1978142ed0 con 0x7f19780a1d70 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.168+0000 7f197d4e9640 10 monclient(hunting): authenticate will time out at 947.299985s 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f197809fe80 0x7f19781d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f19780a02b0 0x7f197813f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f19780a02b0 0x7f197813f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:42890/0 (socket says 172.21.15.43:42890) 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f19780a1d70 0x7f197813f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 1 -- 172.21.15.43:0/2624236748 learned_addr learned my addr 172.21.15.43:0/2624236748 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): get_auth_request con 0x7f19780a02b0 auth_method 0 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): get_auth_request con 0x7f19780a1d70 auth_method 0 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): get_auth_request con 0x7f197809fe80 auth_method 0 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f1976ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 -- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f197809fe80 msgr2=0x7f19781d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f197809fe80 0x7f19781d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 -- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f19780a02b0 msgr2=0x7f197813f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f19780a02b0 0x7f197813f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 monclient: found mon.noname-e 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 -- 172.21.15.43:0/2624236748 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19780a4ee0 con 0x7f19780a1d70 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19777fe640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f19780a1d70 0x7f197813f800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f19780a5370 tx=0x7f19700192f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 client.?.objecter ms_handle_connect 0x7f19780a1d70 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1970009450 con 0x7f19780a1d70 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: dump: 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.195 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.196 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19700a6040 con 0x7f19780a1d70 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1970004860 con 0x7f19780a1d70 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 10 monclient: dump: 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.197 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f19757fa640 4 set_mon_vals no callback set 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 5 monclient: authenticate success, global_id 4373 2024-04-19T00:44:37.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 monclient: _renew_subs 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197813bd70 con 0x7f19780a1d70 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 monclient: _renew_subs 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f197813b4d0 con 0x7f19780a1d70 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 1 librados: init done 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f197806d940 con 0x7f19780a1d70 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f19700043f0 con 0x7f19780a1d70 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.172+0000 7f195f7fe640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1964038e10 0x7f196403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f19700e3a80 con 0x7f19780a1d70 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f195f7fe640 10 client.4373.objecter ms_dispatch 0x7f1978000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f195f7fe640 3 client.4373.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f195f7fe640 3 client.4373.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f195f7fe640 20 client.4373.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f1976ffd640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1964038e10 0x7f196403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f1976ffd640 10 monclient: get_auth_request con 0x7f1964038e10 auth_method 0 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f195f7fe640 10 client.4373.objecter ms_handle_connect 0x7f1964038e10 2024-04-19T00:44:37.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f1976ffd640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1964038e10 0x7f196403b2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1968009ae0 tx=0x7f1968009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f42130d2640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f42130d2640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f42130d2640 10 monclient: monmap: 2024-04-19T00:44:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.202 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.180823+0000 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.180823+0000 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.203 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f42130d2640 1 Processor -- start 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f42130d2640 1 -- start start 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.176+0000 7f42130d2640 10 monclient: init 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient: _add_conns ranks=[2,5,0,9,4,1,7,10,3,6,8] 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f420402ad90 0x7f4204028b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): picked mon.noname-b con 0x7f420402ad90 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040ba120 0x7f4204029040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): picked mon.noname-h con 0x7f42040ba120 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f420402ad90 0x7f4204028b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f421166f640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f42040bd160 0x7f42040bd610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44774/0 (socket says 172.21.15.43:44774) 2024-04-19T00:44:37.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f421166f640 1 -- 172.21.15.43:0/3651300514 learned_addr learned my addr 172.21.15.43:0/3651300514 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 1 --2- 172.21.15.43:0/3651300514 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040ba120 0x7f4204029040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42040bd160 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4204029730 con 0x7f42040bd160 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4204024ad0 con 0x7f420402ad90 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f420402b170 con 0x7f42040ba120 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420b7fe640 1 -- 172.21.15.43:0/3651300514 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3218995621 0 0) 0x7f4204029730 con 0x7f42040bd160 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f42130d2640 10 monclient(hunting): authenticate will time out at 947.308985s 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): get_auth_request con 0x7f420402ad90 auth_method 0 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): get_auth_request con 0x7f42040ba120 auth_method 0 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420b7fe640 10 monclient(hunting): _negotiate 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420b7fe640 10 monclient(hunting): my global_id is 4365 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420b7fe640 1 -- 172.21.15.43:0/3651300514 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41fc003530 con 0x7f42040bd160 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f420bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 1 -- 172.21.15.43:0/3651300514 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040ba120 msgr2=0x7f4204029040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 1 --2- 172.21.15.43:0/3651300514 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040ba120 0x7f4204029040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 1 -- 172.21.15.43:0/3651300514 >> v1:172.21.15.43:6789/0 conn(0x7f42040bd160 legacy=0x7f42040bd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 1 monclient: found mon.noname-b 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.180+0000 7f4210e6e640 1 -- 172.21.15.43:0/3651300514 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42040ba600 con 0x7f420402ad90 2024-04-19T00:44:37.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.184+0000 7f4210e6e640 1 --2- 172.21.15.43:0/3651300514 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f420402ad90 0x7f4204028b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4200009820 tx=0x7f420001c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d054cdb49dca39dc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.184+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f19700c3070 con 0x7f19780a1d70 2024-04-19T00:44:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.184+0000 7f195f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.184+0000 7f195f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 1 -- 172.21.15.43:0/3651300514 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f42000b0070 con 0x7f420402ad90 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: dump: 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.214 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 1 -- 172.21.15.43:0/3651300514 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42000aa810 con 0x7f420402ad90 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 1 -- 172.21.15.43:0/3651300514 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f42000a9300 con 0x7f420402ad90 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f420b7fe640 10 monclient: dump: 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.216 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 5 monclient: authenticate success, global_id 4349 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 4 set_mon_vals no callback set 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 10 monclient: shutdown 2024-04-19T00:44:37.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f420402ad90 msgr2=0x7f4204028b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 --2- 172.21.15.43:0/3651300514 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f420402ad90 0x7f4204028b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4200009820 tx=0x7f420001c370 comp rx=0 tx=0).stop 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 shutdown_connections 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 --2- 172.21.15.43:0/3651300514 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040ba120 0x7f4204029040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 --2- 172.21.15.43:0/3651300514 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f420402ad90 0x7f4204028b00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 >> 172.21.15.43:0/3651300514 conn(0x7f4204008060 msgr2=0x7f4204008470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 shutdown_connections 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 -- 172.21.15.43:0/3651300514 wait complete. 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 10 monclient: monmap: 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 librados: starting msgr at 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 librados: starting objecter 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 Processor -- start 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 -- start start 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 librados: setting wanted keys 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 1 librados: calling monclient init 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.192+0000 7f42130d2640 10 monclient: init 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient: _add_conns ranks=[7,2,5,1,3,6,8,4,0] 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f420402ad90 0x7f4204154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): picked mon.noname-i con 0x7f420402ad90 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 0x7f4204021ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): picked mon.noname-b con 0x7f42040ba120 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:37.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040bd160 0x7f4204022220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): picked mon.noname-h con 0x7f42040bd160 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f42040c7e40 con 0x7f42040ba120 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f42040c7cc0 con 0x7f42040bd160 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f42040c7fc0 con 0x7f420402ad90 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient(hunting): authenticate will time out at 947.324441s 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 0x7f4204021ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f420402ad90 0x7f4204154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 0x7f4204021ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35330/0 (socket says 172.21.15.43:35330) 2024-04-19T00:44:37.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 -- 172.21.15.43:0/2720900135 learned_addr learned my addr 172.21.15.43:0/2720900135 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): get_auth_request con 0x7f42040ba120 auth_method 0 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): get_auth_request con 0x7f420402ad90 auth_method 0 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040bd160 0x7f4204022220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): get_auth_request con 0x7f42040bd160 auth_method 0 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f4210e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f421166f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient(hunting): handle_auth_done global_id 4358 payload 824 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 -- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f420402ad90 msgr2=0x7f4204154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f420402ad90 0x7f4204154f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 -- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040bd160 msgr2=0x7f4204022220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040bd160 0x7f4204022220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 monclient: found mon.noname-b 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 -- 172.21.15.43:0/2720900135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42041555f0 con 0x7f42040ba120 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420bfff640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 0x7f4204021ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f41f800e900 tx=0x7f41f800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 client.?.objecter ms_handle_connect 0x7f42040ba120 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41f8017070 con 0x7f42040ba120 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: dump: 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.224 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f800ce70 con 0x7f42040ba120 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f420affd640 4 set_mon_vals no callback set 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41f80138e0 con 0x7f42040ba120 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42097fa640 10 monclient: dump: 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.226 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 5 monclient: authenticate success, global_id 4358 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient: _renew_subs 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42040b8400 con 0x7f42040ba120 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient: _renew_subs 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4204024d00 con 0x7f42040ba120 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.196+0000 7f42130d2640 1 librados: init done 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f41f801a070 con 0x7f42040ba120 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41f0038da0 0x7f41f003b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f41f800f040 con 0x7f42040ba120 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 10 client.4358.objecter ms_dispatch 0x7f420c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 3 client.4358.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 3 client.4358.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 20 client.4358.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f4210e6e640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41f0038da0 0x7f41f003b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f4210e6e640 10 monclient: get_auth_request con 0x7f41f0038da0 auth_method 0 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f4210e6e640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41f0038da0 0x7f41f003b260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f42000ad000 tx=0x7f42000a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.200+0000 7f42097fa640 10 client.4358.objecter ms_handle_connect 0x7f41f0038da0 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.204+0000 7f42130d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.204+0000 7f42130d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.204+0000 7f42130d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.204+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4204007340 con 0x7f42040ba120 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.212+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f42041555f0 con 0x7f42040ba120 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.212+0000 7f42097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.212+0000 7f42097fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T00:44:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T00:44:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f993c13a0d0 con 0x7f993c0a4ef0 2024-04-19T00:44:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f9932ffd640 1 -- 172.21.15.43:0/1056119682 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f992c024750 con 0x7f993c0a4ef0 2024-04-19T00:44:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f9932ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-19T00:44:37.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f9932ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:37.234 INFO:teuthology.orchestra.run.smithi043.stdout:51539607556 2024-04-19T00:44:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 librados: watch_flush enter 2024-04-19T00:44:37.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 librados: watch_flush exit 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 20 client.4245.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 client.4245.objecter successfully canceled tick 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9920038e10 msgr2=0x7f992003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9920038e10 0x7f992003b2d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9938007fd0 tx=0x7f9938009040 comp rx=0 tx=0).stop 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 10 monclient: shutdown 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 msgr2=0x7f993c12e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 0x7f993c12e030 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f992c00e810 tx=0x7f992c00ecd0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 shutdown_connections 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9920038e10 0x7f992003b2d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f993c13e170 0x7f993c0a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f993c0a4ef0 0x7f993c12e030 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 --2- 172.21.15.43:0/1056119682 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f993c0a4b10 0x7f993c12daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 >> 172.21.15.43:0/1056119682 conn(0x7f993c0a98c0 msgr2=0x7f993c131ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 shutdown_connections 2024-04-19T00:44:37.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 -- 172.21.15.43:0/1056119682 wait complete. 2024-04-19T00:44:37.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.232+0000 7f994468a640 1 librados: shutdown 2024-04-19T00:44:37.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.3 2024-04-19T00:44:37.252 DEBUG:teuthology.parallel:result is None 2024-04-19T00:44:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f0458ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T00:44:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f0458ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T00:44:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f0458ff9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0444005380 con 0x7f046c138a50 2024-04-19T00:44:37.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f045affd640 1 -- 172.21.15.43:0/2217413709 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0464016620 con 0x7f046c138a50 2024-04-19T00:44:37.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f045affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-19T00:44:37.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.312+0000 7f045affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:37.317 INFO:teuthology.orchestra.run.smithi043.stdout:47244640258 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 10 librados: watch_flush enter 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 10 librados: watch_flush exit 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 20 client.4322.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 10 client.4322.objecter successfully canceled tick 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f044c038e10 msgr2=0x7f044c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f044c038e10 0x7f044c03b2d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f045c009a40 tx=0x7f045c009190 comp rx=0 tx=0).stop 2024-04-19T00:44:37.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 10 monclient: shutdown 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 msgr2=0x7f046c13c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 0x7f046c13c840 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f046401c710 tx=0x7f0464019680 comp rx=0 tx=0).stop 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 shutdown_connections 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f044c038e10 0x7f044c03b2d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f046c138a50 0x7f046c13c840 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f046c137960 0x7f046c136a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 --2- 172.21.15.43:0/2217413709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f046c0a5bb0 0x7f046c1d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 >> 172.21.15.43:0/2217413709 conn(0x7f046c0a7cb0 msgr2=0x7f046c06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 shutdown_connections 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 -- 172.21.15.43:0/2217413709 wait complete. 2024-04-19T00:44:37.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.316+0000 7f0458ff9640 1 librados: shutdown 2024-04-19T00:44:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fdde37d9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T00:44:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fdde37d9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T00:44:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fdde37d9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdddc068240 con 0x7fdddc072dd0 2024-04-19T00:44:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fddd1ffb640 1 -- 172.21.15.43:0/4079392547 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fddcc0bc520 con 0x7fdddc072dd0 2024-04-19T00:44:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fddd1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-19T00:44:37.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.320+0000 7fddd1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:37.324 INFO:teuthology.orchestra.run.smithi043.stdout:51539607557 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 10 librados: watch_flush enter 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 10 librados: watch_flush exit 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 20 client.4355.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 10 client.4355.objecter successfully canceled tick 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fddc0038e10 msgr2=0x7fddc003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fddc0038e10 0x7fddc003b2d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fddd8009580 tx=0x7fddd8009190 comp rx=0 tx=0).stop 2024-04-19T00:44:37.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 10 monclient: shutdown 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 msgr2=0x7fdddc1d09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 0x7fdddc1d09f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fddcc0ae040 tx=0x7fddcc0191c0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 shutdown_connections 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdddc0731f0 0x7fdddc1d0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fddc0038e10 0x7fddc003b2d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdddc072dd0 0x7fdddc1d09f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 --2- 172.21.15.43:0/4079392547 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdddc0729f0 0x7fdddc135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 >> 172.21.15.43:0/4079392547 conn(0x7fdddc0a4460 msgr2=0x7fdddc0a5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 shutdown_connections 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 -- 172.21.15.43:0/4079392547 wait complete. 2024-04-19T00:44:37.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.324+0000 7fdde37d9640 1 librados: shutdown 2024-04-19T00:44:37.336 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.1 2024-04-19T00:44:37.336 DEBUG:teuthology.parallel:result is None 2024-04-19T00:44:37.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-04-19T00:44:37.342 DEBUG:teuthology.parallel:result is None 2024-04-19T00:44:37.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f703013f310 con 0x7f7030138a50 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f70177fe640 1 -- 172.21.15.43:0/849879552 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f70280ad840 con 0x7f7030138a50 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f70177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f70177fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:37.410 INFO:teuthology.orchestra.run.smithi043.stdout:47244640260 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 librados: watch_flush enter 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 librados: watch_flush exit 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 20 client.4347.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 client.4347.objecter successfully canceled tick 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f701c038e10 msgr2=0x7f701c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f701c038e10 0x7f701c03b2d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7020009a30 tx=0x7f7020009240 comp rx=0 tx=0).stop 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 10 monclient: shutdown 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 msgr2=0x7f703013ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 0x7f703013ca50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f702801d0a0 tx=0x7f70280195a0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 shutdown_connections 2024-04-19T00:44:37.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f701c038e10 0x7f701c03b2d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7030138a50 0x7f703013ca50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7030137960 0x7f7030136c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 --2- 172.21.15.43:0/849879552 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70300a5bb0 0x7f70301d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 >> 172.21.15.43:0/849879552 conn(0x7f70300a7cb0 msgr2=0x7f703009ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 shutdown_connections 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 -- 172.21.15.43:0/849879552 wait complete. 2024-04-19T00:44:37.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.408+0000 7f703517c640 1 librados: shutdown 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f42040288c0 con 0x7f42040ba120 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42097fa640 1 -- 172.21.15.43:0/2720900135 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f41f8012dc0 con 0x7f42040ba120 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42097fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:37.414 INFO:teuthology.orchestra.run.smithi043.stdout:47244640258 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 librados: watch_flush enter 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 librados: watch_flush exit 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 20 client.4358.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 client.4358.objecter successfully canceled tick 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41f0038da0 msgr2=0x7f41f003b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41f0038da0 0x7f41f003b260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f42000ad000 tx=0x7f42000a6000 comp rx=0 tx=0).stop 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 10 monclient: shutdown 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 msgr2=0x7f4204021ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 0x7f4204021ce0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f41f800e900 tx=0x7f41f800edc0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 shutdown_connections 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41f0038da0 0x7f41f003b260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42040bd160 0x7f4204022220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f42040ba120 0x7f4204021ce0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 --2- 172.21.15.43:0/2720900135 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f420402ad90 0x7f4204154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 >> 172.21.15.43:0/2720900135 conn(0x7f4204008060 msgr2=0x7f42040277d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 shutdown_connections 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 -- 172.21.15.43:0/2720900135 wait complete. 2024-04-19T00:44:37.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.412+0000 7f42130d2640 1 librados: shutdown 2024-04-19T00:44:37.430 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-04-19T00:44:37.430 DEBUG:teuthology.parallel:result is None 2024-04-19T00:44:37.432 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640258 got 47244640258 for osd.0 2024-04-19T00:44:37.433 DEBUG:teuthology.parallel:result is None 2024-04-19T00:44:37.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T00:44:37.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T00:44:37.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1978074ed0 con 0x7f19780a1d70 2024-04-19T00:44:37.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f195f7fe640 1 -- 172.21.15.43:0/2624236748 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f19780a4ee0 con 0x7f19780a1d70 2024-04-19T00:44:37.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f195f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-19T00:44:37.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f195f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:37.439 INFO:teuthology.orchestra.run.smithi043.stdout:51539607557 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 librados: watch_flush enter 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 librados: watch_flush exit 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 20 client.4373.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 client.4373.objecter successfully canceled tick 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1964038e10 msgr2=0x7f196403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1964038e10 0x7f196403b2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1968009ae0 tx=0x7f1968009190 comp rx=0 tx=0).stop 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 10 monclient: shutdown 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f19780a1d70 msgr2=0x7f197813f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f19780a1d70 0x7f197813f800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f19780a5370 tx=0x7f19700192f0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 shutdown_connections 2024-04-19T00:44:37.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1964038e10 0x7f196403b2d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f19780a1d70 0x7f197813f800 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f19780a02b0 0x7f197813f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 --2- 172.21.15.43:0/2624236748 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f197809fe80 0x7f19781d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 >> 172.21.15.43:0/2624236748 conn(0x7f19780a8000 msgr2=0x7f19781376e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.436+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 shutdown_connections 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.440+0000 7f197d4e9640 1 -- 172.21.15.43:0/2624236748 wait complete. 2024-04-19T00:44:37.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.440+0000 7f197d4e9640 1 librados: shutdown 2024-04-19T00:44:37.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-04-19T00:44:37.455 DEBUG:teuthology.parallel:result is None 2024-04-19T00:44:37.456 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T00:44:37.456 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T00:44:37.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.612+0000 7f24216ed640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.612+0000 7f24216ed640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.612+0000 7f24216ed640 10 monclient: monmap: 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.617522+0000 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.617522+0000 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.617 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.612+0000 7f24216ed640 1 Processor -- start 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.612+0000 7f24216ed640 1 -- start start 2024-04-19T00:44:37.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.612+0000 7f24216ed640 10 monclient: init 2024-04-19T00:44:37.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient: _add_conns ranks=[3,8,1,4,2,9,10,5,0,6,7] 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f241c138a50 0x7f241c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): picked mon.noname-g con 0x7f241c138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 0x7f241c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): picked mon.noname-e con 0x7f241c0a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f241c137960 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f241c0a0d40 con 0x7f241c137960 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f241c0a1020 con 0x7f241c138a50 2024-04-19T00:44:37.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f241c1398b0 con 0x7f241c0a5bb0 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient(hunting): authenticate will time out at 947.744545s 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 0x7f241c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 0x7f241c139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59568/0 (socket says 172.21.15.43:59568) 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 -- 172.21.15.43:0/3179475595 learned_addr learned my addr 172.21.15.43:0/3179475595 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): get_auth_request con 0x7f241c0a5bb0 auth_method 0 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241affd640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f241c138a50 0x7f241c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient(hunting): handle_auth_done global_id 4382 payload 274 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 -- 172.21.15.43:0/3179475595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f241c138a50 msgr2=0x7f241c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f241c138a50 0x7f241c138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 -- 172.21.15.43:0/3179475595 >> v1:172.21.15.161:6789/0 conn(0x7f241c137960 legacy=0x7f241c137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 monclient: found mon.noname-e 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 -- 172.21.15.43:0/3179475595 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c0a0a60 con 0x7f241c0a5bb0 2024-04-19T00:44:37.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241affd640 10 monclient: get_auth_request con 0x7f241c138a50 auth_method 0 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241affd640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f241c138a50 0x7f241c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f241a7fc640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 0x7f241c139370 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24100098a0 tx=0x7f241001c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=58c95297af77099c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 1 -- 172.21.15.43:0/3179475595 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24100b0070 con 0x7f241c0a5bb0 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: dump: 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.632 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 1 -- 172.21.15.43:0/3179475595 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24100aa810 con 0x7f241c0a5bb0 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 1 -- 172.21.15.43:0/3179475595 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24100a9300 con 0x7f241c0a5bb0 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 5 monclient: authenticate success, global_id 4382 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 4 set_mon_vals no callback set 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f2419ffb640 10 monclient: dump: 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.634 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.635 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 10 monclient: shutdown 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- 172.21.15.43:0/3179475595 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 msgr2=0x7f241c139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 0x7f241c139370 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f24100098a0 tx=0x7f241001c440 comp rx=0 tx=0).stop 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- 172.21.15.43:0/3179475595 shutdown_connections 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c0a5bb0 0x7f241c139370 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 --2- 172.21.15.43:0/3179475595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f241c138a50 0x7f241c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- 172.21.15.43:0/3179475595 >> 172.21.15.43:0/3179475595 conn(0x7f241c0a7cb0 msgr2=0x7f241c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- 172.21.15.43:0/3179475595 shutdown_connections 2024-04-19T00:44:37.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.616+0000 7f24216ed640 1 -- 172.21.15.43:0/3179475595 wait complete. 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: monmap: 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.637 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 librados: starting msgr at 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 librados: starting objecter 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 Processor -- start 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- start start 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 librados: setting wanted keys 2024-04-19T00:44:37.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 librados: calling monclient init 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: init 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _add_conns ranks=[1,8,4,7,5,0,2,3,6] 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f241c0a5bb0 0x7f241c0a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): picked mon.noname-f con 0x7f241c0a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 0x7f241c0a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): picked mon.noname-e con 0x7f241c137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f241c138a50 0x7f241c0a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): picked mon.noname-c con 0x7f241c138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f241c143bb0 con 0x7f241c0a5bb0 2024-04-19T00:44:37.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f241c143a30 con 0x7f241c138a50 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f241c143d30 con 0x7f241c137960 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient(hunting): authenticate will time out at 947.749273s 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 0x7f241c0a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 0x7f241c0a2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59578/0 (socket says 172.21.15.43:59578) 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 -- 172.21.15.43:0/127907211 learned_addr learned my addr 172.21.15.43:0/127907211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): get_auth_request con 0x7f241c137960 auth_method 0 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f241c138a50 0x7f241c0a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): get_auth_request con 0x7f241c138a50 auth_method 0 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241affd640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f241c0a5bb0 0x7f241c0a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2024-04-19T00:44:37.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 -- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f241c138a50 msgr2=0x7f241c0a3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f241c138a50 0x7f241c0a3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 -- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f241c0a5bb0 msgr2=0x7f241c0a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f241c0a5bb0 0x7f241c0a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 monclient: found mon.noname-e 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 -- 172.21.15.43:0/127907211 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241c0a4ca0 con 0x7f241c137960 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241affd640 10 monclient: get_auth_request con 0x7f241c0a5bb0 auth_method 0 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241affd640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f241c0a5bb0 0x7f241c0a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:37.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241a7fc640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 0x7f241c0a2af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f24100097c0 tx=0x7f24100191e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 client.?.objecter ms_handle_connect 0x7f241c137960 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24100b9070 con 0x7f241c137960 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: dump: 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.643 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24100044b0 con 0x7f241c137960 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24100b0040 con 0x7f241c137960 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 monclient: dump: 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.645 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24197fa640 4 set_mon_vals no callback set 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 5 monclient: authenticate success, global_id 4391 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _renew_subs 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241c0a4160 con 0x7f241c137960 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _renew_subs 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f241c09ee60 con 0x7f241c137960 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 librados: init done 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2410004040 con 0x7f241c137960 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f241c0008e0 con 0x7f241c137960 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2408038e10 0x7f240803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f24100e1e80 con 0x7f241c137960 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 10 client.4391.objecter ms_dispatch 0x7f241c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 3 client.4391.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 3 client.4391.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f24037fe640 20 client.4391.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241affd640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2408038e10 0x7f240803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.620+0000 7f241affd640 10 monclient: get_auth_request con 0x7f2408038e10 auth_method 0 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.624+0000 7f24037fe640 10 client.4391.objecter ms_handle_connect 0x7f2408038e10 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.624+0000 7f241affd640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2408038e10 0x7f240803b2d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f240c008890 tx=0x7f240c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.628+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2410016150 con 0x7f241c137960 2024-04-19T00:44:37.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.628+0000 7f24037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.628+0000 7f24037fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:37.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.784+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f241c09e3e0 con 0x7f2408038e10 2024-04-19T00:44:37.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.788+0000 7f24037fe640 1 -- 172.21.15.43:0/127907211 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33771 (secure 0 0 0) 0x7f241c09e3e0 con 0x7f2408038e10 2024-04-19T00:44:37.791 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:37.791 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T00:44:37.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 10 librados: watch_flush enter 2024-04-19T00:44:37.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 10 librados: watch_flush exit 2024-04-19T00:44:37.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 20 client.4391.objecter shutdown clearing up homeless session... 2024-04-19T00:44:37.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 10 client.4391.objecter successfully canceled tick 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2408038e10 msgr2=0x7f240803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2408038e10 0x7f240803b2d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f240c008890 tx=0x7f240c008040 comp rx=0 tx=0).stop 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 10 monclient: shutdown 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 msgr2=0x7f241c0a2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 0x7f241c0a2af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f24100097c0 tx=0x7f24100191e0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 shutdown_connections 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f241c138a50 0x7f241c0a3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2408038e10 0x7f240803b2d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f241c137960 0x7f241c0a2af0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 --2- 172.21.15.43:0/127907211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f241c0a5bb0 0x7f241c0a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 >> 172.21.15.43:0/127907211 conn(0x7f241c0a7cb0 msgr2=0x7f241c0a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 shutdown_connections 2024-04-19T00:44:37.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 -- 172.21.15.43:0/127907211 wait complete. 2024-04-19T00:44:37.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.792+0000 7f24216ed640 1 librados: shutdown 2024-04-19T00:44:37.809 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-19T00:44:36.879101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168596,"kb_used_data":1688,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062444,"statfs":{"total":579820584960,"available":579647942656,"internally_reserved":0,"allocated":1728512,"data_stored":1156658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063505+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T00:44:35.063505+0000","last_peered":"2024-04-19T00:44:35.063505+0000","last_clean":"2024-04-19T00:44:35.063505+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T00:44:35.063505+0000","last_undegraded":"2024-04-19T00:44:35.063505+0000","last_fullsized":"2024-04-19T00:44:35.063505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.475432+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T00:44:34.475432+0000","last_peered":"2024-04-19T00:44:34.475432+0000","last_clean":"2024-04-19T00:44:34.475432+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T00:44:34.475432+0000","last_undegraded":"2024-04-19T00:44:34.475432+0000","last_fullsized":"2024-04-19T00:44:34.475432+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:58:29.642043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.068610+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T00:44:34.068610+0000","last_peered":"2024-04-19T00:44:34.068610+0000","last_clean":"2024-04-19T00:44:34.068610+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T00:44:34.068610+0000","last_undegraded":"2024-04-19T00:44:34.068610+0000","last_fullsized":"2024-04-19T00:44:34.068610+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:46:49.340444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.062032+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T00:44:35.062032+0000","last_peered":"2024-04-19T00:44:35.062032+0000","last_clean":"2024-04-19T00:44:35.062032+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T00:44:35.062032+0000","last_undegraded":"2024-04-19T00:44:35.062032+0000","last_fullsized":"2024-04-19T00:44:35.062032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":159,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.994228+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T00:44:34.994228+0000","last_peered":"2024-04-19T00:44:34.994228+0000","last_clean":"2024-04-19T00:44:34.994228+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T00:44:34.994228+0000","last_undegraded":"2024-04-19T00:44:34.994228+0000","last_fullsized":"2024-04-19T00:44:34.994228+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:31:50.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.061815+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T00:44:35.061815+0000","last_peered":"2024-04-19T00:44:35.061815+0000","last_clean":"2024-04-19T00:44:35.061815+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T00:44:35.061815+0000","last_undegraded":"2024-04-19T00:44:35.061815+0000","last_fullsized":"2024-04-19T00:44:35.061815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.477000+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T00:44:34.477000+0000","last_peered":"2024-04-19T00:44:34.477000+0000","last_clean":"2024-04-19T00:44:34.477000+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T00:44:34.477000+0000","last_undegraded":"2024-04-19T00:44:34.477000+0000","last_fullsized":"2024-04-19T00:44:34.477000+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:11:08.908621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063480+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T00:44:35.063480+0000","last_peered":"2024-04-19T00:44:35.063480+0000","last_clean":"2024-04-19T00:44:35.063480+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T00:44:35.063480+0000","last_undegraded":"2024-04-19T00:44:35.063480+0000","last_fullsized":"2024-04-19T00:44:35.063480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.478056+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T00:44:34.478056+0000","last_peered":"2024-04-19T00:44:34.478056+0000","last_clean":"2024-04-19T00:44:34.478056+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T00:44:34.478056+0000","last_undegraded":"2024-04-19T00:44:34.478056+0000","last_fullsized":"2024-04-19T00:44:34.478056+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:15:48.438176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":602112,"data_stored":504435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":602112,"data_stored":504435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T00:44:37.810 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.916+0000 7fd695050640 10 monclient: get_monmap_and_config 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.916+0000 7fd695050640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.916+0000 7fd695050640 10 monclient: monmap: 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:37.921621+0000 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:37.921621+0000 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:37.920 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:37.921 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 Processor -- start 2024-04-19T00:44:37.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 -- start start 2024-04-19T00:44:37.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient: init 2024-04-19T00:44:37.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient: _add_conns ranks=[3,7,8,0,9,6,1,5,2,4,10] 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6900a4f50 0x7fd6900a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): picked mon.noname-g con 0x7fd6900a4f50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6900731a0 0x7fd6900a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): picked mon.noname-i con 0x7fd6900731a0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 0x7fd6900a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): picked mon.noname-e con 0x7fd6900a3370 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd6900a5330 con 0x7fd6900a4f50 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd690070660 con 0x7fd6900731a0 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd69009bc10 con 0x7fd6900a3370 2024-04-19T00:44:37.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient(hunting): authenticate will time out at 948.048952s 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 0x7fd6900a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 0x7fd6900a37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59582/0 (socket says 172.21.15.43:59582) 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 -- 172.21.15.43:0/4280844926 learned_addr learned my addr 172.21.15.43:0/4280844926 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): get_auth_request con 0x7fd6900a3370 auth_method 0 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68e59c640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6900731a0 0x7fd6900a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68ed9d640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6900a4f50 0x7fd6900a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68e59c640 10 monclient(hunting): get_auth_request con 0x7fd6900731a0 auth_method 0 2024-04-19T00:44:37.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68ed9d640 10 monclient(hunting): get_auth_request con 0x7fd6900a4f50 auth_method 0 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 -- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6900731a0 msgr2=0x7fd6900a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6900731a0 0x7fd6900a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 -- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6900a4f50 msgr2=0x7fd6900a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6900a4f50 0x7fd6900a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 monclient: found mon.noname-e 2024-04-19T00:44:37.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 -- 172.21.15.43:0/4280844926 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd690133900 con 0x7fd6900a3370 2024-04-19T00:44:37.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68f59e640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 0x7fd6900a37f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6800098d0 tx=0x7fd68001c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=41216e9b6f449335 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 1 -- 172.21.15.43:0/4280844926 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6800b0070 con 0x7fd6900a3370 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: dump: 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 1 -- 172.21.15.43:0/4280844926 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6800aa810 con 0x7fd6900a3370 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 5 monclient: authenticate success, global_id 4400 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 1 -- 172.21.15.43:0/4280844926 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd6800a9300 con 0x7fd6900a3370 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd695050640 4 set_mon_vals no callback set 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.920+0000 7fd68dd9b640 10 monclient: dump: 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.935 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: shutdown 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/4280844926 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 msgr2=0x7fd6900a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 0x7fd6900a37f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6800098d0 tx=0x7fd68001c440 comp rx=0 tx=0).stop 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/4280844926 shutdown_connections 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900a3370 0x7fd6900a37f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd6900731a0 0x7fd6900a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- 172.21.15.43:0/4280844926 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6900a4f50 0x7fd6900a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/4280844926 >> 172.21.15.43:0/4280844926 conn(0x7fd6900a4460 msgr2=0x7fd6900a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/4280844926 shutdown_connections 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/4280844926 wait complete. 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: build_initial_monmap 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: monmap: 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:37.937 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 librados: starting msgr at 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 librados: starting objecter 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 Processor -- start 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- start start 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 librados: setting wanted keys 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 librados: calling monclient init 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: init 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: _add_conns ranks=[8,5,0,1,6,3,7,4,2] 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 0x7fd6901d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): picked mon.noname-e con 0x7fd6900731a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6900a3370 0x7fd690130140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): picked mon.noname-h con 0x7fd6900a3370 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6900a4f50 0x7fd69013b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): picked mon.noname-a con 0x7fd6900a4f50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd690140700 con 0x7fd6900a4f50 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fd690140580 con 0x7fd6900a3370 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd690140880 con 0x7fd6900731a0 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient(hunting): authenticate will time out at 948.054548s 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 0x7fd6901d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 0x7fd6901d09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59590/0 (socket says 172.21.15.43:59590) 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6900a4f50 0x7fd69013b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 -- 172.21.15.43:0/3194420629 learned_addr learned my addr 172.21.15.43:0/3194420629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): get_auth_request con 0x7fd6900731a0 auth_method 0 2024-04-19T00:44:37.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): get_auth_request con 0x7fd6900a4f50 auth_method 0 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68e59c640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6900a3370 0x7fd690130140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 -- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6900a3370 msgr2=0x7fd690130140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6900a3370 0x7fd690130140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 -- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6900a4f50 msgr2=0x7fd69013b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6900a4f50 0x7fd69013b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 monclient: found mon.noname-e 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 -- 172.21.15.43:0/3194420629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69006f9c0 con 0x7fd6900731a0 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68e59c640 10 monclient: get_auth_request con 0x7fd6900a3370 auth_method 0 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68e59c640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6900a3370 0x7fd690130140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68f59e640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6900a4f50 0x7fd69013b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:37.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68ed9d640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 0x7fd6901d09a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd68800d860 tx=0x7fd68800dd20 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 client.?.objecter ms_handle_connect 0x7fd6900731a0 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd68800de50 con 0x7fd6900731a0 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: dump: 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.943 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd688019070 con 0x7fd6900731a0 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd68800bd40 con 0x7fd6900731a0 2024-04-19T00:44:37.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 5 monclient: authenticate success, global_id 4409 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: _renew_subs 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd690071f90 con 0x7fd6900731a0 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd68cd99640 4 set_mon_vals no callback set 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd6877fe640 10 monclient: dump: 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:37.945 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: _renew_subs 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd690072830 con 0x7fd6900731a0 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.924+0000 7fd695050640 1 librados: init done 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd68800b8d0 con 0x7fd6900731a0 2024-04-19T00:44:37.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd695050640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd695050640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd695050640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd69006fe80 con 0x7fd6900731a0 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd670038e40 0x7fd67003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fd68800f040 con 0x7fd6900731a0 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 10 client.4409.objecter ms_dispatch 0x7fd690000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 3 client.4409.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 3 client.4409.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 20 client.4409.objecter dump_active .. 0 homeless 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd68e59c640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd670038e40 0x7fd67003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd68e59c640 10 monclient: get_auth_request con 0x7fd670038e40 auth_method 0 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd6877fe640 10 client.4409.objecter ms_handle_connect 0x7fd670038e40 2024-04-19T00:44:37.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.928+0000 7fd68e59c640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd670038e40 0x7fd67003b300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd6900a0c90 tx=0x7fd67800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:37.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.932+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd688022020 con 0x7fd6900731a0 2024-04-19T00:44:37.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.932+0000 7fd6877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:37.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:37.932+0000 7fd6877fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:38.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.088+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd690070de0 con 0x7fd670038e40 2024-04-19T00:44:38.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.092+0000 7fd6877fe640 1 -- 172.21.15.43:0/3194420629 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33771 (secure 0 0 0) 0x7fd690070de0 con 0x7fd670038e40 2024-04-19T00:44:38.095 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:38.096 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T00:44:38.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 10 librados: watch_flush enter 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 10 librados: watch_flush exit 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 20 client.4409.objecter shutdown clearing up homeless session... 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 10 client.4409.objecter successfully canceled tick 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd670038e40 msgr2=0x7fd67003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd670038e40 0x7fd67003b300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd6900a0c90 tx=0x7fd67800b040 comp rx=0 tx=0).stop 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 10 monclient: shutdown 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 msgr2=0x7fd6901d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 0x7fd6901d09a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd68800d860 tx=0x7fd68800dd20 comp rx=0 tx=0).stop 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 shutdown_connections 2024-04-19T00:44:38.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6900a4f50 0x7fd69013b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6900a3370 0x7fd690130140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd670038e40 0x7fd67003b300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 --2- 172.21.15.43:0/3194420629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd6900731a0 0x7fd6901d09a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 >> 172.21.15.43:0/3194420629 conn(0x7fd6900a4460 msgr2=0x7fd690131300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 shutdown_connections 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 -- 172.21.15.43:0/3194420629 wait complete. 2024-04-19T00:44:38.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.096+0000 7fd695050640 1 librados: shutdown 2024-04-19T00:44:38.113 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-19T00:44:36.879101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":168596,"kb_used_data":1688,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566062444,"statfs":{"total":579820584960,"available":579647942656,"internally_reserved":0,"allocated":1728512,"data_stored":1156658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063505+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T00:44:35.063505+0000","last_peered":"2024-04-19T00:44:35.063505+0000","last_clean":"2024-04-19T00:44:35.063505+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T00:44:35.063505+0000","last_undegraded":"2024-04-19T00:44:35.063505+0000","last_fullsized":"2024-04-19T00:44:35.063505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.475432+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T00:44:34.475432+0000","last_peered":"2024-04-19T00:44:34.475432+0000","last_clean":"2024-04-19T00:44:34.475432+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T00:44:34.475432+0000","last_undegraded":"2024-04-19T00:44:34.475432+0000","last_fullsized":"2024-04-19T00:44:34.475432+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:58:29.642043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.068610+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T00:44:34.068610+0000","last_peered":"2024-04-19T00:44:34.068610+0000","last_clean":"2024-04-19T00:44:34.068610+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T00:44:34.068610+0000","last_undegraded":"2024-04-19T00:44:34.068610+0000","last_fullsized":"2024-04-19T00:44:34.068610+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:46:49.340444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.062032+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T00:44:35.062032+0000","last_peered":"2024-04-19T00:44:35.062032+0000","last_clean":"2024-04-19T00:44:35.062032+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T00:44:35.062032+0000","last_undegraded":"2024-04-19T00:44:35.062032+0000","last_fullsized":"2024-04-19T00:44:35.062032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":159,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.994228+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T00:44:34.994228+0000","last_peered":"2024-04-19T00:44:34.994228+0000","last_clean":"2024-04-19T00:44:34.994228+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T00:44:34.994228+0000","last_undegraded":"2024-04-19T00:44:34.994228+0000","last_fullsized":"2024-04-19T00:44:34.994228+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:31:50.307421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.061815+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T00:44:35.061815+0000","last_peered":"2024-04-19T00:44:35.061815+0000","last_clean":"2024-04-19T00:44:35.061815+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T00:44:35.061815+0000","last_undegraded":"2024-04-19T00:44:35.061815+0000","last_fullsized":"2024-04-19T00:44:35.061815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.477000+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T00:44:34.477000+0000","last_peered":"2024-04-19T00:44:34.477000+0000","last_clean":"2024-04-19T00:44:34.477000+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T00:44:34.477000+0000","last_undegraded":"2024-04-19T00:44:34.477000+0000","last_fullsized":"2024-04-19T00:44:34.477000+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:11:08.908621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063480+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T00:44:35.063480+0000","last_peered":"2024-04-19T00:44:35.063480+0000","last_clean":"2024-04-19T00:44:35.063480+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T00:44:35.063480+0000","last_undegraded":"2024-04-19T00:44:35.063480+0000","last_fullsized":"2024-04-19T00:44:35.063480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-19T00:44:34.478056+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T00:44:34.478056+0000","last_peered":"2024-04-19T00:44:34.478056+0000","last_clean":"2024-04-19T00:44:34.478056+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T00:44:34.478056+0000","last_undegraded":"2024-04-19T00:44:34.478056+0000","last_fullsized":"2024-04-19T00:44:34.478056+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:15:48.438176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607556,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":602112,"data_stored":504435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31308,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340532,"statfs":{"total":96636764160,"available":96604704768,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27920,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343920,"statfs":{"total":96636764160,"available":96608174080,"internally_reserved":0,"allocated":602112,"data_stored":504435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640258,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":36947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T00:44:38.113 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T00:44:38.113 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-19T00:44:38.114 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-19T00:44:38.114 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-19T00:44:38.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient: get_monmap_and_config 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient: build_initial_monmap 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient: monmap: 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:38.232214+0000 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:38.232214+0000 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:38.231 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:38.232 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 Processor -- start 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 -- start start 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient: init 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient: _add_conns ranks=[8,5,7,3,1,6,2,9,0,4,10] 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 0x7f40f0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): picked mon.noname-e con 0x7f40f0138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40f00a5bb0 0x7f40f0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): picked mon.noname-h con 0x7f40f00a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f0137960 0x7f40f0137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): picked mon.noname-i con 0x7f40f0137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f40f00a1020 con 0x7f40f00a5bb0 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f40f01398b0 con 0x7f40f0137960 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f40f00a0780 con 0x7f40f0138a50 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f72dd640 10 monclient(hunting): authenticate will time out at 948.359492s 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 0x7f40f0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 0x7f40f0138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59596/0 (socket says 172.21.15.43:59596) 2024-04-19T00:44:38.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 1 -- 172.21.15.43:0/676183218 learned_addr learned my addr 172.21.15.43:0/676183218 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): get_auth_request con 0x7f40f0138a50 auth_method 0 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f5079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f587a640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f0137960 0x7f40f0137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f4878640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40f00a5bb0 0x7f40f0139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f587a640 10 monclient(hunting): get_auth_request con 0x7f40f0137960 auth_method 0 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f587a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f587a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f587a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f4878640 10 monclient(hunting): get_auth_request con 0x7f40f00a5bb0 auth_method 0 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f4878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f4878640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.228+0000 7f40f4878640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 -- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f0137960 msgr2=0x7f40f0137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f0137960 0x7f40f0137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 -- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40f00a5bb0 msgr2=0x7f40f0139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40f00a5bb0 0x7f40f0139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 monclient: found mon.noname-e 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:38.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 -- 172.21.15.43:0/676183218 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f00a0d40 con 0x7f40f0138a50 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f587a640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f0137960 0x7f40f0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f5079640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 0x7f40f0138e30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009e80 tx=0x7f40ec01c8e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=766cba6bded4e8f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 1 -- 172.21.15.43:0/676183218 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40ec0090d0 con 0x7f40f0138a50 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: dump: 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.241 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 1 -- 172.21.15.43:0/676183218 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ec0a5b50 con 0x7f40f0138a50 2024-04-19T00:44:38.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 5 monclient: authenticate success, global_id 4418 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 1 -- 172.21.15.43:0/676183218 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40ec0a5d40 con 0x7f40f0138a50 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 4 set_mon_vals no callback set 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40e7fff640 10 monclient: dump: 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.243 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 10 monclient: shutdown 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:38.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 -- 172.21.15.43:0/676183218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 msgr2=0x7f40f0138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 0x7f40f0138e30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009e80 tx=0x7f40ec01c8e0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 -- 172.21.15.43:0/676183218 shutdown_connections 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f0137960 0x7f40f0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40f00a5bb0 0x7f40f0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 --2- 172.21.15.43:0/676183218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40f0138a50 0x7f40f0138e30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 -- 172.21.15.43:0/676183218 >> 172.21.15.43:0/676183218 conn(0x7f40f00a7cb0 msgr2=0x7f40f00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 -- 172.21.15.43:0/676183218 shutdown_connections 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.232+0000 7f40f72dd640 1 -- 172.21.15.43:0/676183218 wait complete. 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: build_initial_monmap 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: monmap: 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:38.245 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 librados: starting msgr at 2024-04-19T00:44:38.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 librados: starting objecter 2024-04-19T00:44:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 Processor -- start 2024-04-19T00:44:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- start start 2024-04-19T00:44:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 librados: setting wanted keys 2024-04-19T00:44:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 librados: calling monclient init 2024-04-19T00:44:38.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: init 2024-04-19T00:44:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _add_conns ranks=[7,3,6,1,2,4,8,5,0] 2024-04-19T00:44:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f00a5bb0 0x7f40f0135460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): picked mon.noname-i con 0x7f40f00a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40f0137960 0x7f40f01359a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): picked mon.noname-g con 0x7f40f0137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 0x7f40f0135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): picked mon.noname-d con 0x7f40f0138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f40f00a2100 con 0x7f40f0137960 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f40f00a1f80 con 0x7f40f0138a50 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f40f0142810 con 0x7f40f00a5bb0 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient(hunting): authenticate will time out at 948.365140s 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 0x7f40f0135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 0x7f40f0135ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42180/0 (socket says 172.21.15.43:42180) 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 -- 172.21.15.43:0/1693406290 learned_addr learned my addr 172.21.15.43:0/1693406290 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:38.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): get_auth_request con 0x7f40f0138a50 auth_method 0 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f4878640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40f0137960 0x7f40f01359a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f00a5bb0 0x7f40f0135460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f4878640 10 monclient(hunting): get_auth_request con 0x7f40f0137960 auth_method 0 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f4878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f4878640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f4878640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 10 monclient(hunting): get_auth_request con 0x7f40f00a5bb0 auth_method 0 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 -- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f00a5bb0 msgr2=0x7f40f0135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f00a5bb0 0x7f40f0135460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 -- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40f0137960 msgr2=0x7f40f01359a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40f0137960 0x7f40f01359a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 monclient: found mon.noname-d 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 -- 172.21.15.43:0/1693406290 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f009fc90 con 0x7f40f0138a50 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f587a640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 0x7f40f0135ee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e7e0 tx=0x7f40e800eca0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 client.?.objecter ms_handle_connect 0x7f40f0138a50 2024-04-19T00:44:38.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40e8017070 con 0x7f40f0138a50 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: dump: 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.255 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e800edd0 con 0x7f40f0138a50 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40e800f040 con 0x7f40f0138a50 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 5 monclient: authenticate success, global_id 4254 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _renew_subs 2024-04-19T00:44:38.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f00a2ef0 con 0x7f40f0138a50 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e77fe640 4 set_mon_vals no callback set 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 monclient: dump: 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.257 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _renew_subs 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f00a2c10 con 0x7f40f0138a50 2024-04-19T00:44:38.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 librados: init done 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f01366b0 con 0x7f40f0138a50 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f40e801a070 con 0x7f40f0138a50 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40d4038e10 0x7f40d403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f40e80487a0 con 0x7f40f0138a50 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 10 client.4254.objecter ms_dispatch 0x7f40f0000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 3 client.4254.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 3 client.4254.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40e5ffb640 20 client.4254.objecter dump_active .. 0 homeless 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40d4038e10 0x7f40d403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.236+0000 7f40f5079640 10 monclient: get_auth_request con 0x7f40d4038e10 auth_method 0 2024-04-19T00:44:38.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.240+0000 7f40e5ffb640 10 client.4254.objecter ms_handle_connect 0x7f40d4038e10 2024-04-19T00:44:38.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.240+0000 7f40f5079640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40d4038e10 0x7f40d403b2d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0ae000 tx=0x7f40ec0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.244+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f40f009fc90 con 0x7f40f0138a50 2024-04-19T00:44:38.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.244+0000 7f40e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.244+0000 7f40e5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:38.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-19T00:44:38.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-19T00:44:38.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:38.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f40f0076190 con 0x7f40f0138a50 2024-04-19T00:44:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40e5ffb640 1 -- 172.21.15.43:0/1693406290 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f40e800cc40 con 0x7f40f0138a50 2024-04-19T00:44:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-19T00:44:38.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40e5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:38.446 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:38.446 INFO:teuthology.orchestra.run.smithi043.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-19T00:44:38.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 librados: watch_flush enter 2024-04-19T00:44:38.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 librados: watch_flush exit 2024-04-19T00:44:38.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 20 client.4254.objecter shutdown clearing up homeless session... 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 client.4254.objecter successfully canceled tick 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40d4038e10 msgr2=0x7f40d403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40d4038e10 0x7f40d403b2d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0ae000 tx=0x7f40ec0a7070 comp rx=0 tx=0).stop 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 10 monclient: shutdown 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 msgr2=0x7f40f0135ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 0x7f40f0135ee0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e7e0 tx=0x7f40e800eca0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 shutdown_connections 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40d4038e10 0x7f40d403b2d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f40f0138a50 0x7f40f0135ee0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f40f0137960 0x7f40f01359a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 --2- 172.21.15.43:0/1693406290 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f40f00a5bb0 0x7f40f0135460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 >> 172.21.15.43:0/1693406290 conn(0x7f40f00a7cb0 msgr2=0x7f40f006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:38.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 shutdown_connections 2024-04-19T00:44:38.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 -- 172.21.15.43:0/1693406290 wait complete. 2024-04-19T00:44:38.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.444+0000 7f40f72dd640 1 librados: shutdown 2024-04-19T00:44:38.465 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-19T00:44:38.465 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-19T00:44:38.478 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-19T00:44:38.478 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T00:44:38.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.596+0000 7fec37a31640 10 monclient: get_monmap_and_config 2024-04-19T00:44:38.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.596+0000 7fec37a31640 10 monclient: build_initial_monmap 2024-04-19T00:44:38.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.596+0000 7fec37a31640 10 monclient: monmap: 2024-04-19T00:44:38.600 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:38.600 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.600 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:38.601838+0000 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:38.601838+0000 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:38.601 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:38.602 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:38.602 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 1 Processor -- start 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 1 -- start start 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient: init 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient: _add_conns ranks=[1,3,0,2,8,7,5,4,6,9,10] 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fec30138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec30139540 0x7fec3013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient(hunting): picked mon.noname-g con 0x7fec30139540 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:38.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec300a5bb0 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec300a0d40 con 0x7fec300a5bb0 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec300a0a60 con 0x7fec30138a50 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fec300a1020 con 0x7fec30139540 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 10 monclient(hunting): authenticate will time out at 948.729027s 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fec300a5bb0 0x7fec30140530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44788/0 (socket says 172.21.15.43:44788) 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 1 -- 172.21.15.43:0/2050688608 learned_addr learned my addr 172.21.15.43:0/2050688608 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec30139540 0x7fec3013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 60643815 0 0) 0x7fec300a0d40 con 0x7fec300a5bb0 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): _negotiate 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): my global_id is 4383 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec1c003530 con 0x7fec300a5bb0 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): get_auth_request con 0x7fec30139540 auth_method 0 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3483894880 0 0) 0x7fec1c003530 con 0x7fec300a5bb0 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec30139540 msgr2=0x7fec3013dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec30139540 0x7fec3013dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 >> v1:172.21.15.161:6789/0 conn(0x7fec30138a50 legacy=0x7fec30138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec300a17b0 con 0x7fec300a5bb0 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 5 monclient: authenticate success, global_id 4383 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fec28001cf0 con 0x7fec300a5bb0 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: dump: 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.614 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:44:38.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 >> v1:172.21.15.43:6789/0 conn(0x7fec300a5bb0 legacy=0x7fec30140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient: _add_conns ranks=[0,4,7,6,2,8,5,3,1] 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec1c0041f0 0x7fec1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): picked mon.a con 0x7fec1c0041f0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec1c004d50 0x7fec1c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): picked mon.c con 0x7fec1c004d50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec1c009bf0 0x7fec1c00c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): picked mon.i con 0x7fec1c009bf0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fec300a1020 con 0x7fec1c0041f0 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fec1c00c940 con 0x7fec1c004d50 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fec1c00caf0 con 0x7fec1c009bf0 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec27fff640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec37a31640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec1c0041f0 0x7fec1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec1c004d50 0x7fec1c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): get_auth_request con 0x7fec1c0041f0 auth_method 0 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): get_auth_request con 0x7fec1c004d50 auth_method 0 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:38.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec1c009bf0 0x7fec1c00c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 10 monclient(hunting): get_auth_request con 0x7fec1c009bf0 auth_method 0 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient(hunting): handle_auth_done global_id 4383 payload 293 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 -- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec1c009bf0 msgr2=0x7fec1c00c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec1c009bf0 0x7fec1c00c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 -- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec1c004d50 msgr2=0x7fec1c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec1c004d50 0x7fec1c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 monclient: found mon.a 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 1 -- 172.21.15.43:0/2050688608 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1c003530 con 0x7fec1c0041f0 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec35fce640 1 -- 172.21.15.43:0/2050688608 reap_dead start 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec357cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.600+0000 7fec34fcc640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec1c004d50 0x7fec1c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec357cd640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec1c0041f0 0x7fec1c004810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec2c001790 tx=0x7fec2c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8025ad305285eedd server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec2c0013e0 con 0x7fec1c0041f0 2024-04-19T00:44:38.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: dump: 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.620 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec2c0050d0 con 0x7fec1c0041f0 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 1 -- 172.21.15.43:0/2050688608 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec2c0052c0 con 0x7fec1c0041f0 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 4 set_mon_vals no callback set 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec27fff640 10 monclient: dump: 2024-04-19T00:44:38.621 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.622 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: shutdown 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec1c0041f0 msgr2=0x7fec1c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec1c0041f0 0x7fec1c004810 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec2c001790 tx=0x7fec2c01c410 comp rx=0 tx=0).stop 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- 172.21.15.43:0/2050688608 shutdown_connections 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 --2- 172.21.15.43:0/2050688608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec1c0041f0 0x7fec1c004810 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- 172.21.15.43:0/2050688608 >> 172.21.15.43:0/2050688608 conn(0x7fec300a7cb0 msgr2=0x7fec300a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- 172.21.15.43:0/2050688608 shutdown_connections 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- 172.21.15.43:0/2050688608 wait complete. 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: build_initial_monmap 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: monmap: 2024-04-19T00:44:38.623 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:38.624 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 librados: starting msgr at 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 librados: starting objecter 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 Processor -- start 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- start start 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 librados: setting wanted keys 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 librados: calling monclient init 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: init 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient: _add_conns ranks=[3,2,0,8,7,4,6,1,5] 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec1c0041f0 0x7fec301ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): picked mon.noname-g con 0x7fec1c0041f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 0x7fec30136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): picked mon.noname-b con 0x7fec301cd670 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 0x7fec301d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): picked mon.noname-a con 0x7fec30137210 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:38.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fec30138500 con 0x7fec30137210 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fec30138380 con 0x7fec301cd670 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fec30138680 con 0x7fec1c0041f0 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 10 monclient(hunting): authenticate will time out at 948.735045s 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 0x7fec301d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 0x7fec30136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 0x7fec301d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54252/0 (socket says 172.21.15.43:54252) 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 1 -- 172.21.15.43:0/3942548613 learned_addr learned my addr 172.21.15.43:0/3942548613 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): get_auth_request con 0x7fec30137210 auth_method 0 2024-04-19T00:44:38.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 0x7fec30136cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35342/0 (socket says 172.21.15.43:35342) 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): get_auth_request con 0x7fec301cd670 auth_method 0 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec357cd640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec1c0041f0 0x7fec301ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec357cd640 10 monclient(hunting): get_auth_request con 0x7fec1c0041f0 auth_method 0 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec357cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec357cd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec357cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 -- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec1c0041f0 msgr2=0x7fec301ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec1c0041f0 0x7fec301ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 -- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 msgr2=0x7fec301d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 0x7fec301d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 monclient: found mon.noname-b 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:38.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 -- 172.21.15.43:0/3942548613 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3009e5c0 con 0x7fec301cd670 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec35fce640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 0x7fec301d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec34fcc640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 0x7fec30136cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fec3006e8c0 tx=0x7fec20011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 client.?.objecter ms_handle_connect 0x7fec301cd670 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec20019070 con 0x7fec301cd670 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 monclient: dump: 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.629 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec37a31640 5 monclient: authenticate success, global_id 4367 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.604+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec2000fc30 con 0x7fec301cd670 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 monclient: _renew_subs 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec30137720 con 0x7fec301cd670 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec200096b0 con 0x7fec301cd670 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec277fe640 4 set_mon_vals no callback set 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 10 monclient: dump: 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.631 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 monclient: _renew_subs 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec301240d0 con 0x7fec301cd670 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 1 librados: init done 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec301cdad0 con 0x7fec301cd670 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fec2001c070 con 0x7fec301cd670 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec14038e10 0x7fec1403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fec2000f040 con 0x7fec301cd670 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 10 client.4367.objecter ms_dispatch 0x7fec30000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 3 client.4367.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 3 client.4367.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:38.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 20 client.4367.objecter dump_active .. 0 homeless 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec357cd640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec14038e10 0x7fec1403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec357cd640 10 monclient: get_auth_request con 0x7fec14038e10 auth_method 0 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec25ffb640 10 client.4367.objecter ms_handle_connect 0x7fec14038e10 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.608+0000 7fec357cd640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec14038e10 0x7fec1403b2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0ac000 tx=0x7fec2c0011a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.612+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fec20004240 con 0x7fec301cd670 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.612+0000 7fec25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.612+0000 7fec25ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:38.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec37a31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:38.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec37a31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:38.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec37a31640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:38.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec30076190 con 0x7fec301cd670 2024-04-19T00:44:38.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec25ffb640 1 -- 172.21.15.43:0/3942548613 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12229 (secure 0 0 0) 0x7fec2001cc10 con 0x7fec301cd670 2024-04-19T00:44:38.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T00:44:38.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.780+0000 7fec25ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:38.783 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:38.783 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":15,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T00:44:35.054176+0000","last_up_change":"2024-04-19T00:44:32.017099+0000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T00:44:32.605528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T00:44:32.926773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6804","nonce":110940973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6809","nonce":110940973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6821","nonce":110940973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6815","nonce":110940973}]},"public_addr":"172.21.15.43:6804/110940973","cluster_addr":"172.21.15.43:6809/110940973","heartbeat_back_addr":"172.21.15.43:6821/110940973","heartbeat_front_addr":"172.21.15.43:6815/110940973","state":["exists","up"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6802","nonce":1380723486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6808","nonce":1380723486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6820","nonce":1380723486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6814","nonce":1380723486}]},"public_addr":"172.21.15.43:6802/1380723486","cluster_addr":"172.21.15.43:6808/1380723486","heartbeat_back_addr":"172.21.15.43:6820/1380723486","heartbeat_front_addr":"172.21.15.43:6814/1380723486","state":["exists","up"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6806","nonce":1290935720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6813","nonce":1290935720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6823","nonce":1290935720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6819","nonce":1290935720}]},"public_addr":"172.21.15.43:6806/1290935720","cluster_addr":"172.21.15.43:6813/1290935720","heartbeat_back_addr":"172.21.15.43:6823/1290935720","heartbeat_front_addr":"172.21.15.43:6819/1290935720","state":["exists","up"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6801","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6804","nonce":2204255832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6810","nonce":2204255832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6821","nonce":2204255832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6815","nonce":2204255832}]},"public_addr":"172.21.15.161:6804/2204255832","cluster_addr":"172.21.15.161:6810/2204255832","heartbeat_back_addr":"172.21.15.161:6821/2204255832","heartbeat_front_addr":"172.21.15.161:6815/2204255832","state":["exists","up"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6803","nonce":3653492594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6809","nonce":3653492594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6822","nonce":3653492594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6816","nonce":3653492594}]},"public_addr":"172.21.15.161:6803/3653492594","cluster_addr":"172.21.15.161:6809/3653492594","heartbeat_back_addr":"172.21.15.161:6822/3653492594","heartbeat_front_addr":"172.21.15.161:6816/3653492594","state":["exists","up"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6805","nonce":760479844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6811","nonce":760479844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6823","nonce":760479844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6817","nonce":760479844}]},"public_addr":"172.21.15.161:6805/760479844","cluster_addr":"172.21.15.161:6811/760479844","heartbeat_back_addr":"172.21.15.161:6823/760479844","heartbeat_front_addr":"172.21.15.161:6817/760479844","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.756279+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.840443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.708058+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.848897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.721708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.855016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 10 librados: watch_flush enter 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 10 librados: watch_flush exit 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 20 client.4367.objecter shutdown clearing up homeless session... 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 10 client.4367.objecter successfully canceled tick 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec14038e10 msgr2=0x7fec1403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec14038e10 0x7fec1403b2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec2c0ac000 tx=0x7fec2c0011a0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 10 monclient: shutdown 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:38.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 msgr2=0x7fec30136cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 0x7fec30136cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fec3006e8c0 tx=0x7fec20011d70 comp rx=0 tx=0).stop 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 shutdown_connections 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec14038e10 0x7fec1403b2d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fec30137210 0x7fec301d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec301cd670 0x7fec30136cd0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 --2- 172.21.15.43:0/3942548613 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec1c0041f0 0x7fec301ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 >> 172.21.15.43:0/3942548613 conn(0x7fec300a7cb0 msgr2=0x7fec301c0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 shutdown_connections 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 -- 172.21.15.43:0/3942548613 wait complete. 2024-04-19T00:44:38.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.784+0000 7fec37a31640 1 librados: shutdown 2024-04-19T00:44:38.802 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-04-19T00:44:32.605528+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-04-19T00:44:32.926773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-19T00:44:38.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-19T00:44:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.928+0000 7fdd10f61640 10 monclient: get_monmap_and_config 2024-04-19T00:44:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.928+0000 7fdd10f61640 10 monclient: build_initial_monmap 2024-04-19T00:44:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.928+0000 7fdd10f61640 10 monclient: monmap: 2024-04-19T00:44:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:38.933860+0000 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:38.933860+0000 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:38.933 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:38.934 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 Processor -- start 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 -- start start 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient: init 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient: _add_conns ranks=[9,6,2,8,3,1,7,0,4,10,5] 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 0x7fdd0c072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): picked mon.noname-a con 0x7fdd0c0729d0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdd0c139100 0x7fdd0c0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): picked mon.noname-d con 0x7fdd0c139100 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): picked mon.noname-b con 0x7fdd0c0a4fe0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fdd0c06f830 con 0x7fdd0c0729d0 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdd0c06f270 con 0x7fdd0c0a4fe0 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fdd0c06df80 con 0x7fdd0c139100 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd10f61640 10 monclient(hunting): authenticate will time out at 949.061391s 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 0x7fdd0c072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 0x7fdd0c072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54264/0 (socket says 172.21.15.43:54264) 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 1 -- 172.21.15.43:0/2585192669 learned_addr learned my addr 172.21.15.43:0/2585192669 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): get_auth_request con 0x7fdd0c0729d0 auth_method 0 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 1 --2- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdd0c139100 0x7fdd0c0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a4fe0 auth_method 0 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): get_auth_request con 0x7fdd0c139100 auth_method 0 2024-04-19T00:44:38.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 -- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdd0c139100 msgr2=0x7fdd0c0a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 --2- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdd0c139100 0x7fdd0c0a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 -- 172.21.15.43:0/2585192669 >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 msgr2=0x7fdd0c072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 --2- 172.21.15.43:0/2585192669 >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 0x7fdd0c072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 monclient: found mon.noname-b 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0ad9d640 1 -- 172.21.15.43:0/2585192669 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c0732f0 con 0x7fdd0c0a4fe0 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.932+0000 7fdd0a59c640 1 --2- 172.21.15.43:0/2585192669 >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 0x7fdd0c072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:38.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 1 --2- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a5490 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009d90 tx=0x7fdd0401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2fcb1c3ec052162 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 1 -- 172.21.15.43:0/2585192669 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd040ac070 con 0x7fdd0c0a4fe0 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: dump: 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 1 -- 172.21.15.43:0/2585192669 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd040a68f0 con 0x7fdd0c0a4fe0 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:38.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 5 monclient: authenticate success, global_id 4376 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 1 -- 172.21.15.43:0/2585192669 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd040a7460 con 0x7fdd0c0a4fe0 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd09d9b640 10 monclient: dump: 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.949 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 4 set_mon_vals no callback set 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: shutdown 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd0c0a4fe0 msgr2=0x7fdd0c0a5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a5490 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009d90 tx=0x7fdd0401c830 comp rx=0 tx=0).stop 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2585192669 shutdown_connections 2024-04-19T00:44:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a5490 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2585192669 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdd0c139100 0x7fdd0c0a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2585192669 >> v2:172.21.15.43:3300/0 conn(0x7fdd0c0729d0 0x7fdd0c072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2585192669 >> 172.21.15.43:0/2585192669 conn(0x7fdd0c0a6410 msgr2=0x7fdd0c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2585192669 shutdown_connections 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2585192669 wait complete. 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: build_initial_monmap 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: monmap: 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 librados: starting msgr at 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 librados: starting objecter 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 Processor -- start 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- start start 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 librados: setting wanted keys 2024-04-19T00:44:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 librados: calling monclient init 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: init 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: _add_conns ranks=[0,4,8,6,3,5,7,1,2] 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 0x7fdd0c0a0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): picked mon.noname-a con 0x7fdd0c0729d0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): picked mon.noname-c con 0x7fdd0c0a4fe0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdd0c139100 0x7fdd0c0a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): picked mon.noname-e con 0x7fdd0c139100 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdd0c142ad0 con 0x7fdd0c0729d0 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdd0c142950 con 0x7fdd0c0a4fe0 2024-04-19T00:44:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdd0c142c50 con 0x7fdd0c139100 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient(hunting): authenticate will time out at 949.066632s 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 0x7fdd0c0a0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 0x7fdd0c0a0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54280/0 (socket says 172.21.15.43:54280) 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 -- 172.21.15.43:0/2329530667 learned_addr learned my addr 172.21.15.43:0/2329530667 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): get_auth_request con 0x7fdd0c0729d0 auth_method 0 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdd0c139100 0x7fdd0c0a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): get_auth_request con 0x7fdd0c139100 auth_method 0 2024-04-19T00:44:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): get_auth_request con 0x7fdd0c0a4fe0 auth_method 0 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient(hunting): handle_auth_done global_id 4410 payload 824 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 -- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdd0c139100 msgr2=0x7fdd0c0a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdd0c139100 0x7fdd0c0a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 -- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd0c0a4fe0 msgr2=0x7fdd0c0a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient: _finish_hunting 0 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 monclient: found mon.noname-a 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:38.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 -- 172.21.15.43:0/2329530667 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0c0a2800 con 0x7fdd0c0729d0 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0a59c640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 0x7fdd0c0a0f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00e9d0 tx=0x7fdcfc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 client.?.objecter ms_handle_connect 0x7fdd0c0729d0 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcfc017070 con 0x7fdd0c0729d0 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: dump: 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:38.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcfc013470 con 0x7fdd0c0729d0 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 5 monclient: authenticate success, global_id 4410 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: _renew_subs 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0c06e4f0 con 0x7fdd0c0729d0 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcfc013a90 con 0x7fdd0c0729d0 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd02ffd640 10 monclient: dump: 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:38.959 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd0959a640 4 set_mon_vals no callback set 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: _renew_subs 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0c13ab80 con 0x7fdd0c0729d0 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.936+0000 7fdd10f61640 1 librados: init done 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdcfc01a070 con 0x7fdd0c0729d0 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd10f61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd10f61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd10f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd0c0008e0 con 0x7fdd0c0729d0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcec038e10 0x7fdcec03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fdcfc00f040 con 0x7fdd0c0729d0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 10 client.4410.objecter ms_dispatch 0x7fdd0c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 3 client.4410.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 3 client.4410.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 20 client.4410.objecter dump_active .. 0 homeless 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd03fff640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcec038e10 0x7fdcec03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd03fff640 10 monclient: get_auth_request con 0x7fdcec038e10 auth_method 0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd02ffd640 10 client.4410.objecter ms_handle_connect 0x7fdcec038e10 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.940+0000 7fdd03fff640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcec038e10 0x7fdcec03b2d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0a2400 tx=0x7fdcf4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.944+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdcfc021070 con 0x7fdd0c0729d0 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.944+0000 7fdd02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:38.944+0000 7fdd02ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:39.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd10f61640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-19T00:44:39.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd10f61640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-19T00:44:39.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd10f61640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:39.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7fdd0c0a3360 con 0x7fdd0c0729d0 2024-04-19T00:44:39.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd02ffd640 1 -- 172.21.15.43:0/2329530667 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) ==== 92+0+10 (secure 0 0 0) 0x7fdcfc011350 con 0x7fdd0c0729d0 2024-04-19T00:44:39.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-19T00:44:39.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.112+0000 7fdd02ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:39.117 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 8 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 10 librados: watch_flush enter 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 10 librados: watch_flush exit 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 20 client.4410.objecter shutdown clearing up homeless session... 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 10 client.4410.objecter successfully canceled tick 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcec038e10 msgr2=0x7fdcec03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcec038e10 0x7fdcec03b2d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0a2400 tx=0x7fdcf4009190 comp rx=0 tx=0).stop 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 10 monclient: shutdown 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 msgr2=0x7fdd0c0a0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 0x7fdd0c0a0f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00e9d0 tx=0x7fdcfc00ee90 comp rx=0 tx=0).stop 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 shutdown_connections 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcec038e10 0x7fdcec03b2d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdd0c139100 0x7fdd0c0a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd0c0a4fe0 0x7fdd0c0a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 --2- 172.21.15.43:0/2329530667 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd0c0729d0 0x7fdd0c0a0f10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 >> 172.21.15.43:0/2329530667 conn(0x7fdd0c0a6410 msgr2=0x7fdd0c0a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:39.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 shutdown_connections 2024-04-19T00:44:39.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 -- 172.21.15.43:0/2329530667 wait complete. 2024-04-19T00:44:39.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.116+0000 7fdd10f61640 1 librados: shutdown 2024-04-19T00:44:39.135 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.260+0000 7fdcf415c640 10 monclient: get_monmap_and_config 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.260+0000 7fdcf415c640 10 monclient: build_initial_monmap 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.260+0000 7fdcf415c640 10 monclient: monmap: 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:39.265730+0000 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:39.265730+0000 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:39.271 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 Processor -- start 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 -- start start 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient: init 2024-04-19T00:44:39.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient: _add_conns ranks=[6,8,4,1,3,5,9,0,10,7,2] 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 0x7fdcec0a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): picked mon.noname-d con 0x7fdcec0a16f0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): picked mon.noname-e con 0x7fdcec0a0370 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdcec0a11c0 0x7fdcec06bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): picked mon.noname-c con 0x7fdcec0a11c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdcec0a3340 con 0x7fdcec0a11c0 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fdcec0a1ad0 con 0x7fdcec0a16f0 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdcec0a4820 con 0x7fdcec0a0370 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf415c640 10 monclient(hunting): authenticate will time out at 949.393307s 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 0x7fdcec0a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdcec0a11c0 0x7fdcec06bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 0x7fdcec0a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42206/0 (socket says 172.21.15.43:42206) 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): get_auth_request con 0x7fdcec0a11c0 auth_method 0 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 1 -- 172.21.15.43:0/2058878955 learned_addr learned my addr 172.21.15.43:0/2058878955 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec06b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59600/0 (socket says 172.21.15.43:59600) 2024-04-19T00:44:39.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): get_auth_request con 0x7fdcec0a0370 auth_method 0 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): get_auth_request con 0x7fdcec0a16f0 auth_method 0 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf16f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 -- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 msgr2=0x7fdcec06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec06b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 -- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 msgr2=0x7fdcec0a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 0x7fdcec0a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 monclient: found mon.noname-c 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf26f9640 1 -- 172.21.15.43:0/2058878955 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcec0a09e0 con 0x7fdcec0a11c0 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.264+0000 7fdcf1ef8640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 0x7fdcec0a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf26f9640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdcec0a11c0 0x7fdcec06bf20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0a0bb0 tx=0x7fdce401c7d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2b8587d6691cb772 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 1 -- 172.21.15.43:0/2058878955 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdce40ac070 con 0x7fdcec0a11c0 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: dump: 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.277 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 1 -- 172.21.15.43:0/2058878955 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce40a6890 con 0x7fdcec0a11c0 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 1 -- 172.21.15.43:0/2058878955 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdce40a7400 con 0x7fdcec0a11c0 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf0ef6640 10 monclient: dump: 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.279 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 5 monclient: authenticate success, global_id 4342 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 4 set_mon_vals no callback set 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: shutdown 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdcec0a11c0 msgr2=0x7fdcec06bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdcec0a11c0 0x7fdcec06bf20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0a0bb0 tx=0x7fdce401c7d0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/2058878955 shutdown_connections 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdcec0a11c0 0x7fdcec06bf20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- 172.21.15.43:0/2058878955 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdcec0a16f0 0x7fdcec0a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/2058878955 >> 172.21.15.43:0/2058878955 conn(0x7fdcec0a8500 msgr2=0x7fdcec0a8910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/2058878955 shutdown_connections 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/2058878955 wait complete. 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: build_initial_monmap 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: monmap: 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:39.281 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 librados: starting msgr at 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 librados: starting objecter 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 Processor -- start 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- start start 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 librados: setting wanted keys 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 librados: calling monclient init 2024-04-19T00:44:39.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: init 2024-04-19T00:44:39.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: _add_conns ranks=[8,3,7,2,5,4,1,6,0] 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec09dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): picked mon.noname-e con 0x7fdcec0a0370 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdcec0a11c0 0x7fdcec135ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): picked mon.noname-g con 0x7fdcec0a11c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcec0a16f0 0x7fdcec136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): picked mon.noname-i con 0x7fdcec0a16f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdcec1426e0 con 0x7fdcec0a11c0 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdcec142560 con 0x7fdcec0a16f0 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdcec142860 con 0x7fdcec0a0370 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient(hunting): authenticate will time out at 949.398810s 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec09dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec09dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59616/0 (socket says 172.21.15.43:59616) 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 -- 172.21.15.43:0/3478314462 learned_addr learned my addr 172.21.15.43:0/3478314462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): get_auth_request con 0x7fdcec0a0370 auth_method 0 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf26f9640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcec0a16f0 0x7fdcec136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 -- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcec0a16f0 msgr2=0x7fdcec136000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcec0a16f0 0x7fdcec136000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 -- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdcec0a11c0 msgr2=0x7fdcec135ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf16f7640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdcec0a11c0 0x7fdcec135ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdcec0a11c0 0x7fdcec135ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 monclient: found mon.noname-e 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 -- 172.21.15.43:0/3478314462 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcec09d480 con 0x7fdcec0a0370 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf1ef8640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec09dc00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdce800ea00 tx=0x7fdce800eec0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 client.?.objecter ms_handle_connect 0x7fdcec0a0370 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdce8017070 con 0x7fdcec0a0370 2024-04-19T00:44:39.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: dump: 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.290 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce8013470 con 0x7fdcec0a0370 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 5 monclient: authenticate success, global_id 4445 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: _renew_subs 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcec09ba70 con 0x7fdcec0a0370 2024-04-19T00:44:39.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdce8013a90 con 0x7fdcec0a0370 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce3fff640 4 set_mon_vals no callback set 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdce27fc640 10 monclient: dump: 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: _renew_subs 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcec0a38e0 con 0x7fdcec0a0370 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.268+0000 7fdcf415c640 1 librados: init done 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdce801a070 con 0x7fdcec0a0370 2024-04-19T00:44:39.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf415c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf415c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf415c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcec0008e0 con 0x7fdcec0a0370 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcd0038e10 0x7fdcd003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fdce800f040 con 0x7fdcec0a0370 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 10 client.4445.objecter ms_dispatch 0x7fdcec000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 3 client.4445.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 3 client.4445.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 20 client.4445.objecter dump_active .. 0 homeless 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf16f7640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcd0038e10 0x7fdcd003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf16f7640 10 monclient: get_auth_request con 0x7fdcd0038e10 auth_method 0 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdce27fc640 10 client.4445.objecter ms_handle_connect 0x7fdcd0038e10 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.272+0000 7fdcf16f7640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcd0038e10 0x7fdcd003b2d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdcec13fab0 tx=0x7fdcdc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.276+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdce8021070 con 0x7fdcec0a0370 2024-04-19T00:44:39.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.276+0000 7fdce27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.276+0000 7fdce27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:39.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.440+0000 7fdcf415c640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-19T00:44:39.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.440+0000 7fdcf415c640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-19T00:44:39.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.440+0000 7fdcf415c640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.440+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdcec09d100 con 0x7fdcec0a0370 2024-04-19T00:44:39.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdce27fc640 1 -- 172.21.15.43:0/3478314462 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) ==== 93+0+10 (secure 0 0 0) 0x7fdce8011350 con 0x7fdcec0a0370 2024-04-19T00:44:39.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdce27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-19T00:44:39.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdce27fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:39.445 INFO:teuthology.orchestra.run.smithi043.stdout:pg_num: 1 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 10 librados: watch_flush enter 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 10 librados: watch_flush exit 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 20 client.4445.objecter shutdown clearing up homeless session... 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 10 client.4445.objecter successfully canceled tick 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcd0038e10 msgr2=0x7fdcd003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcd0038e10 0x7fdcd003b2d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdcec13fab0 tx=0x7fdcdc008040 comp rx=0 tx=0).stop 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 10 monclient: shutdown 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 msgr2=0x7fdcec09dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec09dc00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdce800ea00 tx=0x7fdce800eec0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 shutdown_connections 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcec0a16f0 0x7fdcec136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdcec0a11c0 0x7fdcec135ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdcd0038e10 0x7fdcd003b2d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 --2- 172.21.15.43:0/3478314462 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdcec0a0370 0x7fdcec09dc00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 >> 172.21.15.43:0/3478314462 conn(0x7fdcec0a8500 msgr2=0x7fdcec13c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:39.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 shutdown_connections 2024-04-19T00:44:39.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 -- 172.21.15.43:0/3478314462 wait complete. 2024-04-19T00:44:39.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.444+0000 7fdcf415c640 1 librados: shutdown 2024-04-19T00:44:39.465 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-19T00:44:39.465 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 10 monclient: get_monmap_and_config 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 10 monclient: build_initial_monmap 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 10 monclient: monmap: 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:39.585019+0000 2024-04-19T00:44:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:39.585019+0000 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:39.584 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 1 Processor -- start 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 1 -- start start 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 10 monclient: init 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.580+0000 7f4efb198640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient: _add_conns ranks=[4,9,5,2,8,6,0,7,3,1,10] 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4138a50 0x7f4ef4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef4138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4ef40a5bb0 0x7f4ef4139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef40a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4ef4137960 0x7f4ef4137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): picked mon.noname-h con 0x7f4ef4137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef40a1020 con 0x7f4ef40a5bb0 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4ef41398b0 con 0x7f4ef4138a50 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4ef40a0780 con 0x7f4ef4137960 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient(hunting): authenticate will time out at 949.712105s 2024-04-19T00:44:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4ef40a5bb0 0x7f4ef4139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4138a50 0x7f4ef4138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4ef40a5bb0 0x7f4ef4139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54284/0 (socket says 172.21.15.43:54284) 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 -- 172.21.15.43:0/2762251783 learned_addr learned my addr 172.21.15.43:0/2762251783 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): get_auth_request con 0x7f4ef40a5bb0 auth_method 0 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): get_auth_request con 0x7f4ef4138a50 auth_method 0 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef9735640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4ef4137960 0x7f4ef4137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef9735640 10 monclient(hunting): get_auth_request con 0x7f4ef4137960 auth_method 0 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef9735640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef9735640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef9735640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_done global_id 4419 payload 274 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 -- 172.21.15.43:0/2762251783 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4ef4137960 msgr2=0x7f4ef4137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4ef4137960 0x7f4ef4137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 -- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4138a50 msgr2=0x7f4ef4138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4138a50 0x7f4ef4138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 monclient: found mon.noname-a 2024-04-19T00:44:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 -- 172.21.15.43:0/2762251783 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef40a0d40 con 0x7f4ef40a5bb0 2024-04-19T00:44:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef8f34640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4138a50 0x7f4ef4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef3fff640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef40a5bb0 0x7f4ef4139370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009870 tx=0x7f4ee801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f40277ddcc35351 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 1 -- 172.21.15.43:0/2762251783 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ee80b0070 con 0x7f4ef40a5bb0 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: dump: 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 1 -- 172.21.15.43:0/2762251783 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee80aa810 con 0x7f4ef40a5bb0 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 1 -- 172.21.15.43:0/2762251783 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ee80a9300 con 0x7f4ef40a5bb0 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 5 monclient: authenticate success, global_id 4419 2024-04-19T00:44:39.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 4 set_mon_vals no callback set 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4ef37fe640 10 monclient: dump: 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.597 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 10 monclient: shutdown 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef40a5bb0 msgr2=0x7f4ef4139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef40a5bb0 0x7f4ef4139370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8009870 tx=0x7f4ee801c3f0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- 172.21.15.43:0/2762251783 shutdown_connections 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4ef4137960 0x7f4ef4137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4ef40a5bb0 0x7f4ef4139370 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 --2- 172.21.15.43:0/2762251783 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4138a50 0x7f4ef4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- 172.21.15.43:0/2762251783 >> 172.21.15.43:0/2762251783 conn(0x7f4ef40a7cb0 msgr2=0x7f4ef40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:39.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- 172.21.15.43:0/2762251783 shutdown_connections 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.584+0000 7f4efb198640 1 -- 172.21.15.43:0/2762251783 wait complete. 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: build_initial_monmap 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: monmap: 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:39.599 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 librados: starting msgr at 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 librados: starting objecter 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 Processor -- start 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- start start 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 librados: setting wanted keys 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 librados: calling monclient init 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: init 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _add_conns ranks=[3,4,1,0,5,8,7,6,2] 2024-04-19T00:44:39.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ef40a5bb0 0x7f4ef40a45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): picked mon.noname-g con 0x7f4ef40a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 0x7f4ef41d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef4137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ef4138a50 0x7f4ef41d1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): picked mon.noname-f con 0x7f4ef4138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef40a2100 con 0x7f4ef4138a50 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef40a1f80 con 0x7f4ef40a5bb0 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4ef4142810 con 0x7f4ef4137960 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient(hunting): authenticate will time out at 949.717266s 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 0x7f4ef41d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 0x7f4ef41d0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48616/0 (socket says 172.21.15.43:48616) 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 -- 172.21.15.43:0/1978834852 learned_addr learned my addr 172.21.15.43:0/1978834852 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): get_auth_request con 0x7f4ef4137960 auth_method 0 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef8f34640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ef40a5bb0 0x7f4ef40a45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef9735640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ef4138a50 0x7f4ef41d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 -- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ef40a5bb0 msgr2=0x7f4ef40a45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ef40a5bb0 0x7f4ef40a45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 -- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ef4138a50 msgr2=0x7f4ef41d1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ef4138a50 0x7f4ef41d1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 monclient: found mon.noname-c 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 -- 172.21.15.43:0/1978834852 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef409c130 con 0x7f4ef4137960 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef9735640 10 monclient: get_auth_request con 0x7f4ef4138a50 auth_method 0 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef9735640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ef4138a50 0x7f4ef41d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef8f34640 10 monclient: get_auth_request con 0x7f4ef40a5bb0 auth_method 0 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef8f34640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ef40a5bb0 0x7f4ef40a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef3fff640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 0x7f4ef41d0dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80ae040 tx=0x7f4ee8019190 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 client.?.objecter ms_handle_connect 0x7f4ef4137960 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:39.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ee80b0040 con 0x7f4ef4137960 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: dump: 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.604 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee80044a0 con 0x7f4ef4137960 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 5 monclient: authenticate success, global_id 4360 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _renew_subs 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef409c410 con 0x7f4ef4137960 2024-04-19T00:44:39.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ee80aad70 con 0x7f4ef4137960 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 monclient: dump: 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.606 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _renew_subs 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef409fc90 con 0x7f4ef4137960 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 librados: init done 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef2ffd640 4 set_mon_vals no callback set 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef409ee60 con 0x7f4ef4137960 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4ee8004030 con 0x7f4ef4137960 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ee0038e10 0x7f4ee003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f4ee80e2080 con 0x7f4ef4137960 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 10 client.4360.objecter ms_dispatch 0x7f4ef4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 3 client.4360.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 3 client.4360.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef17fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef8f34640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ee0038e10 0x7f4ee003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.588+0000 7f4ef8f34640 10 monclient: get_auth_request con 0x7f4ee0038e10 auth_method 0 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.592+0000 7f4ef17fa640 10 client.4360.objecter ms_handle_connect 0x7f4ee0038e10 2024-04-19T00:44:39.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.592+0000 7f4ef8f34640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ee0038e10 0x7f4ee003b2d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40099c0 tx=0x7f4ee4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.596+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4ee8016130 con 0x7f4ef4137960 2024-04-19T00:44:39.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.596+0000 7f4ef17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.596+0000 7f4ef17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.748+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4ef41d1850 con 0x7f4ee0038e10 2024-04-19T00:44:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.752+0000 7f4ef17fa640 1 -- 172.21.15.43:0/1978834852 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33776 (secure 0 0 0) 0x7f4ef41d1850 con 0x7f4ee0038e10 2024-04-19T00:44:39.756 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 10 librados: watch_flush enter 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 10 librados: watch_flush exit 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 20 client.4360.objecter shutdown clearing up homeless session... 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 10 client.4360.objecter successfully canceled tick 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ee0038e10 msgr2=0x7f4ee003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ee0038e10 0x7f4ee003b2d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40099c0 tx=0x7f4ee4008040 comp rx=0 tx=0).stop 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 10 monclient: shutdown 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 msgr2=0x7f4ef41d0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 0x7f4ef41d0dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ee80ae040 tx=0x7f4ee8019190 comp rx=0 tx=0).stop 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 shutdown_connections 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ee0038e10 0x7f4ee003b2d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4ef4138a50 0x7f4ef41d1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ef4137960 0x7f4ef41d0dd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 --2- 172.21.15.43:0/1978834852 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4ef40a5bb0 0x7f4ef40a45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 >> 172.21.15.43:0/1978834852 conn(0x7f4ef40a7cb0 msgr2=0x7f4ef406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 shutdown_connections 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 -- 172.21.15.43:0/1978834852 wait complete. 2024-04-19T00:44:39.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.756+0000 7f4efb198640 1 librados: shutdown 2024-04-19T00:44:39.773 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-19T00:44:38.879683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":169060,"kb_used_data":1880,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566061980,"statfs":{"total":579820584960,"available":579647467520,"internally_reserved":0,"allocated":1925120,"data_stored":1311062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000581"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063505+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T00:44:35.063505+0000","last_peered":"2024-04-19T00:44:35.063505+0000","last_clean":"2024-04-19T00:44:35.063505+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T00:44:35.063505+0000","last_undegraded":"2024-04-19T00:44:35.063505+0000","last_fullsized":"2024-04-19T00:44:35.063505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.842484+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T00:44:35.842484+0000","last_peered":"2024-04-19T00:44:35.842484+0000","last_clean":"2024-04-19T00:44:35.842484+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T00:44:35.842484+0000","last_undegraded":"2024-04-19T00:44:35.842484+0000","last_fullsized":"2024-04-19T00:44:35.842484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:36:04.048348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.843538+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T00:44:35.843538+0000","last_peered":"2024-04-19T00:44:35.843538+0000","last_clean":"2024-04-19T00:44:35.843538+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T00:44:35.843538+0000","last_undegraded":"2024-04-19T00:44:35.843538+0000","last_fullsized":"2024-04-19T00:44:35.843538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:04:03.225536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.062032+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T00:44:35.062032+0000","last_peered":"2024-04-19T00:44:35.062032+0000","last_clean":"2024-04-19T00:44:35.062032+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T00:44:35.062032+0000","last_undegraded":"2024-04-19T00:44:35.062032+0000","last_fullsized":"2024-04-19T00:44:35.062032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.842725+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T00:44:35.842725+0000","last_peered":"2024-04-19T00:44:35.842725+0000","last_clean":"2024-04-19T00:44:35.842725+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T00:44:35.842725+0000","last_undegraded":"2024-04-19T00:44:35.842725+0000","last_fullsized":"2024-04-19T00:44:35.842725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:02:45.312062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.061815+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T00:44:35.061815+0000","last_peered":"2024-04-19T00:44:35.061815+0000","last_clean":"2024-04-19T00:44:35.061815+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T00:44:35.061815+0000","last_undegraded":"2024-04-19T00:44:35.061815+0000","last_fullsized":"2024-04-19T00:44:35.061815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.843551+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T00:44:35.843551+0000","last_peered":"2024-04-19T00:44:35.843551+0000","last_clean":"2024-04-19T00:44:35.843551+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T00:44:35.843551+0000","last_undegraded":"2024-04-19T00:44:35.843551+0000","last_fullsized":"2024-04-19T00:44:35.843551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:48:30.196169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063480+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T00:44:35.063480+0000","last_peered":"2024-04-19T00:44:35.063480+0000","last_clean":"2024-04-19T00:44:35.063480+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T00:44:35.063480+0000","last_undegraded":"2024-04-19T00:44:35.063480+0000","last_fullsized":"2024-04-19T00:44:35.063480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.842463+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T00:44:35.842463+0000","last_peered":"2024-04-19T00:44:35.842463+0000","last_clean":"2024-04-19T00:44:35.842463+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T00:44:35.842463+0000","last_undegraded":"2024-04-19T00:44:35.842463+0000","last_fullsized":"2024-04-19T00:44:35.842463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:18:46.046655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":675840,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":675840,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T00:44:39.774 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T00:44:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.884+0000 7f71e0adb640 10 monclient: get_monmap_and_config 2024-04-19T00:44:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.884+0000 7f71e0adb640 10 monclient: build_initial_monmap 2024-04-19T00:44:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.884+0000 7f71e0adb640 10 monclient: monmap: 2024-04-19T00:44:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.887 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:39.889018+0000 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:39.889018+0000 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:39.888 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.884+0000 7f71e0adb640 1 Processor -- start 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.884+0000 7f71e0adb640 1 -- start start 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.884+0000 7f71e0adb640 10 monclient: init 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient: _add_conns ranks=[1,3,0,9,7,6,4,5,10,2,8] 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f71dc138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71dc139540 0x7f71dc13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient(hunting): picked mon.noname-g con 0x7f71dc139540 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:39.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71dc0a5bb0 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71dc0a0d40 con 0x7f71dc0a5bb0 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71dc0a0a60 con 0x7f71dc138a50 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc0a1020 con 0x7f71dc139540 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient(hunting): authenticate will time out at 950.016187s 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f71dc0a5bb0 0x7f71dc140530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44804/0 (socket says 172.21.15.43:44804) 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 -- 172.21.15.43:0/1417135445 learned_addr learned my addr 172.21.15.43:0/1417135445 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71dc139540 0x7f71dc13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request con 0x7f71dc139540 auth_method 0 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29084588 0 0) 0x7f71dc0a0d40 con 0x7f71dc0a5bb0 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): _negotiate 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): my global_id is 4428 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c8003530 con 0x7f71dc0a5bb0 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3356467031 0 0) 0x7f71c8003530 con 0x7f71dc0a5bb0 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71dc139540 msgr2=0x7f71dc13dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f71dc139540 0x7f71dc13dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 >> v1:172.21.15.161:6789/0 conn(0x7f71dc138a50 legacy=0x7f71dc138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc0a17b0 con 0x7f71dc0a5bb0 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f71d4001cf0 con 0x7f71dc0a5bb0 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 5 monclient: authenticate success, global_id 4428 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: dump: 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.897 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.898 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 >> v1:172.21.15.43:6789/0 conn(0x7f71dc0a5bb0 legacy=0x7f71dc140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: _add_conns ranks=[6,1,8,7,5,0,2,4,3] 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71c8004240 0x7f71c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): picked mon.d con 0x7f71c8004240 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71c8004da0 0x7f71c8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): picked mon.f con 0x7f71c8004da0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71c8009c60 0x7f71c800c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): picked mon.e con 0x7f71c8009c60 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f71dc0a1020 con 0x7f71c8004da0 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f71c800c9d0 con 0x7f71c8004240 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f71c800cb80 con 0x7f71c8009c60 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71c8009c60 0x7f71c800c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71c8004240 0x7f71c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): get_auth_request con 0x7f71c8009c60 auth_method 0 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): get_auth_request con 0x7f71c8004240 auth_method 0 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71da59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71c8004da0 0x7f71c8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request con 0x7f71c8004da0 auth_method 0 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient(hunting): handle_auth_done global_id 4428 payload 293 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 -- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71c8004240 msgr2=0x7f71c8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71c8004240 0x7f71c8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 -- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71c8004da0 msgr2=0x7f71c8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f71c8004da0 0x7f71c8005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 monclient: found mon.e 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 -- 172.21.15.43:0/1417135445 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71c8003530 con 0x7f71c8009c60 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 -- 172.21.15.43:0/1417135445 reap_dead start 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71dad9d640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71c8009c60 0x7f71c800c0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f71d400a660 tx=0x7f71d401d0c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b96b7fd21b7e75bc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71d400a280 con 0x7f71c8009c60 2024-04-19T00:44:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: dump: 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.903 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d40a9070 con 0x7f71c8009c60 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 1 -- 172.21.15.43:0/1417135445 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71d40079b0 con 0x7f71c8009c60 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71e0adb640 4 set_mon_vals no callback set 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.888+0000 7f71d959a640 10 monclient: dump: 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.904 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: shutdown 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71c8009c60 msgr2=0x7f71c800c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71c8009c60 0x7f71c800c0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f71d400a660 tx=0x7f71d401d0c0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/1417135445 shutdown_connections 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 --2- 172.21.15.43:0/1417135445 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71c8009c60 0x7f71c800c0f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/1417135445 >> 172.21.15.43:0/1417135445 conn(0x7f71dc0a7cb0 msgr2=0x7f71dc0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/1417135445 shutdown_connections 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/1417135445 wait complete. 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: build_initial_monmap 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: monmap: 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:39.906 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 librados: starting msgr at 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 librados: starting objecter 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 Processor -- start 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- start start 2024-04-19T00:44:39.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 librados: setting wanted keys 2024-04-19T00:44:39.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 librados: calling monclient init 2024-04-19T00:44:39.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: init 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _add_conns ranks=[5,2,8,1,3,7,4,6,0] 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71c8009c60 0x7f71dc140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): picked mon.noname-h con 0x7f71c8009c60 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 0x7f71dc136ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): picked mon.noname-b con 0x7f71dc136ac0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 0x7f71dc13c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): picked mon.noname-e con 0x7f71dc1373e0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f71dc1391c0 con 0x7f71dc136ac0 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f71dc139040 con 0x7f71c8009c60 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f71dc139340 con 0x7f71dc1373e0 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient(hunting): authenticate will time out at 950.022031s 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 0x7f71dc136ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 0x7f71dc13c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 0x7f71dc136ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35354/0 (socket says 172.21.15.43:35354) 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 0x7f71dc13c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59644/0 (socket says 172.21.15.43:59644) 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 1 -- 172.21.15.43:0/2007365239 learned_addr learned my addr 172.21.15.43:0/2007365239 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request con 0x7f71dc136ac0 auth_method 0 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): get_auth_request con 0x7f71dc1373e0 auth_method 0 2024-04-19T00:44:39.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71da59c640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71c8009c60 0x7f71dc140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71da59c640 10 monclient(hunting): get_auth_request con 0x7f71c8009c60 auth_method 0 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71da59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71da59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:39.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 -- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71c8009c60 msgr2=0x7f71dc140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71c8009c60 0x7f71dc140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 -- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 msgr2=0x7f71dc136ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 0x7f71dc136ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient: _finish_hunting 0 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 monclient: found mon.noname-e 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 -- 172.21.15.43:0/2007365239 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71dc1401c0 con 0x7f71dc1373e0 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d9d9b640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 0x7f71dc136ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71dad9d640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 0x7f71dc13c880 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71dc09fb00 tx=0x7f71c8005840 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 client.?.objecter ms_handle_connect 0x7f71dc1373e0 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71d40bb070 con 0x7f71dc1373e0 2024-04-19T00:44:39.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: dump: 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.916 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d4001cd0 con 0x7f71dc1373e0 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71d400a280 con 0x7f71dc1373e0 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 5 monclient: authenticate success, global_id 4454 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _renew_subs 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71dc1404a0 con 0x7f71dc1373e0 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d8d99640 4 set_mon_vals no callback set 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:39.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71d2ffd640 10 monclient: dump: 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:39.918 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _renew_subs 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f71dc137860 con 0x7f71dc1373e0 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 1 librados: init done 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.892+0000 7f71e0adb640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71dc0008e0 con 0x7f71dc1373e0 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f71d40ab030 con 0x7f71dc1373e0 2024-04-19T00:44:39.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f71bc038e10 0x7f71bc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f71d40b3070 con 0x7f71dc1373e0 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 10 client.4454.objecter ms_dispatch 0x7f71dc000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 3 client.4454.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 3 client.4454.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 20 client.4454.objecter dump_active .. 0 homeless 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71da59c640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f71bc038e10 0x7f71bc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71da59c640 10 monclient: get_auth_request con 0x7f71bc038e10 auth_method 0 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71da59c640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f71bc038e10 0x7f71bc03b2d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71cc007fd0 tx=0x7f71cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.896+0000 7f71d2ffd640 10 client.4454.objecter ms_handle_connect 0x7f71bc038e10 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.900+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f71d4016e70 con 0x7f71dc1373e0 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.900+0000 7f71d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:39.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:39.900+0000 7f71d2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:40.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.060+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f71dc13fee0 con 0x7f71bc038e10 2024-04-19T00:44:40.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71d2ffd640 1 -- 172.21.15.43:0/2007365239 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33776 (secure 0 0 0) 0x7f71dc13fee0 con 0x7f71bc038e10 2024-04-19T00:44:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T00:44:40.066 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 10 librados: watch_flush enter 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 10 librados: watch_flush exit 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 20 client.4454.objecter shutdown clearing up homeless session... 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 10 client.4454.objecter successfully canceled tick 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f71bc038e10 msgr2=0x7f71bc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f71bc038e10 0x7f71bc03b2d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f71cc007fd0 tx=0x7f71cc009040 comp rx=0 tx=0).stop 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 10 monclient: shutdown 2024-04-19T00:44:40.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 msgr2=0x7f71dc13c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 0x7f71dc13c880 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f71dc09fb00 tx=0x7f71c8005840 comp rx=0 tx=0).stop 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 shutdown_connections 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f71bc038e10 0x7f71bc03b2d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f71dc1373e0 0x7f71dc13c880 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f71dc136ac0 0x7f71dc136ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 --2- 172.21.15.43:0/2007365239 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71c8009c60 0x7f71dc140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 >> 172.21.15.43:0/2007365239 conn(0x7f71dc0a7cb0 msgr2=0x7f71dc0a5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 shutdown_connections 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 -- 172.21.15.43:0/2007365239 wait complete. 2024-04-19T00:44:40.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.064+0000 7f71e0adb640 1 librados: shutdown 2024-04-19T00:44:40.084 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-19T00:44:38.879683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":169060,"kb_used_data":1880,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566061980,"statfs":{"total":579820584960,"available":579647467520,"internally_reserved":0,"allocated":1925120,"data_stored":1311062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000581"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063505+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T00:44:35.063505+0000","last_peered":"2024-04-19T00:44:35.063505+0000","last_clean":"2024-04-19T00:44:35.063505+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T00:44:35.063505+0000","last_undegraded":"2024-04-19T00:44:35.063505+0000","last_fullsized":"2024-04-19T00:44:35.063505+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.842484+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T00:44:35.842484+0000","last_peered":"2024-04-19T00:44:35.842484+0000","last_clean":"2024-04-19T00:44:35.842484+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T00:44:35.842484+0000","last_undegraded":"2024-04-19T00:44:35.842484+0000","last_fullsized":"2024-04-19T00:44:35.842484+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:36:04.048348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.843538+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T00:44:35.843538+0000","last_peered":"2024-04-19T00:44:35.843538+0000","last_clean":"2024-04-19T00:44:35.843538+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T00:44:35.843538+0000","last_undegraded":"2024-04-19T00:44:35.843538+0000","last_fullsized":"2024-04-19T00:44:35.843538+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:04:03.225536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.062032+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T00:44:35.062032+0000","last_peered":"2024-04-19T00:44:35.062032+0000","last_clean":"2024-04-19T00:44:35.062032+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T00:44:35.062032+0000","last_undegraded":"2024-04-19T00:44:35.062032+0000","last_fullsized":"2024-04-19T00:44:35.062032+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":162,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.842725+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T00:44:35.842725+0000","last_peered":"2024-04-19T00:44:35.842725+0000","last_clean":"2024-04-19T00:44:35.842725+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T00:44:35.842725+0000","last_undegraded":"2024-04-19T00:44:35.842725+0000","last_fullsized":"2024-04-19T00:44:35.842725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:02:45.312062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.061815+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T00:44:35.061815+0000","last_peered":"2024-04-19T00:44:35.061815+0000","last_clean":"2024-04-19T00:44:35.061815+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T00:44:35.061815+0000","last_undegraded":"2024-04-19T00:44:35.061815+0000","last_fullsized":"2024-04-19T00:44:35.061815+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.843551+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T00:44:35.843551+0000","last_peered":"2024-04-19T00:44:35.843551+0000","last_clean":"2024-04-19T00:44:35.843551+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T00:44:35.843551+0000","last_undegraded":"2024-04-19T00:44:35.843551+0000","last_fullsized":"2024-04-19T00:44:35.843551+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:48:30.196169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.063480+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T00:44:35.063480+0000","last_peered":"2024-04-19T00:44:35.063480+0000","last_clean":"2024-04-19T00:44:35.063480+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T00:44:35.063480+0000","last_undegraded":"2024-04-19T00:44:35.063480+0000","last_fullsized":"2024-04-19T00:44:35.063480+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-19T00:44:35.842463+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T00:44:35.842463+0000","last_peered":"2024-04-19T00:44:35.842463+0000","last_clean":"2024-04-19T00:44:35.842463+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T00:44:35.842463+0000","last_undegraded":"2024-04-19T00:44:35.842463+0000","last_fullsized":"2024-04-19T00:44:35.842463+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:18:46.046655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":675840,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31324,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340516,"statfs":{"total":96636764160,"available":96604688384,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28124,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343716,"statfs":{"total":96636764160,"available":96607965184,"internally_reserved":0,"allocated":675840,"data_stored":571129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T00:44:40.085 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-19T00:44:40.085 DEBUG:tasks.mon_thrash:Yielding 2024-04-19T00:44:40.085 INFO:teuthology.run_tasks:Running task workunit... 2024-04-19T00:44:40.098 INFO:tasks.workunit:Pulling workunits from ref 8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:44:40.099 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-19T00:44:40.099 DEBUG:teuthology.orchestra.run.smithi161:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-19T00:44:40.099 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-19T00:44:40.099 INFO:tasks.mon_thrash.mon_thrasher:seed: 1713487480, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-04-19T00:44:40.099 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:44:40.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:44:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T00:44:40.106 INFO:teuthology.orchestra.run.smithi161.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-19T00:44:40.106 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-19T00:44:40.159 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-19T00:44:40.159 DEBUG:teuthology.orchestra.run.smithi161:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-19T00:44:40.211 INFO:tasks.workunit:timeout=3h 2024-04-19T00:44:40.211 INFO:tasks.workunit:cleanup=True 2024-04-19T00:44:40.212 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8b4ab94f9c30da63487fa2f241ce12f483378860 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient: get_monmap_and_config 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient: monmap: 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:40.219585+0000 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:40.219585+0000 2024-04-19T00:44:40.219 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.220 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:40.221 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:40.221 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 Processor -- start 2024-04-19T00:44:40.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 -- start start 2024-04-19T00:44:40.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient: init 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient: _add_conns ranks=[10,0,5,1,3,4,7,9,6,2,8] 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f60bc135ca0 0x7f60bc136080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): picked mon.noname-f con 0x7f60bc135ca0 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60bc073200 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60bc0a3fe0 0x7f60bc13db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): picked mon.noname-h con 0x7f60bc0a3fe0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60bc09fdf0 con 0x7f60bc073200 2024-04-19T00:44:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f60bc0a16f0 con 0x7f60bc135ca0 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f60bc0a2320 con 0x7f60bc0a3fe0 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 10 monclient(hunting): authenticate will time out at 950.346906s 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60ba7fc640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f60bc073200 0x7f60bc0a38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44816/0 (socket says 172.21.15.43:44816) 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60ba7fc640 1 -- 172.21.15.43:0/1887389444 learned_addr learned my addr 172.21.15.43:0/1887389444 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 1 --2- 172.21.15.43:0/1887389444 >> v2:172.21.15.161:3300/0 conn(0x7f60bc135ca0 0x7f60bc136080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): get_auth_request con 0x7f60bc135ca0 auth_method 0 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60bc0a3fe0 0x7f60bc13db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2034939286 0 0) 0x7f60bc09fdf0 con 0x7f60bc073200 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): my global_id is 4437 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60a4003530 con 0x7f60bc073200 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): get_auth_request con 0x7f60bc0a3fe0 auth_method 0 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 608926096 0 0) 0x7f60a4003530 con 0x7f60bc073200 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60bc0a3fe0 msgr2=0x7f60bc13db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60bc0a3fe0 0x7f60bc13db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 >> v2:172.21.15.161:3300/0 conn(0x7f60bc135ca0 msgr2=0x7f60bc136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 --2- 172.21.15.43:0/1887389444 >> v2:172.21.15.161:3300/0 conn(0x7f60bc135ca0 0x7f60bc136080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc0a10a0 con 0x7f60bc073200 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60c141c640 5 monclient: authenticate success, global_id 4437 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f60b0001970 con 0x7f60bc073200 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: dump: 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 >> v1:172.21.15.43:6789/0 conn(0x7f60bc073200 legacy=0x7f60bc0a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient: _add_conns ranks=[6,3,4,7,8,5,2,0,1] 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60a4004210 0x7f60a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): picked mon.d con 0x7f60a4004210 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60a4004da0 0x7f60a4005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): picked mon.g con 0x7f60a4004da0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f60a4009ab0 0x7f60a400bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): picked mon.c con 0x7f60a4009ab0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f60bc0a16f0 con 0x7f60a4004da0 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f60bc0a2320 con 0x7f60a4009ab0 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f60a400c850 con 0x7f60a4004210 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.216+0000 7f60b9ffb640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60a4004210 0x7f60a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): get_auth_request con 0x7f60a4004210 auth_method 0 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f60a4009ab0 0x7f60a400bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): get_auth_request con 0x7f60a4009ab0 auth_method 0 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60ba7fc640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60a4004da0 0x7f60a4005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60ba7fc640 10 monclient(hunting): get_auth_request con 0x7f60a4004da0 auth_method 0 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60ba7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60ba7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient(hunting): handle_auth_done global_id 4437 payload 293 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 -- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f60a4009ab0 msgr2=0x7f60a400bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f60a4009ab0 0x7f60a400bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 -- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60a4004da0 msgr2=0x7f60a4005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60a4004da0 0x7f60a4005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 monclient: found mon.d 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 -- 172.21.15.43:0/1887389444 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a4003530 con 0x7f60a4004210 2024-04-19T00:44:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60bb7fe640 1 -- 172.21.15.43:0/1887389444 reap_dead start 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60baffd640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60a4004210 0x7f60a4004810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009f90 tx=0x7f60ac01cc40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6bb93d606df5d623 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60ac0b2070 con 0x7f60a4004210 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: dump: 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.241 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ac0a5ca0 con 0x7f60a4004210 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 4 set_mon_vals no callback set 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 1 -- 172.21.15.43:0/1887389444 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60ac009c10 con 0x7f60a4004210 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60b9ffb640 10 monclient: dump: 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 10 monclient: shutdown 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 -- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60a4004210 msgr2=0x7f60a4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60a4004210 0x7f60a4004810 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f60ac009f90 tx=0x7f60ac01cc40 comp rx=0 tx=0).stop 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 -- 172.21.15.43:0/1887389444 shutdown_connections 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 --2- 172.21.15.43:0/1887389444 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60a4004210 0x7f60a4004810 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 -- 172.21.15.43:0/1887389444 >> 172.21.15.43:0/1887389444 conn(0x7f60bc0a84c0 msgr2=0x7f60bc0a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 -- 172.21.15.43:0/1887389444 shutdown_connections 2024-04-19T00:44:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 1 -- 172.21.15.43:0/1887389444 wait complete. 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.220+0000 7f60c141c640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: monmap: 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 librados: starting msgr at 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 librados: starting objecter 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 Processor -- start 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- start start 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 librados: setting wanted keys 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 librados: calling monclient init 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: init 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _add_conns ranks=[2,7,1,5,0,3,6,8,4] 2024-04-19T00:44:40.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 0x7f60bc1cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): picked mon.noname-b con 0x7f60a4004210 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f60bc11ed10 0x7f60bc11f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): picked mon.noname-i con 0x7f60bc11ed10 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60bc11f690 0x7f60bc11fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): picked mon.noname-f con 0x7f60bc11f690 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc0a3e60 con 0x7f60bc11f690 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f60bc0a3ce0 con 0x7f60a4004210 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f60bc06b9e0 con 0x7f60bc11ed10 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient(hunting): authenticate will time out at 950.352991s 2024-04-19T00:44:40.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 0x7f60bc1cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 0x7f60bc1cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35364/0 (socket says 172.21.15.43:35364) 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 -- 172.21.15.43:0/1351768124 learned_addr learned my addr 172.21.15.43:0/1351768124 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): get_auth_request con 0x7f60a4004210 auth_method 0 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f60bc11ed10 0x7f60bc11f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): get_auth_request con 0x7f60bc11ed10 auth_method 0 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60bb7fe640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60bc11f690 0x7f60bc11fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60bb7fe640 10 monclient(hunting): get_auth_request con 0x7f60bc11f690 auth_method 0 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 -- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f60bc11ed10 msgr2=0x7f60bc11f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f60bc11ed10 0x7f60bc11f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 -- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60bc11f690 msgr2=0x7f60bc11fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60bc11f690 0x7f60bc11fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 monclient: found mon.noname-b 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 -- 172.21.15.43:0/1351768124 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc0a05d0 con 0x7f60a4004210 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60baffd640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 0x7f60bc1cd050 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60ac01ca70 tx=0x7f60ac009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 client.?.objecter ms_handle_connect 0x7f60a4004210 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60ac0b2040 con 0x7f60a4004210 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: dump: 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.250 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ac0bb070 con 0x7f60a4004210 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 5 monclient: authenticate success, global_id 4394 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _renew_subs 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60bc1214f0 con 0x7f60a4004210 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60ac0044b0 con 0x7f60a4004210 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60b97fa640 4 set_mon_vals no callback set 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 monclient: dump: 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.252 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:40.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _renew_subs 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60bc1229a0 con 0x7f60a4004210 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 librados: init done 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f60ac004650 con 0x7f60a4004210 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60bc120990 con 0x7f60a4004210 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f60a8038e40 0x7f60a803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f60ac0e32a0 con 0x7f60a4004210 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 10 client.4394.objecter ms_dispatch 0x7f60bc000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 3 client.4394.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 3 client.4394.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:40.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60a37fe640 20 client.4394.objecter dump_active .. 0 homeless 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f60a8038e40 0x7f60a803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.224+0000 7f60ba7fc640 10 monclient: get_auth_request con 0x7f60a8038e40 auth_method 0 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.228+0000 7f60ba7fc640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f60a8038e40 0x7f60a803b300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f60b0001ae0 tx=0x7f60b0007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.228+0000 7f60a37fe640 10 client.4394.objecter ms_handle_connect 0x7f60a8038e40 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.228+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f60bc120990 con 0x7f60a4004210 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.228+0000 7f60a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.228+0000 7f60a37fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:40.262 INFO:tasks.workunit.client.0.smithi161.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-19T00:44:40.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.456+0000 7f60c141c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:44:40.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.456+0000 7f60c141c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:44:40.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.456+0000 7f60c141c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.456+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60bc076500 con 0x7f60a4004210 2024-04-19T00:44:40.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60a37fe640 1 -- 172.21.15.43:0/1351768124 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f60bc0a05d0 con 0x7f60a4004210 2024-04-19T00:44:40.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:44:40.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60a37fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:40.461 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":8,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:44:40.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 10 librados: watch_flush enter 2024-04-19T00:44:40.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 10 librados: watch_flush exit 2024-04-19T00:44:40.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 20 client.4394.objecter shutdown clearing up homeless session... 2024-04-19T00:44:40.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 10 client.4394.objecter successfully canceled tick 2024-04-19T00:44:40.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f60a8038e40 msgr2=0x7f60a803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f60a8038e40 0x7f60a803b300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f60b0001ae0 tx=0x7f60b0007040 comp rx=0 tx=0).stop 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 10 monclient: shutdown 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 msgr2=0x7f60bc1cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 0x7f60bc1cd050 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f60ac01ca70 tx=0x7f60ac009750 comp rx=0 tx=0).stop 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 shutdown_connections 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f60a8038e40 0x7f60a803b300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f60bc11f690 0x7f60bc11fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f60bc11ed10 0x7f60bc11f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 --2- 172.21.15.43:0/1351768124 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f60a4004210 0x7f60bc1cd050 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 >> 172.21.15.43:0/1351768124 conn(0x7f60bc0a84c0 msgr2=0x7f60bc070020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 shutdown_connections 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 -- 172.21.15.43:0/1351768124 wait complete. 2024-04-19T00:44:40.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.460+0000 7f60c141c640 1 librados: shutdown 2024-04-19T00:44:40.479 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:44:40.480 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:44:40.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:44:40.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:44:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient: get_monmap_and_config 2024-04-19T00:44:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient: monmap: 2024-04-19T00:44:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.593 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:40.594973+0000 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:40.594973+0000 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.594 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.595 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:40.595 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:40.595 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 1 Processor -- start 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 1 -- start start 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient: init 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient: _add_conns ranks=[1,0,3,7,9,10,2,8,4,6,5] 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2738138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2738139540 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 0x7f2738141710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient(hunting): picked mon.noname-g con 0x7f27380a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:40.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27380a0d40 con 0x7f2738139540 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27380a0a60 con 0x7f2738138a50 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f27380a1020 con 0x7f27380a5bb0 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 10 monclient(hunting): authenticate will time out at 950.722631s 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2738139540 0x7f273813dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44824/0 (socket says 172.21.15.43:44824) 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 1 -- 172.21.15.43:0/1591438513 learned_addr learned my addr 172.21.15.43:0/1591438513 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 0x7f2738141710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360064802 0 0) 0x7f27380a0d40 con 0x7f2738139540 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): _negotiate 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): my global_id is 4446 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2724003530 con 0x7f2738139540 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): get_auth_request con 0x7f27380a5bb0 auth_method 0 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273debd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3997136699 0 0) 0x7f2724003530 con 0x7f2738139540 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 msgr2=0x7f2738141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 0x7f2738141710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 >> v1:172.21.15.161:6789/0 conn(0x7f2738138a50 legacy=0x7f2738138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27380a17b0 con 0x7f2738139540 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 5 monclient: authenticate success, global_id 4446 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f2728001970 con 0x7f2738139540 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: dump: 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.604 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 >> v1:172.21.15.43:6789/0 conn(0x7f2738139540 legacy=0x7f273813dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient: _add_conns ranks=[3,0,7,8,6,4,2,5,1] 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 0x7f2724004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): picked mon.g con 0x7f27380a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2724004860 0x7f2724004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): picked mon.a con 0x7f2724004860 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27240099c0 0x7f272400bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): picked mon.i con 0x7f27240099c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f27380a1020 con 0x7f2724004860 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f272400c3d0 con 0x7f27380a5bb0 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f272400c580 con 0x7f27240099c0 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f272ffff640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273f920640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2724004860 0x7f2724004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): get_auth_request con 0x7f2724004860 auth_method 0 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273cebb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.592+0000 7f273d6bc640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 0x7f2724004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273d6bc640 10 monclient(hunting): get_auth_request con 0x7f27380a5bb0 auth_method 0 2024-04-19T00:44:40.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273d6bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273d6bc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273d6bc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 10 monclient(hunting): handle_auth_done global_id 4446 payload 293 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 -- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27240099c0 msgr2=0x7f272400bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273debd640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27240099c0 0x7f272400bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27240099c0 0x7f272400bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 -- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 msgr2=0x7f2724004240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27380a5bb0 0x7f2724004240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 monclient: found mon.a 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:40.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 -- 172.21.15.43:0/1591438513 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2724003530 con 0x7f2724004860 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273cebb640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2724004860 0x7f2724004cc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2728007f00 tx=0x7f272801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dcb50f1875b9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2728007b20 con 0x7f2724004860 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: dump: 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.609 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27280a9020 con 0x7f2724004860 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:40.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 1 -- 172.21.15.43:0/1591438513 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2728005d50 con 0x7f2724004860 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 4 set_mon_vals no callback set 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f272ffff640 10 monclient: dump: 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.611 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 10 monclient: shutdown 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 1 -- 172.21.15.43:0/1591438513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2724004860 msgr2=0x7f2724004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 1 --2- 172.21.15.43:0/1591438513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2724004860 0x7f2724004cc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2728007f00 tx=0x7f272801c9e0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273debd640 1 -- 172.21.15.43:0/1591438513 reap_dead start 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 1 -- 172.21.15.43:0/1591438513 shutdown_connections 2024-04-19T00:44:40.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 1 -- 172.21.15.43:0/1591438513 >> 172.21.15.43:0/1591438513 conn(0x7f27380a7cb0 msgr2=0x7f27380a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 1 -- 172.21.15.43:0/1591438513 shutdown_connections 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.596+0000 7f273f920640 1 -- 172.21.15.43:0/1591438513 wait complete. 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: monmap: 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.613 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 librados: starting msgr at 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 librados: starting objecter 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 Processor -- start 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- start start 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 librados: setting wanted keys 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 librados: calling monclient init 2024-04-19T00:44:40.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: init 2024-04-19T00:44:40.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: _add_conns ranks=[8,2,7,1,4,0,3,6,5] 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f27381d0450 0x7f2738137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): picked mon.noname-e con 0x7f27381d0450 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2738121f50 0x7f273813c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): picked mon.noname-b con 0x7f2738121f50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27381380d0 0x7f273813cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): picked mon.noname-i con 0x7f27381380d0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2738142130 con 0x7f2738121f50 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f27380a0780 con 0x7f27381380d0 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f27381422b0 con 0x7f27381d0450 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): authenticate will time out at 950.729000s 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f27381d0450 0x7f2738137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2738121f50 0x7f273813c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f27381d0450 0x7f2738137b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59646/0 (socket says 172.21.15.43:59646) 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 -- 172.21.15.43:0/74882211 learned_addr learned my addr 172.21.15.43:0/74882211 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): get_auth_request con 0x7f2738121f50 auth_method 0 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): get_auth_request con 0x7f27381d0450 auth_method 0 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27381380d0 0x7f273813cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): get_auth_request con 0x7f27381380d0 auth_method 0 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f27381d0450 msgr2=0x7f2738137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f27381d0450 0x7f2738137b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27381380d0 msgr2=0x7f273813cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f27381380d0 0x7f273813cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 monclient: found mon.noname-b 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2738123250 con 0x7f2738121f50 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f27381d0450 0x7f2738137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273cebb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2738121f50 0x7f273813c9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2728001ec0 tx=0x7f2724005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 client.?.objecter ms_handle_connect 0x7f2738121f50 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2728007b20 con 0x7f2738121f50 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: dump: 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.625 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2728000ee0 con 0x7f2738121f50 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 5 monclient: authenticate success, global_id 4403 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: _renew_subs 2024-04-19T00:44:40.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2738123530 con 0x7f2738121f50 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2728007790 con 0x7f2738121f50 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: dump: 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.627 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272f7fe640 4 set_mon_vals no callback set 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: _renew_subs 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f273813e580 con 0x7f2738121f50 2024-04-19T00:44:40.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 librados: init done 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 0x7f2738139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f273806d140 con 0x7f273813d420 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 0x7f2738139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient: get_auth_request con 0x7f273813d420 auth_method 0 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2728007320 con 0x7f2738121f50 2024-04-19T00:44:40.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f273809fc90 con 0x7f273813d420 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f271c038fe0 0x7f271c03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273d6bc640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 0x7f2738139b60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2734002180 tx=0x7f273400f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 client.4403.objecter ms_handle_connect 0x7f273813d420 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 client.4403.objecter resend_mon_ops 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f27280aa030 con 0x7f2738121f50 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 client.4403.objecter ms_dispatch 0x7f2738000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 3 client.4403.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 3 client.4403.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 20 client.4403.objecter dump_active .. 0 homeless 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2734003980 con 0x7f273813d420 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:40.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: dump: 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.631 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f271c038fe0 0x7f271c03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f273debd640 10 monclient: get_auth_request con 0x7f271c038fe0 auth_method 0 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2734006b60 con 0x7f273813d420 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 msgr2=0x7f2738139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 0x7f2738139b60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2734002180 tx=0x7f273400f5a0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 msgr2=0x7f2738139b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.600+0000 7f272dffb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813d420 0x7f2738139b60 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2734002180 tx=0x7f273400f5a0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.604+0000 7f272dffb640 10 client.4403.objecter ms_handle_connect 0x7f271c038fe0 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.604+0000 7f273debd640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f271c038fe0 0x7f271c03b4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2738139ac0 tx=0x7f2730008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 0x7f273813a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f273809f030 con 0x7f273813a0a0 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 0x7f273813a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient: get_auth_request con 0x7f273813a0a0 auth_method 0 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 1 -- 172.21.15.43:0/74882211 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f273813d800 con 0x7f273813a0a0 2024-04-19T00:44:40.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 client.4403.objecter ms_handle_connect 0x7f273813a0a0 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 client.4403.objecter resend_mon_ops 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273d6bc640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 0x7f273813a4b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f273400f5e0 tx=0x7f27340014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f273401fcb0 con 0x7f273813a0a0 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 monclient: dump: 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.634 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f2734005110 con 0x7f273813a0a0 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 msgr2=0x7f273813a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 0x7f273813a4b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f273400f5e0 tx=0x7f27340014b0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 msgr2=0x7f273813a4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f272dffb640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f273813a0a0 0x7f273813a4b0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f273400f5e0 tx=0x7f27340014b0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24103, 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:40.636 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 18, 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:40.637 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:40.638 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:40.639 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:40.640 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:40.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:40.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:40.648 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:40.651 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:40.653 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:40.656 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.657 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:40.658 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 10 librados: watch_flush enter 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 10 librados: watch_flush exit 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 20 client.4403.objecter shutdown clearing up homeless session... 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 10 client.4403.objecter successfully canceled tick 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f271c038fe0 msgr2=0x7f271c03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f271c038fe0 0x7f271c03b4a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2738139ac0 tx=0x7f2730008040 comp rx=0 tx=0).stop 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 10 monclient: shutdown 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2738121f50 msgr2=0x7f273813c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 --2- 172.21.15.43:0/74882211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2738121f50 0x7f273813c9a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2728001ec0 tx=0x7f2724005840 comp rx=0 tx=0).stop 2024-04-19T00:44:40.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273debd640 1 -- 172.21.15.43:0/74882211 reap_dead start 2024-04-19T00:44:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273debd640 1 -- 172.21.15.43:0/74882211 reap_dead start 2024-04-19T00:44:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 shutdown_connections 2024-04-19T00:44:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 >> 172.21.15.43:0/74882211 conn(0x7f27380a7cb0 msgr2=0x7f273809ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 shutdown_connections 2024-04-19T00:44:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 -- 172.21.15.43:0/74882211 wait complete. 2024-04-19T00:44:40.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.608+0000 7f273f920640 1 librados: shutdown 2024-04-19T00:44:40.661 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:40.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:44:40.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:44:40.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient: get_monmap_and_config 2024-04-19T00:44:40.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient: monmap: 2024-04-19T00:44:40.778 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.778 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.778 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:40.779379+0000 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:40.779379+0000 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.779 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:40.780 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 Processor -- start 2024-04-19T00:44:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 -- start start 2024-04-19T00:44:40.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient: init 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient: _add_conns ranks=[7,5,6,4,2,1,3,0,10,9,8] 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f01ac132620 0x7f01ac072f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): picked mon.noname-i con 0x7f01ac132620 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01ac132a00 0x7f01ac0a3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): picked mon.noname-h con 0x7f01ac132a00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 0x7f01ac0a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): picked mon.noname-d con 0x7f01ac132e20 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f01ac1321d0 con 0x7f01ac132a00 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f01ac0720d0 con 0x7f01ac132e20 2024-04-19T00:44:40.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f01ac0723b0 con 0x7f01ac132620 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01b2215640 10 monclient(hunting): authenticate will time out at 950.906414s 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 0x7f01ac0a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 0x7f01ac0a38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42232/0 (socket says 172.21.15.43:42232) 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 -- 172.21.15.43:0/342325266 learned_addr learned my addr 172.21.15.43:0/342325266 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): get_auth_request con 0x7f01ac132e20 auth_method 0 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f01ac132620 0x7f01ac072f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01ab7fe640 10 monclient(hunting): get_auth_request con 0x7f01ac132620 auth_method 0 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01a3fff640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01ac132a00 0x7f01ac0a3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01a3fff640 10 monclient(hunting): get_auth_request con 0x7f01ac132a00 auth_method 0 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01a3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 -- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f01ac132620 msgr2=0x7f01ac072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f01ac132620 0x7f01ac072f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 -- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01ac132a00 msgr2=0x7f01ac0a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01ac132a00 0x7f01ac0a3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 monclient: found mon.noname-d 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.776+0000 7f01abfff640 1 -- 172.21.15.43:0/342325266 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac072690 con 0x7f01ac132e20 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 0x7f01ac0a38b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f01a400d180 tx=0x7f01a401eb40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6e7671748a4d22ba server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 1 -- 172.21.15.43:0/342325266 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01a40a7740 con 0x7f01ac132e20 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: dump: 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.793 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 5 monclient: authenticate success, global_id 4281 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 1 -- 172.21.15.43:0/342325266 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a40a7900 con 0x7f01ac132e20 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 4 set_mon_vals no callback set 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 1 -- 172.21.15.43:0/342325266 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01a400b0d0 con 0x7f01ac132e20 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01aaffd640 10 monclient: dump: 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.795 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.796 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: shutdown 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- 172.21.15.43:0/342325266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 msgr2=0x7f01ac0a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 0x7f01ac0a38b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f01a400d180 tx=0x7f01a401eb40 comp rx=0 tx=0).stop 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- 172.21.15.43:0/342325266 shutdown_connections 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01ac132e20 0x7f01ac0a38b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01ac132a00 0x7f01ac0a3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- 172.21.15.43:0/342325266 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f01ac132620 0x7f01ac072f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- 172.21.15.43:0/342325266 >> 172.21.15.43:0/342325266 conn(0x7f01ac0a4460 msgr2=0x7f01ac0a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- 172.21.15.43:0/342325266 shutdown_connections 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- 172.21.15.43:0/342325266 wait complete. 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: monmap: 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.798 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 librados: starting msgr at 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 librados: starting objecter 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 Processor -- start 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- start start 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 librados: setting wanted keys 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 librados: calling monclient init 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: init 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient: _add_conns ranks=[1,3,4,7,0,8,2,6,5] 2024-04-19T00:44:40.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01ac132620 0x7f01ac1d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): picked mon.noname-f con 0x7f01ac132620 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01ac132a00 0x7f01ac13b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): picked mon.noname-g con 0x7f01ac132a00 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01ac132e20 0x7f01ac13ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): picked mon.noname-c con 0x7f01ac132e20 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01ac070560 con 0x7f01ac132620 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f01ac0703e0 con 0x7f01ac132a00 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f01ac0706e0 con 0x7f01ac132e20 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01b2215640 10 monclient(hunting): authenticate will time out at 950.911316s 2024-04-19T00:44:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01ac132e20 0x7f01ac13ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01ac132e20 0x7f01ac13ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:48646/0 (socket says 172.21.15.43:48646) 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 -- 172.21.15.43:0/2385816881 learned_addr learned my addr 172.21.15.43:0/2385816881 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): get_auth_request con 0x7f01ac132e20 auth_method 0 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01ac132620 0x7f01ac1d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01a3fff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01ac132a00 0x7f01ac13b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2024-04-19T00:44:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01ac132a00 msgr2=0x7f01ac13b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01ac132a00 0x7f01ac13b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01ac132620 msgr2=0x7f01ac1d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01ac132620 0x7f01ac1d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 monclient: found mon.noname-c 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01ac138f80 con 0x7f01ac132e20 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01a3fff640 10 monclient: get_auth_request con 0x7f01ac132a00 auth_method 0 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01ab7fe640 10 monclient: get_auth_request con 0x7f01ac132620 auth_method 0 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01a3fff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01ac132a00 0x7f01ac13b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01ac132620 0x7f01ac1d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:40.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01abfff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01ac132e20 0x7f01ac13ba00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f01a400b040 tx=0x7f01a401b940 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01a8ff9640 10 client.?.objecter ms_handle_connect 0x7f01ac132e20 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.780+0000 7f01a8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01a40a7740 con 0x7f01ac132e20 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: dump: 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.803 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a40be070 con 0x7f01ac132e20 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01a4003e10 con 0x7f01ac132e20 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01aa7fc640 4 set_mon_vals no callback set 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: dump: 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 5 monclient: authenticate success, global_id 4369 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 monclient: _renew_subs 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01ac0712c0 con 0x7f01ac132e20 2024-04-19T00:44:40.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 monclient: _renew_subs 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f01ac1386e0 con 0x7f01ac132e20 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 1 librados: init done 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f01a400b2e0 con 0x7f01ac132e20 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 0x7f01ac09e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f01ac06d140 con 0x7f01ac1377f0 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 0x7f01ac09e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient: get_auth_request con 0x7f01ac1377f0 auth_method 0 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f018c038e10 0x7f018c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f01a40e3fd0 con 0x7f01ac132e20 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 client.4369.objecter ms_dispatch 0x7f01ac000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 3 client.4369.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 3 client.4369.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 20 client.4369.objecter dump_active .. 0 homeless 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01ac06dab0 con 0x7f01ac1377f0 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a3fff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f018c038e10 0x7f018c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a3fff640 10 monclient: get_auth_request con 0x7f018c038e10 auth_method 0 2024-04-19T00:44:40.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 client.4369.objecter ms_handle_connect 0x7f01ac1377f0 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 client.4369.objecter resend_mon_ops 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 0x7f01ac09e7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e530 tx=0x7f019c00e9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f019c017020 con 0x7f01ac1377f0 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: dump: 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.809 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f019c004030 con 0x7f01ac1377f0 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 msgr2=0x7f01ac09e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 0x7f01ac09e7b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e530 tx=0x7f019c00e9f0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 msgr2=0x7f01ac09e7b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:40.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac1377f0 0x7f01ac09e7b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a8ff9640 10 client.4369.objecter ms_handle_connect 0x7f018c038e10 2024-04-19T00:44:40.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.784+0000 7f01a3fff640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f018c038e10 0x7f018c03b2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f01ac09fed0 tx=0x7f0194008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:40.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 0x7f01ac09f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f01ac1d17f0 con 0x7f01ac09ecf0 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 0x7f01ac09f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient: get_auth_request con 0x7f01ac09ecf0 auth_method 0 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 1 -- 172.21.15.43:0/2385816881 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01ac09f640 con 0x7f01ac09ecf0 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01ab7fe640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 0x7f01ac09f100 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e530 tx=0x7f019c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 client.4369.objecter ms_handle_connect 0x7f01ac09ecf0 2024-04-19T00:44:40.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 client.4369.objecter resend_mon_ops 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f019c017040 con 0x7f01ac09ecf0 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 monclient: dump: 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.814 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f019c003d90 con 0x7f01ac09ecf0 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 msgr2=0x7f01ac09f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 0x7f01ac09f100 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e530 tx=0x7f019c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 msgr2=0x7f01ac09f100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:40.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01a8ff9640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f01ac09ecf0 0x7f01ac09f100 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f019c00e530 tx=0x7f019c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24291, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:40.816 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 18, 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:40.817 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:40.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:40.819 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:40.820 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:40.821 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.822 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.824 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:40.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:40.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:40.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:40.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:40.842 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:40.844 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:40.845 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:40.846 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 10 librados: watch_flush enter 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 10 librados: watch_flush exit 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 20 client.4369.objecter shutdown clearing up homeless session... 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 10 client.4369.objecter successfully canceled tick 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f018c038e10 msgr2=0x7f018c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f018c038e10 0x7f018c03b2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f01ac09fed0 tx=0x7f0194008040 comp rx=0 tx=0).stop 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 10 monclient: shutdown 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01ac132e20 msgr2=0x7f01ac13ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 --2- 172.21.15.43:0/2385816881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01ac132e20 0x7f01ac13ba00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f01a400b040 tx=0x7f01a401b940 comp rx=0 tx=0).stop 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01abfff640 1 -- 172.21.15.43:0/2385816881 reap_dead start 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01abfff640 1 -- 172.21.15.43:0/2385816881 reap_dead start 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 shutdown_connections 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 >> 172.21.15.43:0/2385816881 conn(0x7f01ac0a4460 msgr2=0x7f01ac12e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 shutdown_connections 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 -- 172.21.15.43:0/2385816881 wait complete. 2024-04-19T00:44:40.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.792+0000 7f01b2215640 1 librados: shutdown 2024-04-19T00:44:40.849 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:40.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:44:40.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:44:40.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.960+0000 7fa6d993f640 10 monclient: get_monmap_and_config 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.960+0000 7fa6d993f640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.960+0000 7fa6d993f640 10 monclient: monmap: 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:40.966374+0000 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:40.966374+0000 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:40.965 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:40.966 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 Processor -- start 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 -- start start 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient: init 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient: _add_conns ranks=[3,8,1,10,4,5,7,6,0,2,9] 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6d4138a50 0x7fa6d4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): picked mon.noname-g con 0x7fa6d4138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 0x7fa6d4139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): picked mon.noname-e con 0x7fa6d40a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa6d4137960 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6d40a0d40 con 0x7fa6d4137960 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa6d40a1020 con 0x7fa6d4138a50 2024-04-19T00:44:40.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa6d41398b0 con 0x7fa6d40a5bb0 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d993f640 10 monclient(hunting): authenticate will time out at 951.093542s 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 0x7fa6d4139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 0x7fa6d4139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:55822/0 (socket says 172.21.15.43:55822) 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 -- 172.21.15.43:0/1198328854 learned_addr learned my addr 172.21.15.43:0/1198328854 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): get_auth_request con 0x7fa6d40a5bb0 auth_method 0 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d2ffd640 1 --2- 172.21.15.43:0/1198328854 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6d4138a50 0x7fa6d4138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d2ffd640 10 monclient(hunting): get_auth_request con 0x7fa6d4138a50 auth_method 0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_done global_id 4472 payload 274 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 -- 172.21.15.43:0/1198328854 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6d4138a50 msgr2=0x7fa6d4138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 --2- 172.21.15.43:0/1198328854 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6d4138a50 0x7fa6d4138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 -- 172.21.15.43:0/1198328854 >> v1:172.21.15.161:6789/0 conn(0x7fa6d4137960 legacy=0x7fa6d4137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 monclient: found mon.noname-e 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.964+0000 7fa6d27fc640 1 -- 172.21.15.43:0/1198328854 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d40a0a60 con 0x7fa6d40a5bb0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 1 --2- 172.21.15.43:0/1198328854 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 0x7fa6d4139370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80098a0 tx=0x7fa6c801c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e415f002a82c6451 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 1 -- 172.21.15.43:0/1198328854 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6c80b1070 con 0x7fa6d40a5bb0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: dump: 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.986 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 5 monclient: authenticate success, global_id 4472 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 1 -- 172.21.15.43:0/1198328854 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6c80aa340 con 0x7fa6d40a5bb0 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 1 -- 172.21.15.43:0/1198328854 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6c80aa9e0 con 0x7fa6d40a5bb0 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d1ffb640 10 monclient: dump: 2024-04-19T00:44:40.987 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.988 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 4 set_mon_vals no callback set 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: shutdown 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- 172.21.15.43:0/1198328854 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 msgr2=0x7fa6d4139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 --2- 172.21.15.43:0/1198328854 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 0x7fa6d4139370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80098a0 tx=0x7fa6c801c440 comp rx=0 tx=0).stop 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- 172.21.15.43:0/1198328854 shutdown_connections 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 --2- 172.21.15.43:0/1198328854 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6d40a5bb0 0x7fa6d4139370 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 --2- 172.21.15.43:0/1198328854 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa6d4138a50 0x7fa6d4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- 172.21.15.43:0/1198328854 >> 172.21.15.43:0/1198328854 conn(0x7fa6d40a7cb0 msgr2=0x7fa6d40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- 172.21.15.43:0/1198328854 shutdown_connections 2024-04-19T00:44:40.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- 172.21.15.43:0/1198328854 wait complete. 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: build_initial_monmap 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: monmap: 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:40.990 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 librados: starting msgr at 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 librados: starting objecter 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 Processor -- start 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- start start 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 librados: setting wanted keys 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 librados: calling monclient init 2024-04-19T00:44:40.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: init 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient: _add_conns ranks=[7,4,6,3,2,8,5,1,0] 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa6d40a5bb0 0x7fa6d4136210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): picked mon.noname-i con 0x7fa6d40a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d4137960 0x7fa6d4136750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): picked mon.noname-c con 0x7fa6d4137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa6d4138a50 0x7fa6d413c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): picked mon.noname-d con 0x7fa6d4138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa6d4143bb0 con 0x7fa6d4137960 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa6d4143a30 con 0x7fa6d4138a50 2024-04-19T00:44:40.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa6d4143d30 con 0x7fa6d40a5bb0 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d993f640 10 monclient(hunting): authenticate will time out at 951.098722s 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa6d4138a50 0x7fa6d413c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa6d4138a50 0x7fa6d413c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39086/0 (socket says 172.21.15.43:39086) 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 -- 172.21.15.43:0/619856128 learned_addr learned my addr 172.21.15.43:0/619856128 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): get_auth_request con 0x7fa6d4138a50 auth_method 0 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d4137960 0x7fa6d4136750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): get_auth_request con 0x7fa6d4137960 auth_method 0 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:40.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d2ffd640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa6d40a5bb0 0x7fa6d4136210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d2ffd640 10 monclient(hunting): get_auth_request con 0x7fa6d40a5bb0 auth_method 0 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa6d40a5bb0 msgr2=0x7fa6d4136210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa6d40a5bb0 0x7fa6d4136210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d4137960 msgr2=0x7fa6d4136750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d4137960 0x7fa6d4136750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 monclient: found mon.noname-d 2024-04-19T00:44:40.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d4140590 con 0x7fa6d4138a50 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d27fc640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d4137960 0x7fa6d4136750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d37fe640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa6d4138a50 0x7fa6d413c600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00c7f0 tx=0x7fa6cc00ccb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 client.?.objecter ms_handle_connect 0x7fa6d4138a50 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6cc001880 con 0x7fa6d4138a50 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: dump: 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.996 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: _finish_auth 0 2024-04-19T00:44:40.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6cc003df0 con 0x7fa6d4138a50 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6cc00acc0 con 0x7fa6d4138a50 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6bbfff640 10 monclient: dump: 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:40.998 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:40.999 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.968+0000 7fa6d17fa640 4 set_mon_vals no callback set 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 5 monclient: authenticate success, global_id 4290 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 monclient: _renew_subs 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d409fc90 con 0x7fa6d4138a50 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 monclient: _renew_subs 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d413e2a0 con 0x7fa6d4138a50 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 1 librados: init done 2024-04-19T00:44:41.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 0x7fa6d413cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa6d406d140 con 0x7fa6d413cb40 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa6cc00a850 con 0x7fa6d4138a50 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 0x7fa6d413cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient: get_auth_request con 0x7fa6d413cb40 auth_method 0 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6c0038e10 0x7fa6c003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fa6cc048b00 con 0x7fa6d4138a50 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 client.4290.objecter ms_dispatch 0x7fa6d4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 3 client.4290.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 3 client.4290.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 20 client.4290.objecter dump_active .. 0 homeless 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6d41409b0 con 0x7fa6d413cb40 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d27fc640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6c0038e10 0x7fa6c003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 client.4290.objecter ms_handle_connect 0x7fa6d413cb40 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 client.4290.objecter resend_mon_ops 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d2ffd640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 0x7fa6d413cf20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400f410 tx=0x7fa6c400f8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d27fc640 10 monclient: get_auth_request con 0x7fa6c0038e10 auth_method 0 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6c4019020 con 0x7fa6d413cb40 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 monclient: dump: 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.003 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6c4014670 con 0x7fa6d413cb40 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 msgr2=0x7fa6d413cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 0x7fa6d413cf20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400f410 tx=0x7fa6c400f8d0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 msgr2=0x7fa6d413cf20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413cb40 0x7fa6d413cf20 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa6c400f410 tx=0x7fa6c400f8d0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6bbfff640 10 client.4290.objecter ms_handle_connect 0x7fa6c0038e10 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.972+0000 7fa6d27fc640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6c0038e10 0x7fa6c003b2d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80ae000 tx=0x7fa6c80a6070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 0x7fa6d409c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa6d4076190 con 0x7fa6d413d460 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 0x7fa6d409c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient: get_auth_request con 0x7fa6d413d460 auth_method 0 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 10 monclient(hunting): handle_auth_done global_id 4396 payload 824 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 1 -- 172.21.15.43:0/619856128 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6d413ee00 con 0x7fa6d413d460 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d2ffd640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 0x7fa6d409c230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa6c401a000 tx=0x7fa6c40046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 client.4290.objecter ms_handle_connect 0x7fa6d413d460 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 client.4290.objecter resend_mon_ops 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6c4019040 con 0x7fa6d413d460 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:41.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 monclient: dump: 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.007 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fa6c4014e20 con 0x7fa6d413d460 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 msgr2=0x7fa6d409c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 0x7fa6d409c230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa6c401a000 tx=0x7fa6c40046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 msgr2=0x7fa6d409c230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6bbfff640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6d413d460 0x7fa6d409c230 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa6c401a000 tx=0x7fa6c40046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.008 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24499, 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:41.009 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 19, 2024-04-19T00:44:41.010 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.011 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:41.012 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.013 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:41.014 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:41.015 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.016 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.017 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.020 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.026 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.031 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.039 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:41.040 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:41.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 10 librados: watch_flush enter 2024-04-19T00:44:41.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 10 librados: watch_flush exit 2024-04-19T00:44:41.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 20 client.4290.objecter shutdown clearing up homeless session... 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 10 client.4290.objecter successfully canceled tick 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6c0038e10 msgr2=0x7fa6c003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6c0038e10 0x7fa6c003b2d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6c80ae000 tx=0x7fa6c80a6070 comp rx=0 tx=0).stop 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 10 monclient: shutdown 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa6d4138a50 msgr2=0x7fa6d413c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 --2- 172.21.15.43:0/619856128 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa6d4138a50 0x7fa6d413c600 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc00c7f0 tx=0x7fa6cc00ccb0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d37fe640 1 -- 172.21.15.43:0/619856128 reap_dead start 2024-04-19T00:44:41.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d37fe640 1 -- 172.21.15.43:0/619856128 reap_dead start 2024-04-19T00:44:41.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 shutdown_connections 2024-04-19T00:44:41.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 >> 172.21.15.43:0/619856128 conn(0x7fa6d40a7cb0 msgr2=0x7fa6d40a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 shutdown_connections 2024-04-19T00:44:41.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 -- 172.21.15.43:0/619856128 wait complete. 2024-04-19T00:44:41.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:40.980+0000 7fa6d993f640 1 librados: shutdown 2024-04-19T00:44:41.044 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:41.044 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:44:41.044 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:44:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.160+0000 7f3dba649640 10 monclient: get_monmap_and_config 2024-04-19T00:44:41.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.160+0000 7f3dba649640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.160+0000 7f3dba649640 10 monclient: monmap: 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:41.166199+0000 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:41.166199+0000 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.165 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:41.166 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:41.166 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 Processor -- start 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 -- start start 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient: init 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient: _add_conns ranks=[1,4,6,8,2,10,3,5,7,9,0] 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3db40a4b10 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 0x7f3db40a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): picked mon.noname-c con 0x7f3db40a4ef0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 0x7f3db4140560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): picked mon.noname-d con 0x7f3db413e170 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3db41398f0 con 0x7f3db40a4b10 2024-04-19T00:44:41.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f3db406e750 con 0x7f3db40a4ef0 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3db406f250 con 0x7f3db413e170 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3dba649640 10 monclient(hunting): authenticate will time out at 951.293276s 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 0x7f3db40a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 0x7f3db4140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 0x7f3db4140560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39092/0 (socket says 172.21.15.43:39092) 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 0x7f3db40a5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:40996/0 (socket says 172.21.15.43:40996) 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 1 -- 172.21.15.43:0/3446523584 learned_addr learned my addr 172.21.15.43:0/3446523584 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): get_auth_request con 0x7f3db413e170 auth_method 0 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): get_auth_request con 0x7f3db40a4ef0 auth_method 0 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 -- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 msgr2=0x7f3db4140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 --2- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 0x7f3db4140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 -- 172.21.15.43:0/3446523584 >> v1:172.21.15.161:6789/0 conn(0x7f3db40a4b10 legacy=0x7f3db41391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 monclient: found mon.noname-c 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 -- 172.21.15.43:0/3446523584 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db406f530 con 0x7f3db40a4ef0 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db8be6640 1 --2- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 0x7f3db4140560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.164+0000 7f3db37fe640 1 --2- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 0x7f3db40a5370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3da40097f0 tx=0x7f3da401c3a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=8aaf1abbb14e067f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 1 -- 172.21.15.43:0/3446523584 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3da40b0070 con 0x7f3db40a4ef0 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: dump: 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.172 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.173 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 5 monclient: authenticate success, global_id 4405 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 1 -- 172.21.15.43:0/3446523584 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da40aa810 con 0x7f3db40a4ef0 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 1 -- 172.21.15.43:0/3446523584 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3da40a9300 con 0x7f3db40a4ef0 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 4 set_mon_vals no callback set 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db2ffd640 10 monclient: dump: 2024-04-19T00:44:41.174 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.175 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.176 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: shutdown 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 msgr2=0x7f3db40a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 --2- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 0x7f3db40a5370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3da40097f0 tx=0x7f3da401c3a0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- 172.21.15.43:0/3446523584 shutdown_connections 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 --2- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db413e170 0x7f3db4140560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 --2- 172.21.15.43:0/3446523584 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3db40a4ef0 0x7f3db40a5370 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- 172.21.15.43:0/3446523584 >> 172.21.15.43:0/3446523584 conn(0x7f3db40a98c0 msgr2=0x7f3db40a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- 172.21.15.43:0/3446523584 shutdown_connections 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- 172.21.15.43:0/3446523584 wait complete. 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: monmap: 2024-04-19T00:44:41.185 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.186 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 librados: starting msgr at 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 librados: starting objecter 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 Processor -- start 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- start start 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 librados: setting wanted keys 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 librados: calling monclient init 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: init 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient: _add_conns ranks=[5,7,3,1,4,0,2,6,8] 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3db40a4b10 0x7f3db413bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): picked mon.noname-h con 0x7f3db40a4b10 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:41.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3db40a4ef0 0x7f3db411fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): picked mon.noname-i con 0x7f3db40a4ef0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3db413e170 0x7f3db4136b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): picked mon.noname-g con 0x7f3db413e170 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3db4143650 con 0x7f3db413e170 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f3db41434d0 con 0x7f3db40a4b10 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3db41437d0 con 0x7f3db40a4ef0 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3dba649640 10 monclient(hunting): authenticate will time out at 951.298464s 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3db40a4ef0 0x7f3db411fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3db40a4ef0 0x7f3db411fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:39178/0 (socket says 172.21.15.43:39178) 2024-04-19T00:44:41.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 -- 172.21.15.43:0/3403863470 learned_addr learned my addr 172.21.15.43:0/3403863470 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): get_auth_request con 0x7f3db40a4ef0 auth_method 0 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3db413e170 0x7f3db4136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3db40a4b10 0x7f3db413bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): get_auth_request con 0x7f3db413e170 auth_method 0 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): get_auth_request con 0x7f3db40a4b10 auth_method 0 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db8be6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient(hunting): handle_auth_done global_id 4453 payload 824 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3db40a4b10 msgr2=0x7f3db413bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3db40a4b10 0x7f3db413bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3db413e170 msgr2=0x7f3db4136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3db413e170 0x7f3db4136b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 monclient: found mon.noname-i 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db406e920 con 0x7f3db40a4ef0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db37fe640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3db40a4ef0 0x7f3db411fcf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3da401aa50 tx=0x7f3da4019140 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db0ff9640 10 client.?.objecter ms_handle_connect 0x7f3db40a4ef0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.168+0000 7f3db0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3da40b0040 con 0x7f3db40a4ef0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:44:41.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: dump: 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.192 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da40044b0 con 0x7f3db40a4ef0 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3da40aaca0 con 0x7f3db40a4ef0 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: dump: 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.193 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db27fc640 4 set_mon_vals no callback set 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 5 monclient: authenticate success, global_id 4453 2024-04-19T00:44:41.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 monclient: _renew_subs 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db413d5f0 con 0x7f3db40a4ef0 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 monclient: _renew_subs 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db4120c50 con 0x7f3db40a4ef0 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 1 librados: init done 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 0x7f3db41d0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3db406d400 con 0x7f3db41370a0 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 0x7f3db41d0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient: get_auth_request con 0x7f3db41370a0 auth_method 0 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3db4069630 con 0x7f3db41370a0 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 client.4453.objecter ms_handle_connect 0x7f3db41370a0 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 client.4453.objecter resend_mon_ops 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db3fff640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 0x7f3db41d0850 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3da800f770 tx=0x7f3da800fc30 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3da800d700 con 0x7f3db41370a0 2024-04-19T00:44:41.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: dump: 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.197 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.7 v2:172.21.15.161:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3da4004040 con 0x7f3db40a4ef0 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3da0039080 0x7f3da003b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.7 v2:172.21.15.161:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f3da40a4e00 con 0x7f3db40a4ef0 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 client.4453.objecter ms_dispatch 0x7f3db4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 3 client.4453.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 3 client.4453.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 20 client.4453.objecter dump_active .. 0 homeless 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3da80050d0 con 0x7f3db41370a0 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 msgr2=0x7f3db41d0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 0x7f3db41d0850 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3da800f770 tx=0x7f3da800fc30 comp rx=0 tx=0).stop 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 msgr2=0x7f3db41d0850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41370a0 0x7f3db41d0850 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db8be6640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3da0039080 0x7f3da003b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db8be6640 10 monclient: get_auth_request con 0x7f3da0039080 auth_method 0 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db0ff9640 10 client.4453.objecter ms_handle_connect 0x7f3da0039080 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.172+0000 7f3db8be6640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3da0039080 0x7f3da003b540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3db41219e0 tx=0x7f3dac00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 0x7f3db41d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3db41d16e0 con 0x7f3db41d0d90 2024-04-19T00:44:41.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 0x7f3db41d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient: get_auth_request con 0x7f3db41d0d90 auth_method 0 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 1 -- 172.21.15.43:0/3403863470 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3db4071d30 con 0x7f3db41d0d90 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 client.4453.objecter ms_handle_connect 0x7f3db41d0d90 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db3fff640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 0x7f3db41d11a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3da800f770 tx=0x7f3da8000ee0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 client.4453.objecter resend_mon_ops 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3da801dc90 con 0x7f3db41d0d90 2024-04-19T00:44:41.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 monclient: dump: 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.201 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f3da8003600 con 0x7f3db41d0d90 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 msgr2=0x7f3db41d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 0x7f3db41d11a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3da800f770 tx=0x7f3da8000ee0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 msgr2=0x7f3db41d11a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db0ff9640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3db41d0d90 0x7f3db41d11a0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3da800f770 tx=0x7f3da8000ee0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24695, 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:41.203 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 19, 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.204 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.205 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:41.206 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:41.207 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.208 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.209 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.211 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.214 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.216 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:41.218 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.220 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.221 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.222 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:41.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:41.224 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 10 librados: watch_flush enter 2024-04-19T00:44:41.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 10 librados: watch_flush exit 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 20 client.4453.objecter shutdown clearing up homeless session... 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 10 client.4453.objecter successfully canceled tick 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3da0039080 msgr2=0x7f3da003b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3da0039080 0x7f3da003b540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3db41219e0 tx=0x7f3dac00b040 comp rx=0 tx=0).stop 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 10 monclient: shutdown 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3db40a4ef0 msgr2=0x7f3db411fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 --2- 172.21.15.43:0/3403863470 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3db40a4ef0 0x7f3db411fcf0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3da401aa50 tx=0x7f3da4019140 comp rx=0 tx=0).stop 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db8be6640 1 -- 172.21.15.43:0/3403863470 reap_dead start 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3db8be6640 1 -- 172.21.15.43:0/3403863470 reap_dead start 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 shutdown_connections 2024-04-19T00:44:41.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 >> 172.21.15.43:0/3403863470 conn(0x7f3db40a98c0 msgr2=0x7f3db40a88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 shutdown_connections 2024-04-19T00:44:41.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 -- 172.21.15.43:0/3403863470 wait complete. 2024-04-19T00:44:41.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.180+0000 7f3dba649640 1 librados: shutdown 2024-04-19T00:44:41.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:41.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:44:41.227 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:44:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient: get_monmap_and_config 2024-04-19T00:44:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient: monmap: 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:41.351880+0000 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:41.351880+0000 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.351 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.352 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.352 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:41.352 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:41.352 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 1 Processor -- start 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 1 -- start start 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient: init 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient: _add_conns ranks=[0,7,1,6,8,5,4,9,2,10,3] 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcd60138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 0x7fcd6013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient(hunting): picked mon.noname-i con 0x7fcd60139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fcd600a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd600a0d40 con 0x7fcd60138a50 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd600a0a60 con 0x7fcd600a5bb0 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fcd600a1020 con 0x7fcd60139540 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd6874a640 10 monclient(hunting): authenticate will time out at 951.479218s 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd664e6640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fcd60138a50 0x7fcd60138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:34704/0 (socket says 172.21.15.43:34704) 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd664e6640 1 -- 172.21.15.43:0/3139029284 learned_addr learned my addr 172.21.15.43:0/3139029284 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 0x7fcd6013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request con 0x7fcd60139540 auth_method 0 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2640186189 0 0) 0x7fcd600a0d40 con 0x7fcd60138a50 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 10 monclient(hunting): _negotiate 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 10 monclient(hunting): my global_id is 4473 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd4c003530 con 0x7fcd60138a50 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.348+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1414259982 0 0) 0x7fcd4c003530 con 0x7fcd60138a50 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 msgr2=0x7fcd6013dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 0x7fcd6013dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 >> v1:172.21.15.161:6789/0 conn(0x7fcd600a5bb0 legacy=0x7fcd60140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd600a17b0 con 0x7fcd60138a50 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 5 monclient: authenticate success, global_id 4473 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fcd54001970 con 0x7fcd60138a50 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: dump: 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.366 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.367 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 >> v1:172.21.15.43:6789/0 conn(0x7fcd60138a50 legacy=0x7fcd60138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: _add_conns ranks=[7,5,8,0,1,6,4,3,2] 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 0x7fcd4c004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): picked mon.i con 0x7fcd60139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcd4c004860 0x7fcd4c004ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): picked mon.h con 0x7fcd4c004860 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd4c0099c0 0x7fcd4c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): picked mon.e con 0x7fcd4c0099c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fcd600a1020 con 0x7fcd4c004860 2024-04-19T00:44:41.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fcd4c00c450 con 0x7fcd60139540 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcd4c00c600 con 0x7fcd4c0099c0 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd4c0099c0 0x7fcd4c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): get_auth_request con 0x7fcd4c0099c0 auth_method 0 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 0x7fcd4c004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): get_auth_request con 0x7fcd60139540 auth_method 0 2024-04-19T00:44:41.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcd4c004860 0x7fcd4c004ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request con 0x7fcd4c004860 auth_method 0 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd65ce5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd65ce5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd664e6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient(hunting): handle_auth_done global_id 4473 payload 293 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 -- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 msgr2=0x7fcd4c004240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd60139540 0x7fcd4c004240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 -- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcd4c004860 msgr2=0x7fcd4c004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcd4c004860 0x7fcd4c004ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 monclient: found mon.e 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 -- 172.21.15.43:0/3139029284 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4c003530 con 0x7fcd4c0099c0 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcd4c004860 0x7fcd4c004ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd4c0099c0 0x7fcd4c00bdb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd58001840 tx=0x7fcd5801c500 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f176788f6bf411cc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd580b0070 con 0x7fcd4c0099c0 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: dump: 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.372 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd580aa810 con 0x7fcd4c0099c0 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 4 set_mon_vals no callback set 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 1 -- 172.21.15.43:0/3139029284 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd58001460 con 0x7fcd4c0099c0 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd654e4640 10 monclient: dump: 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.374 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 10 monclient: shutdown 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 1 -- 172.21.15.43:0/3139029284 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd4c0099c0 msgr2=0x7fcd4c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 1 --2- 172.21.15.43:0/3139029284 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd4c0099c0 0x7fcd4c00bdb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcd58001840 tx=0x7fcd5801c500 comp rx=0 tx=0).stop 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd66ce7640 1 -- 172.21.15.43:0/3139029284 reap_dead start 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 1 -- 172.21.15.43:0/3139029284 shutdown_connections 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 1 -- 172.21.15.43:0/3139029284 >> 172.21.15.43:0/3139029284 conn(0x7fcd600a7cb0 msgr2=0x7fcd600a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 1 -- 172.21.15.43:0/3139029284 shutdown_connections 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.352+0000 7fcd6874a640 1 -- 172.21.15.43:0/3139029284 wait complete. 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: monmap: 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.376 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 librados: starting msgr at 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 librados: starting objecter 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 Processor -- start 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- start start 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 librados: setting wanted keys 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 librados: calling monclient init 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: init 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: _add_conns ranks=[8,4,7,1,3,5,2,0,6] 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd601d0880 0x7fcd601d0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): picked mon.noname-e con 0x7fcd601d0880 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:41.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcd601d11a0 0x7fcd6012e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): picked mon.noname-c con 0x7fcd601d11a0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd6012ee10 0x7fcd6012f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): picked mon.noname-i con 0x7fcd6012ee10 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fcd60142670 con 0x7fcd601d11a0 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fcd600a0780 con 0x7fcd6012ee10 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcd601427f0 con 0x7fcd601d0880 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): authenticate will time out at 951.485263s 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcd601d11a0 0x7fcd6012e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd601d0880 0x7fcd601d0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcd601d11a0 0x7fcd6012e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:41008/0 (socket says 172.21.15.43:41008) 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 -- 172.21.15.43:0/2635450362 learned_addr learned my addr 172.21.15.43:0/2635450362 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request con 0x7fcd601d11a0 auth_method 0 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): get_auth_request con 0x7fcd601d0880 auth_method 0 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd6012ee10 0x7fcd6012f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 10 monclient(hunting): get_auth_request con 0x7fcd6012ee10 auth_method 0 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient(hunting): handle_auth_done global_id 4481 payload 824 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd6012ee10 msgr2=0x7fcd6012f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcd6012ee10 0x7fcd6012f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcd601d11a0 msgr2=0x7fcd6012e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcd601d11a0 0x7fcd6012e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 monclient: found mon.noname-e 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd601384a0 con 0x7fcd601d0880 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcd601d11a0 0x7fcd6012e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd664e6640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd601d0880 0x7fcd601d0c60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd54007340 tx=0x7fcd54007800 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 client.?.objecter ms_handle_connect 0x7fcd601d0880 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd5401a070 con 0x7fcd601d0880 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: dump: 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.381 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.382 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd5400fe30 con 0x7fcd601d0880 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd540127c0 con 0x7fcd601d0880 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 monclient: dump: 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.383 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd64ce3640 4 set_mon_vals no callback set 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 5 monclient: authenticate success, global_id 4481 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: _renew_subs 2024-04-19T00:44:41.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd6009fc90 con 0x7fcd601d0880 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: _renew_subs 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd601402c0 con 0x7fcd601d0880 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 librados: init done 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fcd54012350 con 0x7fcd601d0880 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 0x7fcd601332d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcd6006d140 con 0x7fcd60132ed0 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 0x7fcd601332d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient: get_auth_request con 0x7fcd60132ed0 auth_method 0 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd44038e10 0x7fcd4403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fcd54010040 con 0x7fcd601d0880 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 client.4481.objecter ms_dispatch 0x7fcd60000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 3 client.4481.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 3 client.4481.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 20 client.4481.objecter dump_active .. 0 homeless 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_done global_id 4490 payload 824 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd600a0d40 con 0x7fcd60132ed0 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd44038e10 0x7fcd4403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd66ce7640 10 monclient: get_auth_request con 0x7fcd44038e10 auth_method 0 2024-04-19T00:44:41.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 client.4481.objecter ms_handle_connect 0x7fcd60132ed0 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd52ffd640 10 client.4481.objecter resend_mon_ops 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.356+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 0x7fcd601332d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c013850 tx=0x7fcd5c013c50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd5c01c070 con 0x7fcd60132ed0 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 10 monclient: dump: 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.387 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd5c013e40 con 0x7fcd60132ed0 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 msgr2=0x7fcd601332d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 0x7fcd601332d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c013850 tx=0x7fcd5c013c50 comp rx=0 tx=0).stop 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 msgr2=0x7fcd601332d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60132ed0 0x7fcd601332d0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c013850 tx=0x7fcd5c013c50 comp rx=0 tx=0).stop 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd52ffd640 10 client.4481.objecter ms_handle_connect 0x7fcd44038e10 2024-04-19T00:44:41.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.360+0000 7fcd66ce7640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd44038e10 0x7fcd4403b2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcd580ab000 tx=0x7fcd580ad040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd6874a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd6874a640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 0x7fcd601dbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd6874a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcd6013eb30 con 0x7fcd60133810 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 0x7fcd601dbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient: get_auth_request con 0x7fcd60133810 auth_method 0 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 10 monclient(hunting): handle_auth_done global_id 4499 payload 824 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 1 -- 172.21.15.43:0/2635450362 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd60076190 con 0x7fcd60133810 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd65ce5640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 0x7fcd601dbde0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c013850 tx=0x7fcd5c004ac0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 client.4481.objecter ms_handle_connect 0x7fcd60133810 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 client.4481.objecter resend_mon_ops 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd5c02d8a0 con 0x7fcd60133810 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 monclient: dump: 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.392 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7fcd5c02d430 con 0x7fcd60133810 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 msgr2=0x7fcd601dbde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 0x7fcd601dbde0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c013850 tx=0x7fcd5c004ac0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 msgr2=0x7fcd601dbde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.364+0000 7fcd52ffd640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd60133810 0x7fcd601dbde0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24875, 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:41.394 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 19, 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.395 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.396 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:41.397 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:41.398 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:41.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.402 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.404 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.405 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.407 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.408 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.410 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.411 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:41.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:41.415 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 10 librados: watch_flush enter 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 10 librados: watch_flush exit 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 20 client.4481.objecter shutdown clearing up homeless session... 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 10 client.4481.objecter successfully canceled tick 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd44038e10 msgr2=0x7fcd4403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd44038e10 0x7fcd4403b2d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcd580ab000 tx=0x7fcd580ad040 comp rx=0 tx=0).stop 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 10 monclient: shutdown 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd601d0880 msgr2=0x7fcd601d0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 --2- 172.21.15.43:0/2635450362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcd601d0880 0x7fcd601d0c60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fcd54007340 tx=0x7fcd54007800 comp rx=0 tx=0).stop 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd66ce7640 1 -- 172.21.15.43:0/2635450362 reap_dead start 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd66ce7640 1 -- 172.21.15.43:0/2635450362 reap_dead start 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 shutdown_connections 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 >> 172.21.15.43:0/2635450362 conn(0x7fcd600a7cb0 msgr2=0x7fcd60139b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 shutdown_connections 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 -- 172.21.15.43:0/2635450362 wait complete. 2024-04-19T00:44:41.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.368+0000 7fcd6874a640 1 librados: shutdown 2024-04-19T00:44:41.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:41.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:44:41.419 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:44:41.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient: get_monmap_and_config 2024-04-19T00:44:41.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient: monmap: 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:41.539206+0000 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:41.539206+0000 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.538 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.539 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.539 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.539 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.539 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:41.539 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:41.539 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 Processor -- start 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 -- start start 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient: init 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient: _add_conns ranks=[4,5,7,1,6,3,9,0,10,2,8] 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 0x7f4518135f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): picked mon.noname-c con 0x7f4518135b70 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4518136490 0x7f45180a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): picked mon.noname-h con 0x7f4518136490 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4518073190 0x7f45180a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): picked mon.noname-i con 0x7f4518073190 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f45180a1690 con 0x7f4518135b70 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f45180a02b0 con 0x7f4518136490 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f45180a13b0 con 0x7f4518073190 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451f64e640 10 monclient(hunting): authenticate will time out at 951.667068s 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 0x7f4518135f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 0x7f4518135f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:41010/0 (socket says 172.21.15.43:41010) 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 -- 172.21.15.43:0/2074070412 learned_addr learned my addr 172.21.15.43:0/2074070412 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): get_auth_request con 0x7f4518135b70 auth_method 0 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4518136490 0x7f45180a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451dbeb640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4518073190 0x7f45180a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): get_auth_request con 0x7f4518136490 auth_method 0 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451dbeb640 10 monclient(hunting): get_auth_request con 0x7f4518073190 auth_method 0 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451dbeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451dbeb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451dbeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient(hunting): handle_auth_done global_id 4423 payload 274 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 -- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4518073190 msgr2=0x7f45180a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4518073190 0x7f45180a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 -- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4518136490 msgr2=0x7f45180a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4518136490 0x7f45180a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 monclient: found mon.noname-c 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:41.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.536+0000 7f451d3ea640 1 -- 172.21.15.43:0/2074070412 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45180a10d0 con 0x7f4518135b70 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451d3ea640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 0x7f4518135f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f45140098d0 tx=0x7f451401c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3d72c6687eb59609 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 1 -- 172.21.15.43:0/2074070412 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45140b0070 con 0x7f4518135b70 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: dump: 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.553 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 1 -- 172.21.15.43:0/2074070412 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45140aa810 con 0x7f4518135b70 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 1 -- 172.21.15.43:0/2074070412 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45140a9300 con 0x7f4518135b70 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 5 monclient: authenticate success, global_id 4423 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 4 set_mon_vals no callback set 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f450ffff640 10 monclient: dump: 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.555 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 10 monclient: shutdown 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 -- 172.21.15.43:0/2074070412 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 msgr2=0x7f4518135f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 0x7f4518135f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f45140098d0 tx=0x7f451401c440 comp rx=0 tx=0).stop 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 -- 172.21.15.43:0/2074070412 shutdown_connections 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4518136490 0x7f45180a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4518073190 0x7f45180a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 --2- 172.21.15.43:0/2074070412 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518135b70 0x7f4518135f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 -- 172.21.15.43:0/2074070412 >> 172.21.15.43:0/2074070412 conn(0x7f45180a85a0 msgr2=0x7f45180a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 -- 172.21.15.43:0/2074070412 shutdown_connections 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 -- 172.21.15.43:0/2074070412 wait complete. 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 10 monclient: monmap: 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.557 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.558 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 librados: starting msgr at 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 librados: starting objecter 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 Processor -- start 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 -- start start 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 librados: setting wanted keys 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 1 librados: calling monclient init 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.540+0000 7f451f64e640 10 monclient: init 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient: _add_conns ranks=[0,6,4,1,2,3,8,7,5] 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 0x7f451813be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): picked mon.noname-a con 0x7f4518073190 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4518135b70 0x7f4518130d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): picked mon.noname-d con 0x7f4518135b70 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518136490 0x7f45181cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): picked mon.noname-c con 0x7f4518136490 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f45180a4840 con 0x7f4518073190 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f451813f530 con 0x7f4518136490 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f451811ee70 con 0x7f4518135b70 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): authenticate will time out at 951.672460s 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 0x7f451813be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4518135b70 0x7f4518130d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 0x7f451813be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34524/0 (socket says 172.21.15.43:34524) 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 -- 172.21.15.43:0/1294474728 learned_addr learned my addr 172.21.15.43:0/1294474728 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): get_auth_request con 0x7f4518073190 auth_method 0 2024-04-19T00:44:41.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): get_auth_request con 0x7f4518135b70 auth_method 0 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518136490 0x7f45181cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): get_auth_request con 0x7f4518136490 auth_method 0 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4518135b70 msgr2=0x7f4518130d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4518135b70 0x7f4518130d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518136490 msgr2=0x7f45181cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518136490 0x7f45181cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 monclient: found mon.noname-a 2024-04-19T00:44:41.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45181d1af0 con 0x7f4518073190 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4518135b70 0x7f4518130d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4518136490 0x7f45181cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451d3ea640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 0x7f451813be20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f45140097c0 tx=0x7f45140191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 client.?.objecter ms_handle_connect 0x7f4518073190 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45140ba070 con 0x7f4518073190 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: dump: 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.563 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45140b1040 con 0x7f4518073190 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 5 monclient: authenticate success, global_id 4482 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient: _renew_subs 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45180735e0 con 0x7f4518073190 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45140b0070 con 0x7f4518073190 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450f7fe640 4 set_mon_vals no callback set 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 monclient: dump: 2024-04-19T00:44:41.565 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.566 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient: _renew_subs 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45181d10e0 con 0x7f4518073190 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 librados: init done 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 0x7f45181cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f45181359e0 con 0x7f4518131280 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f45140aacf0 con 0x7f4518073190 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f44fc038e40 0x7f44fc03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f45140e1e80 con 0x7f4518073190 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 client.4482.objecter ms_dispatch 0x7f4518000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:41.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 3 client.4482.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 3 client.4482.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 20 client.4482.objecter dump_active .. 0 homeless 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 0x7f45181cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient: get_auth_request con 0x7f4518131280 auth_method 0 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f44fc038e40 0x7f44fc03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 10 monclient: get_auth_request con 0x7f44fc038e40 auth_method 0 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-04-19T00:44:41.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45180a04e0 con 0x7f4518131280 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 client.4482.objecter ms_handle_connect 0x7f4518131280 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 client.4482.objecter resend_mon_ops 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451cbe9640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 0x7f45181cdc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45180a9060 tx=0x7f450800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f450dffb640 10 client.4482.objecter ms_handle_connect 0x7f44fc038e40 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.544+0000 7f451dbeb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f44fc038e40 0x7f44fc03b300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45180a79f0 tx=0x7f4510009820 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4508011e80 con 0x7f4518131280 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 10 monclient: dump: 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.569 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4508011e80 con 0x7f4518131280 2024-04-19T00:44:41.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:41.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 msgr2=0x7f45181cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 0x7f45181cdc10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45180a9060 tx=0x7f450800f610 comp rx=0 tx=0).stop 2024-04-19T00:44:41.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 msgr2=0x7f45181cdc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.548+0000 7f450dffb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518131280 0x7f45181cdc10 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45180a9060 tx=0x7f450800f610 comp rx=0 tx=0).stop 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451f64e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451f64e640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 0x7f45181ce150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451f64e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f451809ec50 con 0x7f4518071cf0 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 0x7f45181ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient: get_auth_request con 0x7f4518071cf0 auth_method 0 2024-04-19T00:44:41.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 1 -- 172.21.15.43:0/1294474728 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f451811f2e0 con 0x7f4518071cf0 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f451cbe9640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 0x7f45181ce150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f45180a9060 tx=0x7f45080046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.552+0000 7f450dffb640 10 client.4482.objecter ms_handle_connect 0x7f4518071cf0 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 10 client.4482.objecter resend_mon_ops 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45080148b0 con 0x7f4518071cf0 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:44:41.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 10 monclient: dump: 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.575 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7f45080273a0 con 0x7f4518071cf0 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 msgr2=0x7f45181ce150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 0x7f45181ce150 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f45180a9060 tx=0x7f45080046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 msgr2=0x7f45181ce150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f450dffb640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4518071cf0 0x7f45181ce150 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f45180a9060 tx=0x7f45080046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:41.576 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 25023, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:41.577 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 19, 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.578 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.579 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.580 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.581 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:41.582 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:41.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.593 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.595 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.596 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.597 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.598 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:41.599 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 10 librados: watch_flush enter 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 10 librados: watch_flush exit 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 20 client.4482.objecter shutdown clearing up homeless session... 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 10 client.4482.objecter successfully canceled tick 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f44fc038e40 msgr2=0x7f44fc03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f44fc038e40 0x7f44fc03b300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45180a79f0 tx=0x7f4510009820 comp rx=0 tx=0).stop 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 10 monclient: shutdown 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 msgr2=0x7f451813be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 0x7f451813be20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f45140097c0 tx=0x7f45140191e0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451dbeb640 1 -- 172.21.15.43:0/1294474728 reap_dead start 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 shutdown_connections 2024-04-19T00:44:41.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 --2- 172.21.15.43:0/1294474728 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4518073190 0x7f451813be20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 >> 172.21.15.43:0/1294474728 conn(0x7f45180a85a0 msgr2=0x7f4518134950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 shutdown_connections 2024-04-19T00:44:41.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 -- 172.21.15.43:0/1294474728 wait complete. 2024-04-19T00:44:41.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.556+0000 7f451f64e640 1 librados: shutdown 2024-04-19T00:44:41.602 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:41.602 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:44:41.602 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.712+0000 7fb4be585640 10 monclient: get_monmap_and_config 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.712+0000 7fb4be585640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.712+0000 7fb4be585640 10 monclient: monmap: 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:41.717827+0000 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:41.717827+0000 2024-04-19T00:44:41.716 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:41.717 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 Processor -- start 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 -- start start 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient: init 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient: _add_conns ranks=[3,9,7,4,5,8,6,1,10,2,0] 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8138cd0 0x7fb4b81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): picked mon.noname-g con 0x7fb4b8138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4b80a5b80 0x7fb4b81395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b80a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:41.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb4b81358b0 0x7fb4b8135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): picked mon.noname-i con 0x7fb4b81358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b80a4410 con 0x7fb4b80a5b80 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b8139b30 con 0x7fb4b8138cd0 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb4b80a3dc0 con 0x7fb4b81358b0 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4be585640 10 monclient(hunting): authenticate will time out at 951.844985s 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4b80a5b80 0x7fb4b81395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb4b80a5b80 0x7fb4b81395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34532/0 (socket says 172.21.15.43:34532) 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 -- 172.21.15.43:0/1649174596 learned_addr learned my addr 172.21.15.43:0/1649174596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): get_auth_request con 0x7fb4b80a5b80 auth_method 0 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4bcb22640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb4b81358b0 0x7fb4b8135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b7fff640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8138cd0 0x7fb4b81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4bcb22640 10 monclient(hunting): get_auth_request con 0x7fb4b81358b0 auth_method 0 2024-04-19T00:44:41.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4bcb22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4bcb22640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4bcb22640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request con 0x7fb4b8138cd0 auth_method 0 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_done global_id 4491 payload 274 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 -- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb4b81358b0 msgr2=0x7fb4b8135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb4b81358b0 0x7fb4b8135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 -- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8138cd0 msgr2=0x7fb4b81390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8138cd0 0x7fb4b81390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 monclient: found mon.noname-a 2024-04-19T00:44:41.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4b77fe640 1 -- 172.21.15.43:0/1649174596 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b80a40a0 con 0x7fb4b80a5b80 2024-04-19T00:44:41.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.716+0000 7fb4bcb22640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb4b81358b0 0x7fb4b8135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81395f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009870 tx=0x7fb4ac01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1b061d7df19f01 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 1 -- 172.21.15.43:0/1649174596 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb4ac0b0070 con 0x7fb4b80a5b80 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: dump: 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.730 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.731 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 1 -- 172.21.15.43:0/1649174596 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ac0aa810 con 0x7fb4b80a5b80 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 1 -- 172.21.15.43:0/1649174596 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb4ac0a9300 con 0x7fb4b80a5b80 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b6ffd640 10 monclient: dump: 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.732 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 5 monclient: authenticate success, global_id 4491 2024-04-19T00:44:41.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 4 set_mon_vals no callback set 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: shutdown 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/1649174596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 msgr2=0x7fb4b81395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81395f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac009870 tx=0x7fb4ac01c3c0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/1649174596 shutdown_connections 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb4b81358b0 0x7fb4b8135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81395f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- 172.21.15.43:0/1649174596 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8138cd0 0x7fb4b81390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/1649174596 >> 172.21.15.43:0/1649174596 conn(0x7fb4b80ab2f0 msgr2=0x7fb4b80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/1649174596 shutdown_connections 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/1649174596 wait complete. 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: monmap: 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.735 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 librados: starting msgr at 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 librados: starting objecter 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 Processor -- start 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- start start 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 librados: setting wanted keys 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 librados: calling monclient init 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: init 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: _add_conns ranks=[0,6,5,4,7,2,3,8,1] 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b80a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:44:41.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb4b81358b0 0x7fb4b813eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): picked mon.noname-d con 0x7fb4b81358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb4b8138cd0 0x7fb4b813f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): picked mon.noname-h con 0x7fb4b8138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b80a2010 con 0x7fb4b80a5b80 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb4b80a1e90 con 0x7fb4b8138cd0 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb4b8142aa0 con 0x7fb4b81358b0 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient(hunting): authenticate will time out at 951.850778s 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb4b81358b0 0x7fb4b813eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81d0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34536/0 (socket says 172.21.15.43:34536) 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 -- 172.21.15.43:0/3553624189 learned_addr learned my addr 172.21.15.43:0/3553624189 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request con 0x7fb4b80a5b80 auth_method 0 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): get_auth_request con 0x7fb4b81358b0 auth_method 0 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient(hunting): handle_auth_done global_id 4500 payload 824 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb4b81358b0 msgr2=0x7fb4b813eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb4b81358b0 0x7fb4b813eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4bcb22640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb4b8138cd0 0x7fb4b813f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb4b8138cd0 msgr2=0x7fb4b813f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb4b8138cd0 0x7fb4b813f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 monclient: found mon.noname-a 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b813b690 con 0x7fb4b80a5b80 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b7fff640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81d0e30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800d5f0 tx=0x7fb4a800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 client.?.objecter ms_handle_connect 0x7fb4b80a5b80 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb4a8012a60 con 0x7fb4b80a5b80 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: dump: 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.740 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4a8019070 con 0x7fb4b80a5b80 2024-04-19T00:44:41.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 5 monclient: authenticate success, global_id 4500 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: _renew_subs 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b80a3590 con 0x7fb4b80a5b80 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb4a8012c20 con 0x7fb4b80a5b80 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b4ff9640 10 monclient: dump: 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.742 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4b67fc640 4 set_mon_vals no callback set 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: _renew_subs 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b813a170 con 0x7fb4b80a5b80 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.720+0000 7fb4be585640 1 librados: init done 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4be585640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4be585640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 0x7fb4b8133b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b8135630 con 0x7fb4b8133730 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb4a80125f0 con 0x7fb4b80a5b80 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb4a4038e10 0x7fb4a403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fb4a800f040 con 0x7fb4b80a5b80 2024-04-19T00:44:41.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 client.4500.objecter ms_dispatch 0x7fb4b8000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 3 client.4500.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 3 client.4500.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 20 client.4500.objecter dump_active .. 0 homeless 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 0x7fb4b8133b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient: get_auth_request con 0x7fb4b8133730 auth_method 0 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4bcb22640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb4a4038e10 0x7fb4a403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4bcb22640 10 monclient: get_auth_request con 0x7fb4a4038e10 auth_method 0 2024-04-19T00:44:41.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_done global_id 4431 payload 824 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4b80693b0 con 0x7fb4b8133730 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 client.4500.objecter ms_handle_connect 0x7fb4b8133730 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 client.4500.objecter resend_mon_ops 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 0x7fb4b8133b10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0ab000 tx=0x7fb4ac01ac60 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 client.4500.objecter ms_handle_connect 0x7fb4a4038e10 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4bcb22640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb4a4038e10 0x7fb4a403b2d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8132040 tx=0x7fb4b000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb4ac0ba070 con 0x7fb4b8133730 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 monclient: dump: 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.746 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb4ac0036e0 con 0x7fb4b8133730 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 msgr2=0x7fb4b8133b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 0x7fb4b8133b10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0ab000 tx=0x7fb4ac01ac60 comp rx=0 tx=0).stop 2024-04-19T00:44:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 msgr2=0x7fb4b8133b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.724+0000 7fb4b4ff9640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8133730 0x7fb4b8133b10 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac0ab000 tx=0x7fb4ac01ac60 comp rx=0 tx=0).stop 2024-04-19T00:44:41.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 0x7fb4b8137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b806d140 con 0x7fb4b8134050 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 0x7fb4b8137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient: get_auth_request con 0x7fb4b8134050 auth_method 0 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-04-19T00:44:41.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 1 -- 172.21.15.43:0/3553624189 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb4b8076190 con 0x7fb4b8134050 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b77fe640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 0x7fb4b8137990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac01cc60 tx=0x7fb4ac0191c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 client.4500.objecter ms_handle_connect 0x7fb4b8134050 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 client.4500.objecter resend_mon_ops 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb4ac0b0040 con 0x7fb4b8134050 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 monclient: dump: 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.751 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fb4ac0b6480 con 0x7fb4b8134050 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 msgr2=0x7fb4b8137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 0x7fb4b8137990 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb4ac01cc60 tx=0x7fb4ac0191c0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 msgr2=0x7fb4b8137990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4b4ff9640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb4b8134050 0x7fb4b8137990 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:41.753 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 25215, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:41.754 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 19, 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.755 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.757 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.758 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.760 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:41.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:41.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.772 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.776 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.776 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:41.776 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 10 librados: watch_flush enter 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 10 librados: watch_flush exit 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 20 client.4500.objecter shutdown clearing up homeless session... 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 10 client.4500.objecter successfully canceled tick 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb4a4038e10 msgr2=0x7fb4a403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb4a4038e10 0x7fb4a403b2d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8132040 tx=0x7fb4b000b040 comp rx=0 tx=0).stop 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 10 monclient: shutdown 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 msgr2=0x7fb4b81d0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81d0e30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4a800d5f0 tx=0x7fb4a800dab0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4bcb22640 1 -- 172.21.15.43:0/3553624189 reap_dead start 2024-04-19T00:44:41.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 shutdown_connections 2024-04-19T00:44:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 --2- 172.21.15.43:0/3553624189 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb4b80a5b80 0x7fb4b81d0e30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 >> 172.21.15.43:0/3553624189 conn(0x7fb4b80ab2f0 msgr2=0x7fb4b80a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 shutdown_connections 2024-04-19T00:44:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 -- 172.21.15.43:0/3553624189 wait complete. 2024-04-19T00:44:41.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.732+0000 7fb4be585640 1 librados: shutdown 2024-04-19T00:44:41.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:41.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:44:41.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:44:41.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.892+0000 7f6657693640 10 monclient: get_monmap_and_config 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.892+0000 7f6657693640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.892+0000 7f6657693640 10 monclient: monmap: 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:41.898391+0000 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:41.898391+0000 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.897 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:41.898 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 Processor -- start 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 -- start start 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient: init 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient: _add_conns ranks=[3,10,5,8,2,6,4,7,9,0,1] 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f6650072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): picked mon.noname-g con 0x7f6650072750 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6650073500 0x7f66500a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): picked mon.noname-f con 0x7f6650073500 addr v2:172.21.15.161:3300/0 2024-04-19T00:44:41.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6650073070 0x7f66500a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): picked mon.noname-h con 0x7f6650073070 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f665006e630 con 0x7f6650073500 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f665006f210 con 0x7f6650072750 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f665006f4f0 con 0x7f6650073070 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6657693640 10 monclient(hunting): authenticate will time out at 952.025472s 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6650073500 0x7f66500a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f6650072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6650073500 0x7f66500a4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:41798/0 (socket says 172.21.15.43:41798) 2024-04-19T00:44:41.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f6650072b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:33572/0 (socket says 172.21.15.43:33572) 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 1 -- 172.21.15.43:0/1724973421 learned_addr learned my addr 172.21.15.43:0/1724973421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): get_auth_request con 0x7f6650073500 auth_method 0 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): get_auth_request con 0x7f6650072750 auth_method 0 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 1 --2- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6650073070 0x7f66500a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): get_auth_request con 0x7f6650073070 auth_method 0 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6654c2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f6655c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 -- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6650073070 msgr2=0x7f66500a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 --2- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6650073070 0x7f66500a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 -- 172.21.15.43:0/1724973421 >> v2:172.21.15.161:3300/0 conn(0x7f6650073500 msgr2=0x7f66500a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 --2- 172.21.15.43:0/1724973421 >> v2:172.21.15.161:3300/0 conn(0x7f6650073500 0x7f66500a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 monclient: found mon.noname-g 2024-04-19T00:44:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:44:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.896+0000 7f665542f640 1 -- 172.21.15.43:0/1724973421 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665006c450 con 0x7f6650072750 2024-04-19T00:44:41.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 1 --2- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f6650072b30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f664c0097c0 tx=0x7f664c01c360 comp rx=0 tx=0).ready entity=mon.3 client_cookie=2d28b1d360c35342 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 1 -- 172.21.15.43:0/1724973421 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f664c0b0070 con 0x7f6650072750 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: dump: 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.912 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 1 -- 172.21.15.43:0/1724973421 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664c0aa810 con 0x7f6650072750 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 1 -- 172.21.15.43:0/1724973421 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f664c0a9300 con 0x7f6650072750 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 5 monclient: authenticate success, global_id 4449 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 4 set_mon_vals no callback set 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6647fff640 10 monclient: dump: 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.914 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: shutdown 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 msgr2=0x7f6650072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f6650072b30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f664c0097c0 tx=0x7f664c01c360 comp rx=0 tx=0).stop 2024-04-19T00:44:41.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- 172.21.15.43:0/1724973421 shutdown_connections 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6650073070 0x7f66500a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- 172.21.15.43:0/1724973421 >> v2:172.21.15.161:3300/0 conn(0x7f6650073500 0x7f66500a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- 172.21.15.43:0/1724973421 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f6650072b30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- 172.21.15.43:0/1724973421 >> 172.21.15.43:0/1724973421 conn(0x7f66500a6350 msgr2=0x7f665006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- 172.21.15.43:0/1724973421 shutdown_connections 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- 172.21.15.43:0/1724973421 wait complete. 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: build_initial_monmap 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: monmap: 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:41.916 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 librados: starting msgr at 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 librados: starting objecter 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 Processor -- start 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- start start 2024-04-19T00:44:41.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 librados: setting wanted keys 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 librados: calling monclient init 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: init 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient: _add_conns ranks=[3,4,7,2,6,1,0,5,8] 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f66501d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): picked mon.noname-g con 0x7f6650072750 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 0x7f66501d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): picked mon.noname-c con 0x7f6650073070 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6650073500 0x7f665013e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): picked mon.noname-i con 0x7f6650073500 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f66501429d0 con 0x7f6650072750 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f6650142850 con 0x7f6650073070 2024-04-19T00:44:41.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6650142b50 con 0x7f6650073500 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6657693640 10 monclient(hunting): authenticate will time out at 952.031180s 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 0x7f66501d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 0x7f66501d11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:41020/0 (socket says 172.21.15.43:41020) 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 -- 172.21.15.43:0/3762661496 learned_addr learned my addr 172.21.15.43:0/3762661496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): get_auth_request con 0x7f6650073070 auth_method 0 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f66501d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): get_auth_request con 0x7f6650072750 auth_method 0 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6650073500 0x7f665013e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): get_auth_request con 0x7f6650073500 auth_method 0 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6655c30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient(hunting): handle_auth_done global_id 4441 payload 824 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6650073500 msgr2=0x7f665013e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6650073500 0x7f665013e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 msgr2=0x7f66501d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6650072750 0x7f66501d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient: _finish_hunting 0 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 monclient: found mon.noname-c 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.900+0000 7f6654c2e640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665006e870 con 0x7f6650073070 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6654c2e640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 0x7f66501d11f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f664000e8a0 tx=0x7f664000ed60 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 client.?.objecter ms_handle_connect 0x7f6650073070 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6640017070 con 0x7f6650073070 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: dump: 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.922 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664000cd50 con 0x7f6650073070 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 5 monclient: authenticate success, global_id 4441 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 monclient: _renew_subs 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:41.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6650120560 con 0x7f6650073070 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66400138e0 con 0x7f6650073070 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f66477fe640 4 set_mon_vals no callback set 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: dump: 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.924 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 monclient: _renew_subs 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6650121110 con 0x7f6650073070 2024-04-19T00:44:41.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 1 librados: init done 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 0x7f665013e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f665006d140 con 0x7f66501d1730 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f664001a070 con 0x7f6650073070 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6634038e10 0x7f663403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 0x7f665013e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f664000f040 con 0x7f6650073070 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 client.4441.objecter ms_dispatch 0x7f6650000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 3 client.4441.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 3 client.4441.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient: get_auth_request con 0x7f66501d1730 auth_method 0 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6655c30640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6634038e10 0x7f663403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6655c30640 10 monclient: get_auth_request con 0x7f6634038e10 auth_method 0 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f665006e420 con 0x7f66501d1730 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 client.4441.objecter ms_handle_connect 0x7f66501d1730 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 client.4441.objecter resend_mon_ops 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f665542f640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 0x7f665013e5d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f665006c750 tx=0x7f664c019100 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f664c0b0040 con 0x7f66501d1730 2024-04-19T00:44:41.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: dump: 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.928 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 client.4441.objecter ms_handle_connect 0x7f6634038e10 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6655c30640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6634038e10 0x7f663403b2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6650120980 tx=0x7f664800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f664c009320 con 0x7f66501d1730 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 msgr2=0x7f665013e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 0x7f665013e5d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f665006c750 tx=0x7f664c019100 comp rx=0 tx=0).stop 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 msgr2=0x7f665013e5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.904+0000 7f6645ffb640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f66501d1730 0x7f665013e5d0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f665006c750 tx=0x7f664c019100 comp rx=0 tx=0).stop 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6657693640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6657693640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 0x7f6650139be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6657693640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6650068240 con 0x7f665013eb10 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 0x7f6650139be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient: get_auth_request con 0x7f665013eb10 auth_method 0 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:41.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 1 -- 172.21.15.43:0/3762661496 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f665013ef20 con 0x7f665013eb10 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f665542f640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 0x7f6650139be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f66500740e0 tx=0x7f664c0a6000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6645ffb640 10 client.4441.objecter ms_handle_connect 0x7f665013eb10 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6645ffb640 10 client.4441.objecter resend_mon_ops 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f664c0b0040 con 0x7f665013eb10 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6645ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.912+0000 7f6645ffb640 10 monclient: dump: 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:41.932 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7f664c0b5450 con 0x7f665013eb10 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:41.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 msgr2=0x7f6650139be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 0x7f6650139be0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f66500740e0 tx=0x7f664c0a6000 comp rx=0 tx=0).stop 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 msgr2=0x7f6650139be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6645ffb640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f665013eb10 0x7f6650139be0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f66500740e0 tx=0x7f664c0a6000 comp rx=0 tx=0).stop 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 25387, 2024-04-19T00:44:41.934 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 20, 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:41.935 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:41.936 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:41.937 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:41.938 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.939 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:41.940 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.942 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.944 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.945 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.946 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.947 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:41.949 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.952 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:44:41.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:41.956 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 10 librados: watch_flush enter 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 10 librados: watch_flush exit 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 20 client.4441.objecter shutdown clearing up homeless session... 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 10 client.4441.objecter successfully canceled tick 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6634038e10 msgr2=0x7f663403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6634038e10 0x7f663403b2d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6650120980 tx=0x7f664800b040 comp rx=0 tx=0).stop 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 10 monclient: shutdown 2024-04-19T00:44:41.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6655c30640 1 -- 172.21.15.43:0/3762661496 reap_dead start 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 msgr2=0x7f66501d11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 0x7f66501d11f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f664000e8a0 tx=0x7f664000ed60 comp rx=0 tx=0).stop 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 shutdown_connections 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 --2- 172.21.15.43:0/3762661496 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6650073070 0x7f66501d11f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 >> 172.21.15.43:0/3762661496 conn(0x7f66500a6350 msgr2=0x7f6650137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 shutdown_connections 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 -- 172.21.15.43:0/3762661496 wait complete. 2024-04-19T00:44:41.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:41.916+0000 7f6657693640 1 librados: shutdown 2024-04-19T00:44:41.959 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:41.960 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:44:41.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:44:42.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient: get_monmap_and_config 2024-04-19T00:44:42.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient: build_initial_monmap 2024-04-19T00:44:42.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient: monmap: 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:42.083196+0000 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:42.083196+0000 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:42.082 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:42.083 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:42.083 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:42.083 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:42.083 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 Processor -- start 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 -- start start 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient: init 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient: _add_conns ranks=[7,9,5,0,6,10,2,1,4,3,8] 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c138a50 0x7fc42c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): picked mon.noname-i con 0x7fc42c138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc42c0a5bb0 0x7fc42c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): picked mon.noname-a con 0x7fc42c0a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c137960 0x7fc42c137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): picked mon.noname-h con 0x7fc42c137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fc42c0a1020 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc42c1398b0 con 0x7fc42c137960 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc42c0a0780 con 0x7fc42c138a50 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43339d640 10 monclient(hunting): authenticate will time out at 952.210287s 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc42c0a5bb0 0x7fc42c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc42c0a5bb0 0x7fc42c139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34544/0 (socket says 172.21.15.43:34544) 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 -- 172.21.15.43:0/2544257341 learned_addr learned my addr 172.21.15.43:0/2544257341 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:42.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): get_auth_request con 0x7fc42c0a5bb0 auth_method 0 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43193a640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c137960 0x7fc42c137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc431139640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c138a50 0x7fc42c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc431139640 10 monclient(hunting): get_auth_request con 0x7fc42c138a50 auth_method 0 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc431139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc431139640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc431139640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43193a640 10 monclient(hunting): get_auth_request con 0x7fc42c137960 auth_method 0 2024-04-19T00:44:42.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43193a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43193a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc43193a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient(hunting): handle_auth_done global_id 4509 payload 274 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 -- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c138a50 msgr2=0x7fc42c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c138a50 0x7fc42c138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 -- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c137960 msgr2=0x7fc42c137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c137960 0x7fc42c137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient: _finish_hunting 0 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 monclient: found mon.noname-a 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:44:42.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.080+0000 7fc430938640 1 -- 172.21.15.43:0/2544257341 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42c0a0d40 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc430938640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c0a5bb0 0x7fc42c139370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc428009e50 tx=0x7fc42801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ccc00bdf78bf27 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 1 -- 172.21.15.43:0/2544257341 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4280090d0 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: dump: 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.096 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: _finish_auth 0 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 5 monclient: authenticate success, global_id 4509 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 1 -- 172.21.15.43:0/2544257341 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4280a5b70 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:42.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 4 set_mon_vals no callback set 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 1 -- 172.21.15.43:0/2544257341 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4280a5d60 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc41ffff640 10 monclient: dump: 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.098 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: shutdown 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- 172.21.15.43:0/2544257341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c0a5bb0 msgr2=0x7fc42c139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c0a5bb0 0x7fc42c139370 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc428009e50 tx=0x7fc42801c8c0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- 172.21.15.43:0/2544257341 shutdown_connections 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c137960 0x7fc42c137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc42c0a5bb0 0x7fc42c139370 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- 172.21.15.43:0/2544257341 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c138a50 0x7fc42c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- 172.21.15.43:0/2544257341 >> 172.21.15.43:0/2544257341 conn(0x7fc42c0a7cb0 msgr2=0x7fc42c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- 172.21.15.43:0/2544257341 shutdown_connections 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- 172.21.15.43:0/2544257341 wait complete. 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: build_initial_monmap 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: monmap: 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:42.100 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 librados: starting msgr at 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 librados: starting objecter 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 Processor -- start 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- start start 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 librados: setting wanted keys 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 librados: calling monclient init 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: init 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:42.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient: _add_conns ranks=[2,7,5,6,1,4,0,8,3] 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc42c0a5bb0 0x7fc42c1d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): picked mon.noname-b con 0x7fc42c0a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c137960 0x7fc42c1d1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): picked mon.noname-i con 0x7fc42c137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c138a50 0x7fc42c13c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): picked mon.noname-h con 0x7fc42c138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc42c0a2100 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc42c0a1f80 con 0x7fc42c138a50 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc42c142810 con 0x7fc42c137960 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:42.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43339d640 10 monclient(hunting): authenticate will time out at 952.215402s 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc42c0a5bb0 0x7fc42c1d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc42c0a5bb0 0x7fc42c1d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:38460/0 (socket says 172.21.15.43:38460) 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 -- 172.21.15.43:0/724411308 learned_addr learned my addr 172.21.15.43:0/724411308 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): get_auth_request con 0x7fc42c0a5bb0 auth_method 0 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc430938640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c137960 0x7fc42c1d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43193a640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c138a50 0x7fc42c13c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc430938640 10 monclient(hunting): get_auth_request con 0x7fc42c137960 auth_method 0 2024-04-19T00:44:42.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc430938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc430938640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc430938640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c137960 msgr2=0x7fc42c1d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c137960 0x7fc42c1d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c138a50 msgr2=0x7fc42c13c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c138a50 0x7fc42c13c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient: _finish_hunting 0 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 monclient: found mon.noname-b 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc431139640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc42c122480 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43193a640 10 monclient: get_auth_request con 0x7fc42c138a50 auth_method 0 2024-04-19T00:44:42.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.084+0000 7fc43193a640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc42c138a50 0x7fc42c13c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc431139640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc42c0a5bb0 0x7fc42c1d0930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc42000d5f0 tx=0x7fc42000dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 client.?.objecter ms_handle_connect 0x7fc42c0a5bb0 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc420007380 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: dump: 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.105 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: _finish_auth 0 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc420019070 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 5 monclient: authenticate success, global_id 4430 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 monclient: _renew_subs 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc42c121800 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc420014430 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41f7fe640 4 set_mon_vals no callback set 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: dump: 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.107 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 monclient: _renew_subs 2024-04-19T00:44:42.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc42c13e490 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 1 librados: init done 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 0x7fc42c13ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc42c06d140 con 0x7fc42c1d1640 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc4200125f0 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc410038e10 0x7fc41003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 0x7fc42c13ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient: get_auth_request con 0x7fc42c1d1640 auth_method 0 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fc42000f040 con 0x7fc42c0a5bb0 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 client.4430.objecter ms_dispatch 0x7fc42c000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 3 client.4430.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 3 client.4430.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 20 client.4430.objecter dump_active .. 0 homeless 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43193a640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc410038e10 0x7fc41003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43193a640 10 monclient: get_auth_request con 0x7fc410038e10 auth_method 0 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 10 monclient(hunting): handle_auth_done global_id 4525 payload 824 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc42c122200 con 0x7fc42c1d1640 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 client.4430.objecter ms_handle_connect 0x7fc42c1d1640 2024-04-19T00:44:42.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 client.4430.objecter resend_mon_ops 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc430938640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 0x7fc42c13ce90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc42c09f9d0 tx=0x7fc428019660 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 client.4430.objecter ms_handle_connect 0x7fc410038e10 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc43193a640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc410038e10 0x7fc41003b2d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc4240097c0 tx=0x7fc424008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4280bd030 con 0x7fc42c1d1640 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: dump: 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.111 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc428004040 con 0x7fc42c1d1640 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 msgr2=0x7fc42c13ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 0x7fc42c13ce90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc42c09f9d0 tx=0x7fc428019660 comp rx=0 tx=0).stop 2024-04-19T00:44:42.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 msgr2=0x7fc42c13ce90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:42.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.088+0000 7fc41dffb640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c1d1640 0x7fc42c13ce90 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc42c09f9d0 tx=0x7fc428019660 comp rx=0 tx=0).stop 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc43339d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc43339d640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 0x7fc42c0a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc43339d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc42c13d7b0 con 0x7fc42c13d3d0 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 0x7fc42c0a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient: get_auth_request con 0x7fc42c13d3d0 auth_method 0 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 10 monclient(hunting): handle_auth_done global_id 4534 payload 824 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 1 -- 172.21.15.43:0/724411308 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc42c076190 con 0x7fc42c13d3d0 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc430938640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 0x7fc42c0a3020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc428000f90 tx=0x7fc4280a7000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc41dffb640 10 client.4430.objecter ms_handle_connect 0x7fc42c13d3d0 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc41dffb640 10 client.4430.objecter resend_mon_ops 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc4280bd040 con 0x7fc42c13d3d0 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc41dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc41dffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.096+0000 7fc41dffb640 10 monclient: dump: 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.116 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7fc4280c27c0 con 0x7fc42c13d3d0 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 msgr2=0x7fc42c0a3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 0x7fc42c0a3020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc428000f90 tx=0x7fc4280a7000 comp rx=0 tx=0).stop 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 msgr2=0x7fc42c0a3020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc41dffb640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc42c13d3d0 0x7fc42c0a3020 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc428000f90 tx=0x7fc4280a7000 comp rx=0 tx=0).stop 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 8, 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 25555, 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:44:42.118 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 20, 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:44:42.119 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:44:42.120 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:44:42.121 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:44:42.122 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:42.123 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:44:42.125 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.126 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:44:42.128 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.130 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.131 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.133 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.135 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:44:42.138 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:42.139 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:44:42.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:44:42.141 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:44:42.141 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 10 librados: watch_flush enter 2024-04-19T00:44:42.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 10 librados: watch_flush exit 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 20 client.4430.objecter shutdown clearing up homeless session... 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 10 client.4430.objecter successfully canceled tick 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc410038e10 msgr2=0x7fc41003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc410038e10 0x7fc41003b2d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc4240097c0 tx=0x7fc424008040 comp rx=0 tx=0).stop 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 10 monclient: shutdown 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc42c0a5bb0 msgr2=0x7fc42c1d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 --2- 172.21.15.43:0/724411308 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc42c0a5bb0 0x7fc42c1d0930 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc42000d5f0 tx=0x7fc42000dab0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43193a640 1 -- 172.21.15.43:0/724411308 reap_dead start 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43193a640 1 -- 172.21.15.43:0/724411308 reap_dead start 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 shutdown_connections 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 >> 172.21.15.43:0/724411308 conn(0x7fc42c0a7cb0 msgr2=0x7fc42c0a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 shutdown_connections 2024-04-19T00:44:42.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 -- 172.21.15.43:0/724411308 wait complete. 2024-04-19T00:44:42.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.100+0000 7fc43339d640 1 librados: shutdown 2024-04-19T00:44:42.144 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:44:42.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['h'] 2024-04-19T00:44:42.144 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['i'] 2024-04-19T00:44:42.144 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-04-19T00:44:42.144 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h store 2024-04-19T00:44:42.145 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h sync_force --yes-i-really-mean-it 2024-04-19T00:44:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient: get_monmap_and_config 2024-04-19T00:44:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient: build_initial_monmap 2024-04-19T00:44:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient: monmap: 2024-04-19T00:44:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:42.262 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:42.263952+0000 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:42.263952+0000 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:42.263 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:42.264 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:42.264 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 Processor -- start 2024-04-19T00:44:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 -- start start 2024-04-19T00:44:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient: init 2024-04-19T00:44:42.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient: _add_conns ranks=[5,1,2,6,3,0,10,9,7,4,8] 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3204138d40 0x7f32040a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): picked mon.noname-h con 0x7f3204138d40 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3204139120 addr v1:172.21.15.161:6789/0 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 0x7f3204140660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): picked mon.noname-b con 0x7f320413e270 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32040a2720 con 0x7f3204139120 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32040a3e30 con 0x7f320413e270 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f32040a2b50 con 0x7f3204138d40 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3209d36640 10 monclient(hunting): authenticate will time out at 952.391317s 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 0x7f3204140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 0x7f3204140660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:38462/0 (socket says 172.21.15.43:38462) 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 -- 172.21.15.43:0/3280421338 learned_addr learned my addr 172.21.15.43:0/3280421338 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): get_auth_request con 0x7f320413e270 auth_method 0 2024-04-19T00:44:42.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f32037fe640 1 --2- 172.21.15.43:0/3280421338 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3204138d40 0x7f32040a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f32037fe640 10 monclient(hunting): get_auth_request con 0x7f3204138d40 auth_method 0 2024-04-19T00:44:42.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f32037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f32037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f32037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient(hunting): handle_auth_done global_id 4439 payload 274 2024-04-19T00:44:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 -- 172.21.15.43:0/3280421338 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3204138d40 msgr2=0x7f32040a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 --2- 172.21.15.43:0/3280421338 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3204138d40 0x7f32040a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 -- 172.21.15.43:0/3280421338 >> v1:172.21.15.161:6789/0 conn(0x7f3204139120 legacy=0x7f3204139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 monclient: found mon.noname-b 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.260+0000 7f3203fff640 1 -- 172.21.15.43:0/3280421338 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32040a2020 con 0x7f320413e270 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3203fff640 1 --2- 172.21.15.43:0/3280421338 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 0x7f3204140660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009e80 tx=0x7f31fc01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3e79563e266d52b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 1 -- 172.21.15.43:0/3280421338 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31fc0090d0 con 0x7f320413e270 2024-04-19T00:44:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: dump: 2024-04-19T00:44:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.281 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.282 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.283 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 5 monclient: authenticate success, global_id 4439 2024-04-19T00:44:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 1 -- 172.21.15.43:0/3280421338 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc0a5a00 con 0x7f320413e270 2024-04-19T00:44:42.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:44:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 4 set_mon_vals no callback set 2024-04-19T00:44:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 1 -- 172.21.15.43:0/3280421338 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31fc0a5bf0 con 0x7f320413e270 2024-04-19T00:44:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f32027fc640 10 monclient: dump: 2024-04-19T00:44:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.286 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.287 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.288 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.288 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 10 monclient: shutdown 2024-04-19T00:44:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 -- 172.21.15.43:0/3280421338 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 msgr2=0x7f3204140660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 --2- 172.21.15.43:0/3280421338 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 0x7f3204140660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009e80 tx=0x7f31fc01c910 comp rx=0 tx=0).stop 2024-04-19T00:44:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 -- 172.21.15.43:0/3280421338 shutdown_connections 2024-04-19T00:44:42.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 --2- 172.21.15.43:0/3280421338 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f320413e270 0x7f3204140660 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 --2- 172.21.15.43:0/3280421338 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3204138d40 0x7f32040a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 -- 172.21.15.43:0/3280421338 >> 172.21.15.43:0/3280421338 conn(0x7f32040ab2f0 msgr2=0x7f32040ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 -- 172.21.15.43:0/3280421338 shutdown_connections 2024-04-19T00:44:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 -- 172.21.15.43:0/3280421338 wait complete. 2024-04-19T00:44:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 10 monclient: build_initial_monmap 2024-04-19T00:44:42.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 10 monclient: monmap: 2024-04-19T00:44:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:42.291 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:42.292 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:42.293 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 librados: starting msgr at 2024-04-19T00:44:42.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.264+0000 7f3209d36640 1 librados: starting objecter 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 Processor -- start 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- start start 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 librados: setting wanted keys 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 librados: calling monclient init 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: init 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:42.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: _add_conns ranks=[3,7,6,1,4,2,0,5,8] 2024-04-19T00:44:42.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3204138d40 0x7f32040a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): picked mon.noname-g con 0x7f3204138d40 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:44:42.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3204139120 0x7f3204135a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): picked mon.noname-i con 0x7f3204139120 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:42.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f320413e270 0x7f3204135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): picked mon.noname-d con 0x7f320413e270 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3204143750 con 0x7f3204138d40 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f32041435d0 con 0x7f320413e270 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f32041438d0 con 0x7f3204139120 2024-04-19T00:44:42.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): authenticate will time out at 952.396557s 2024-04-19T00:44:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f320413e270 0x7f3204135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f320413e270 0x7f3204135f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39140/0 (socket says 172.21.15.43:39140) 2024-04-19T00:44:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 -- 172.21.15.43:0/3224060295 learned_addr learned my addr 172.21.15.43:0/3224060295 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): get_auth_request con 0x7f320413e270 auth_method 0 2024-04-19T00:44:42.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3204138d40 0x7f32040a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3204139120 0x7f3204135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): get_auth_request con 0x7f3204138d40 auth_method 0 2024-04-19T00:44:42.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 10 monclient(hunting): get_auth_request con 0x7f3204139120 auth_method 0 2024-04-19T00:44:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-04-19T00:44:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3204139120 msgr2=0x7f3204135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3204139120 0x7f3204135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3204138d40 msgr2=0x7f32040a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3204138d40 0x7f32040a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient: _finish_hunting 0 2024-04-19T00:44:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 monclient: found mon.noname-d 2024-04-19T00:44:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:44:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32040a2eb0 con 0x7f320413e270 2024-04-19T00:44:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3204139120 0x7f3204135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:44:42.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3203fff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f320413e270 0x7f3204135f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009040 tx=0x7f31fc0196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 client.?.objecter ms_handle_connect 0x7f320413e270 2024-04-19T00:44:42.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:42.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31fc0bc020 con 0x7f320413e270 2024-04-19T00:44:42.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:44:42.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: dump: 2024-04-19T00:44:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.304 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.305 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.306 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.306 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.306 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: _finish_auth 0 2024-04-19T00:44:42.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: _check_auth_tickets 2024-04-19T00:44:42.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:42.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31fc003e10 con 0x7f320413e270 2024-04-19T00:44:42.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:42.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31fc0044b0 con 0x7f320413e270 2024-04-19T00:44:42.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 monclient: dump: 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.308 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.309 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.309 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.309 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.309 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.309 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.309 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.310 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.310 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.310 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.310 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3201ffb640 4 set_mon_vals no callback set 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 5 monclient: authenticate success, global_id 4344 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: _renew_subs 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32040a3140 con 0x7f320413e270 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: _renew_subs 2024-04-19T00:44:42.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:44:42.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f32040a3690 con 0x7f320413e270 2024-04-19T00:44:42.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 librados: init done 2024-04-19T00:44:42.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:42.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 0x7f320409cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f3204074f70 con 0x7f32041364c0 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f31fc0bf020 con 0x7f320413e270 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f31e4038e10 0x7f31e403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f31fc0b3040 con 0x7f320413e270 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 10 client.4344.objecter ms_dispatch 0x7f3204000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 3 client.4344.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:42.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 3 client.4344.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 0x7f320409cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f31fbfff640 20 client.4344.objecter dump_active .. 0 homeless 2024-04-19T00:44:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient: get_auth_request con 0x7f32041364c0 auth_method 0 2024-04-19T00:44:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f31e4038e10 0x7f31e403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f3202ffd640 10 monclient: get_auth_request con 0x7f31e4038e10 auth_method 0 2024-04-19T00:44:42.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 10 monclient(hunting): handle_auth_done global_id 4424 payload 824 2024-04-19T00:44:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.268+0000 7f32037fe640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32041368a0 con 0x7f32041364c0 2024-04-19T00:44:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 client.4344.objecter ms_handle_connect 0x7f32041364c0 2024-04-19T00:44:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 client.4344.objecter resend_mon_ops 2024-04-19T00:44:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f32037fe640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 0x7f320409cdb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31f4010780 tx=0x7f31f4010c40 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 client.4344.objecter ms_handle_connect 0x7f31e4038e10 2024-04-19T00:44:42.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f3202ffd640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f31e4038e10 0x7f31e403b2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f320409dd50 tx=0x7f31ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31f4018020 con 0x7f32041364c0 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 monclient: dump: 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.317 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.318 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31f4004240 con 0x7f32041364c0 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 msgr2=0x7f320409cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 0x7f320409cdb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31f4010780 tx=0x7f31f4010c40 comp rx=0 tx=0).stop 2024-04-19T00:44:42.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 msgr2=0x7f320409cdb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:42.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.272+0000 7f31fbfff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f32041364c0 0x7f320409cdb0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31f4010780 tx=0x7f31f4010c40 comp rx=0 tx=0).stop 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 0x7f320402c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f320409ebb0 con 0x7f320402c450 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 0x7f320402c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient: get_auth_request con 0x7f320402c450 auth_method 0 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:42.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 10 monclient(hunting): handle_auth_done global_id 4433 payload 824 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 1 -- 172.21.15.43:0/3224060295 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f32040693b0 con 0x7f320402c450 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f32037fe640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 0x7f320402c830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31f4019000 tx=0x7f31f40046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 client.4344.objecter ms_handle_connect 0x7f320402c450 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 client.4344.objecter resend_mon_ops 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31f4018040 con 0x7f320402c450 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 monclient: dump: 2024-04-19T00:44:42.323 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:42.324 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f31f4020e50 con 0x7f320402c450 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 msgr2=0x7f320402c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 0x7f320402c830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31f4019000 tx=0x7f31f40046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 msgr2=0x7f320402c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f31fbfff640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f320402c450 0x7f320402c830 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f31f4019000 tx=0x7f31f40046d0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:44:42.325 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:44:42.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 10 librados: watch_flush enter 2024-04-19T00:44:42.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 10 librados: watch_flush exit 2024-04-19T00:44:42.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 20 client.4344.objecter shutdown clearing up homeless session... 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 10 client.4344.objecter successfully canceled tick 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f31e4038e10 msgr2=0x7f31e403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f31e4038e10 0x7f31e403b2d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f320409dd50 tx=0x7f31ec009040 comp rx=0 tx=0).stop 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 10 monclient: shutdown 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f320413e270 msgr2=0x7f3204135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 --2- 172.21.15.43:0/3224060295 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f320413e270 0x7f3204135f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009040 tx=0x7f31fc0196b0 comp rx=0 tx=0).stop 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3203fff640 1 -- 172.21.15.43:0/3224060295 reap_dead start 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3203fff640 1 -- 172.21.15.43:0/3224060295 reap_dead start 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 shutdown_connections 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.280+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 >> 172.21.15.43:0/3224060295 conn(0x7f32040ab2f0 msgr2=0x7f32040a9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.284+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 shutdown_connections 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.284+0000 7f3209d36640 1 -- 172.21.15.43:0/3224060295 wait complete. 2024-04-19T00:44:42.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:42.284+0000 7f3209d36640 1 librados: shutdown 2024-04-19T00:44:42.329 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-04-19T00:44:42.329 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-04-19T00:44:42.329 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:44:42.349 INFO:tasks.ceph.mon.h:Stopped 2024-04-19T00:44:42.349 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-04-19T00:44:42.349 INFO:tasks.ceph.mon.i:Sent signal 19 2024-04-19T00:44:42.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:44:57.351 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-04-19T00:44:57.351 INFO:tasks.ceph.mon.i:Sent signal 18 2024-04-19T00:44:57.351 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-04-19T00:44:57.352 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.480+0000 7f5edc25d640 10 monclient: get_monmap_and_config 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.480+0000 7f5edc25d640 10 monclient: build_initial_monmap 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.480+0000 7f5edc25d640 10 monclient: monmap: 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:44:57.485646+0000 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:44:57.485646+0000 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:44:57.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:44:57.485 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 Processor -- start 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 -- start start 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient: init 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient: _add_conns ranks=[0,2,7,9,6,8,5,10,3,4,1] 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ed4138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ed40a5b80 0x7f5ed413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed40a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed41397c0 0x7f5ed41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): picked mon.noname-i con 0x7f5ed41397c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:57.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ed40a3dc0 con 0x7f5ed4138cd0 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed40a4410 con 0x7f5ed40a5b80 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f5ed40a40a0 con 0x7f5ed41397c0 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient(hunting): authenticate will time out at 967.612885s 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ed40a5b80 0x7f5ed413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed9ff9640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5ed4138cd0 0x7f5ed41390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:60224/0 (socket says 172.21.15.43:60224) 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed9ff9640 1 -- 172.21.15.43:0/248823594 learned_addr learned my addr 172.21.15.43:0/248823594 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): get_auth_request con 0x7f5ed40a5b80 auth_method 0 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:57.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5eda7fa640 1 --2- 172.21.15.43:0/248823594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed41397c0 0x7f5ed41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5eda7fa640 10 monclient(hunting): get_auth_request con 0x7f5ed41397c0 auth_method 0 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5eda7fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5eda7fa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5eda7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2024-04-19T00:44:57.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 -- 172.21.15.43:0/248823594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed41397c0 msgr2=0x7f5ed41407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 --2- 172.21.15.43:0/248823594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed41397c0 0x7f5ed41407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 -- 172.21.15.43:0/248823594 >> v1:172.21.15.43:6789/0 conn(0x7f5ed4138cd0 legacy=0x7f5ed41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient: _finish_hunting 0 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 monclient: found mon.noname-b 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:44:57.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 -- 172.21.15.43:0/248823594 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed40a2ae0 con 0x7f5ed40a5b80 2024-04-19T00:44:57.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed97f8640 1 --2- 172.21.15.43:0/248823594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ed40a5b80 0x7f5ed413e270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80097f0 tx=0x7f5ec801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e49a552e0e36e5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:57.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 1 -- 172.21.15.43:0/248823594 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ec80b0070 con 0x7f5ed40a5b80 2024-04-19T00:44:57.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:57.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:44:57.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: dump: 2024-04-19T00:44:57.496 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:57.497 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: _finish_auth 0 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: _check_auth_tickets 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 1 -- 172.21.15.43:0/248823594 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec80aa810 con 0x7f5ed40a5b80 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:57.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 1 -- 172.21.15.43:0/248823594 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ec80a9300 con 0x7f5ed40a5b80 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 5 monclient: authenticate success, global_id 4448 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 10 monclient: get_monmap_and_config success 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5edc25d640 4 set_mon_vals no callback set 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.484+0000 7f5ed8ff7640 10 monclient: dump: 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:57.499 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: shutdown 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/248823594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ed40a5b80 msgr2=0x7f5ed413e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 --2- 172.21.15.43:0/248823594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ed40a5b80 0x7f5ed413e270 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80097f0 tx=0x7f5ec801c3a0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/248823594 shutdown_connections 2024-04-19T00:44:57.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 --2- 172.21.15.43:0/248823594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed41397c0 0x7f5ed41407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 --2- 172.21.15.43:0/248823594 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5ed40a5b80 0x7f5ed413e270 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/248823594 >> 172.21.15.43:0/248823594 conn(0x7f5ed40ab2f0 msgr2=0x7f5ed40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/248823594 shutdown_connections 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/248823594 wait complete. 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: build_initial_monmap 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: monmap: 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:44:57.501 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 librados: starting msgr at 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 librados: starting objecter 2024-04-19T00:44:57.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 Processor -- start 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- start start 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 librados: setting wanted keys 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 librados: calling monclient init 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: init 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: _reopen_session rank -1 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: _add_conns ranks=[1,7,8,4,2,5,6,3,0] 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ed40a5b80 0x7f5ed4135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): picked mon.noname-f con 0x7f5ed40a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): picked mon.noname-i con 0x7f5ed4138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5ed41397c0 0x7f5ed41d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): picked mon.noname-e con 0x7f5ed41397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:57.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed4143730 con 0x7f5ed40a5b80 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f5ed40a20a0 con 0x7f5ed4138cd0 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f5ed41438b0 con 0x7f5ed41397c0 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): _renew_subs 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient(hunting): authenticate will time out at 967.617966s 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5ed41397c0 0x7f5ed41d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:45904/0 (socket says 172.21.15.43:45904) 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 -- 172.21.15.43:0/817040660 learned_addr learned my addr 172.21.15.43:0/817040660 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 10 monclient(hunting): get_auth_request con 0x7f5ed4138cd0 auth_method 0 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:57.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): get_auth_request con 0x7f5ed41397c0 auth_method 0 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ed40a5b80 0x7f5ed4135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): get_auth_request con 0x7f5ed40a5b80 auth_method 0 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 -- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 msgr2=0x7f5ed4136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T00:44:57.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 -- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 msgr2=0x7f5ed4136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed97f8640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed9ff9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 -- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 msgr2=0x7f5ed4136140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 -- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ed40a5b80 msgr2=0x7f5ed4135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ed40a5b80 0x7f5ed4135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient: _finish_hunting 0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 monclient: found mon.noname-e 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 -- 172.21.15.43:0/817040660 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed40a3040 con 0x7f5ed41397c0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5eda7fa640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5ed41397c0 0x7f5ed41d0930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400d4b0 tx=0x7f5ec400d970 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 client.?.objecter ms_handle_connect 0x7f5ed41397c0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ec4012a60 con 0x7f5ed41397c0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: dump: 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.507 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: _finish_auth 0 2024-04-19T00:44:57.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: _check_auth_tickets 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 5 monclient: authenticate success, global_id 4508 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ec4012c60 con 0x7f5ed41397c0 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ec4012e10 con 0x7f5ed41397c0 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: _renew_subs 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ed40a3590 con 0x7f5ed41397c0 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed3fff640 4 set_mon_vals no callback set 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: _renew_subs 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:57.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ed40a0810 con 0x7f5ed41397c0 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5edc25d640 1 librados: init done 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 10 monclient: dump: 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:44:57.510 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.488+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5ec401b030 con 0x7f5ed41397c0 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5edc25d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5edc25d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5edc25d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ed40693b0 con 0x7f5ed41397c0 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ec0038e10 0x7f5ec003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f5ec4019020 con 0x7f5ed41397c0 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 10 client.4508.objecter ms_dispatch 0x7f5ed4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 3 client.4508.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 3 client.4508.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed9ff9640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ec0038e10 0x7f5ec003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed9ff9640 10 monclient: get_auth_request con 0x7f5ec0038e10 auth_method 0 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed27fc640 10 client.4508.objecter ms_handle_connect 0x7f5ec0038e10 2024-04-19T00:44:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.492+0000 7f5ed9ff9640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ec0038e10 0x7f5ec003b2d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0012f0 tx=0x7f5ecc001aa0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:44:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.496+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ec4022060 con 0x7f5ed41397c0 2024-04-19T00:44:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.496+0000 7f5ed27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:44:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.496+0000 7f5ed27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:44:57.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5edc25d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:44:57.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5edc25d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:44:57.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5edc25d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:44:57.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ed4136680 con 0x7f5ed41397c0 2024-04-19T00:44:57.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5ed27fc640 1 -- 172.21.15.43:0/817040660 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7f5ec40127d0 con 0x7f5ed41397c0 2024-04-19T00:44:57.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5ed27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:44:57.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.720+0000 7f5ed27fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:44:57.723 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,6,8],"quorum_names":["a","f","b","g","c","d","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:44:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 10 librados: watch_flush enter 2024-04-19T00:44:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 10 librados: watch_flush exit 2024-04-19T00:44:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 20 client.4508.objecter shutdown clearing up homeless session... 2024-04-19T00:44:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 10 client.4508.objecter successfully canceled tick 2024-04-19T00:44:57.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ec0038e10 msgr2=0x7f5ec003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ec0038e10 0x7f5ec003b2d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc0012f0 tx=0x7f5ecc001aa0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 10 monclient: shutdown 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5ed41397c0 msgr2=0x7f5ed41d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5ed41397c0 0x7f5ed41d0930 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400d4b0 tx=0x7f5ec400d970 comp rx=0 tx=0).stop 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 shutdown_connections 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ec0038e10 0x7f5ec003b2d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5ed41397c0 0x7f5ed41d0930 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5ed4138cd0 0x7f5ed4136140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 --2- 172.21.15.43:0/817040660 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ed40a5b80 0x7f5ed4135c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 >> 172.21.15.43:0/817040660 conn(0x7f5ed40ab2f0 msgr2=0x7f5ed4136f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 shutdown_connections 2024-04-19T00:44:57.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 -- 172.21.15.43:0/817040660 wait complete. 2024-04-19T00:44:57.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:44:57.724+0000 7f5edc25d640 1 librados: shutdown 2024-04-19T00:45:00.744 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:45:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient: get_monmap_and_config 2024-04-19T00:45:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient: build_initial_monmap 2024-04-19T00:45:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient: monmap: 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:00.880153+0000 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:00.880153+0000 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:00.879 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:00.880 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 Processor -- start 2024-04-19T00:45:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 -- start start 2024-04-19T00:45:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient: init 2024-04-19T00:45:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient: _add_conns ranks=[5,1,10,8,4,6,7,3,9,2,0] 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba34138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): picked mon.noname-h con 0x7fba34138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fba340a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fba34139370 0x7fba34138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): picked mon.noname-f con 0x7fba34139370 addr v2:172.21.15.161:3300/0 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba340a0d40 con 0x7fba340a5bb0 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fba340a1020 con 0x7fba34139370 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fba340a0780 con 0x7fba34138a50 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba3a3ff640 10 monclient(hunting): authenticate will time out at 971.007816s 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba33fff640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 msgr2=0x7fba34138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.876+0000 7fba33fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba34138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fba34139370 0x7fba34138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba337fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fba340a5bb0 0x7fba34137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:54462/0 (socket says 172.21.15.43:54462) 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba337fe640 1 -- 172.21.15.43:0/1961807154 learned_addr learned my addr 172.21.15.43:0/1961807154 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): get_auth_request con 0x7fba34139370 auth_method 0 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:00.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 1 -- 172.21.15.43:0/1961807154 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3827663012 0 0) 0x7fba340a0d40 con 0x7fba340a5bb0 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient(hunting): _negotiate 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient(hunting): my global_id is 4438 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 1 -- 172.21.15.43:0/1961807154 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba1c003530 con 0x7fba340a5bb0 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient(hunting): handle_auth_done global_id 4429 payload 274 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 -- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 msgr2=0x7fba34138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 --2- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba34138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 -- 172.21.15.43:0/1961807154 >> v1:172.21.15.161:6789/0 conn(0x7fba340a5bb0 legacy=0x7fba34137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient: _finish_hunting 0 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 monclient: found mon.noname-f 2024-04-19T00:45:00.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:00.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:45:00.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 -- 172.21.15.43:0/1961807154 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba340a0a60 con 0x7fba34139370 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3899c640 1 --2- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba34138070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009dc0 tx=0x7fba2c01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb103a194097190 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 1 -- 172.21.15.43:0/1961807154 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fba2c0a6980 con 0x7fba34139370 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: dump: 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:00.894 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:00.895 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: _finish_auth 0 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 1 -- 172.21.15.43:0/1961807154 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba2c0a6b40 con 0x7fba34139370 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 1 -- 172.21.15.43:0/1961807154 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fba2c0a6cf0 con 0x7fba34139370 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 5 monclient: authenticate success, global_id 4429 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 4 set_mon_vals no callback set 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba32ffd640 10 monclient: dump: 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:00.896 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:00.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 10 monclient: shutdown 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 -- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 msgr2=0x7fba34138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba34138070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009dc0 tx=0x7fba2c01c830 comp rx=0 tx=0).stop 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 -- 172.21.15.43:0/1961807154 shutdown_connections 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba34138070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/1961807154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba34138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 -- 172.21.15.43:0/1961807154 >> 172.21.15.43:0/1961807154 conn(0x7fba340a7cb0 msgr2=0x7fba340a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 -- 172.21.15.43:0/1961807154 shutdown_connections 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.880+0000 7fba3a3ff640 1 -- 172.21.15.43:0/1961807154 wait complete. 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: build_initial_monmap 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: monmap: 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:00.898 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:00.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 librados: starting msgr at 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 librados: starting objecter 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 Processor -- start 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- start start 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 librados: setting wanted keys 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 librados: calling monclient init 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: init 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _add_conns ranks=[7,5,1,4,6,8,2,3,0] 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): picked mon.noname-i con 0x7fba340a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba341310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): picked mon.noname-h con 0x7fba34138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba341350a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:00.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): picked mon.noname-f con 0x7fba34139370 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fba34138730 con 0x7fba34139370 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fba341385b0 con 0x7fba34138a50 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fba34143820 con 0x7fba340a5bb0 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient(hunting): authenticate will time out at 971.013369s 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba337fe640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 msgr2=0x7fba341310c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba337fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba341310c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba341350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:00.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:59854/0 (socket says 172.21.15.43:59854) 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): get_auth_request con 0x7fba34139370 auth_method 0 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 -- 172.21.15.43:0/223803574 learned_addr learned my addr 172.21.15.43:0/223803574 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 10 monclient(hunting): get_auth_request con 0x7fba340a5bb0 auth_method 0 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 -- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 msgr2=0x7fba34130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T00:45:00.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 -- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 msgr2=0x7fba34130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba33fff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient(hunting): handle_auth_done global_id 4447 payload 824 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 -- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 msgr2=0x7fba34130b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 -- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 msgr2=0x7fba341310c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba341310c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient: _finish_hunting 0 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 monclient: found mon.noname-f 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 -- 172.21.15.43:0/223803574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba3413a410 con 0x7fba34139370 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3899c640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba341350a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0ac000 tx=0x7fba2c0195d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:00.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 client.?.objecter ms_handle_connect 0x7fba34139370 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fba2c0bc070 con 0x7fba34139370 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: dump: 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:00.904 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: _finish_auth 0 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba2c003e10 con 0x7fba34139370 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 5 monclient: authenticate success, global_id 4447 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _renew_subs 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:00.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba34137450 con 0x7fba34139370 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fba2c0044b0 con 0x7fba34139370 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba31ffb640 4 set_mon_vals no callback set 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 monclient: dump: 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:00.906 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _renew_subs 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba3409fc90 con 0x7fba34139370 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 librados: init done 2024-04-19T00:45:00.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba3409ee60 con 0x7fba34139370 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fba2c004650 con 0x7fba34139370 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fba20038e00 0x7fba2003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fba2c0e3e50 con 0x7fba34139370 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 10 client.4447.objecter ms_dispatch 0x7fba34000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 3 client.4447.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 3 client.4447.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.884+0000 7fba30ff9640 20 client.4447.objecter dump_active .. 0 homeless 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.888+0000 7fba33fff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fba20038e00 0x7fba2003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.888+0000 7fba33fff640 10 monclient: get_auth_request con 0x7fba20038e00 auth_method 0 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.888+0000 7fba33fff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fba20038e00 0x7fba2003b2c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba240099f0 tx=0x7fba24008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:00.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.888+0000 7fba30ff9640 10 client.4447.objecter ms_handle_connect 0x7fba20038e00 2024-04-19T00:45:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.896+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fba2c0c5030 con 0x7fba34139370 2024-04-19T00:45:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.896+0000 7fba30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:00.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:00.896+0000 7fba30ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba3a3ff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:45:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba3a3ff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:45:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba3a3ff640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fba34076190 con 0x7fba34139370 2024-04-19T00:45:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba30ff9640 1 -- 172.21.15.43:0/223803574 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7fba2c016540 con 0x7fba34139370 2024-04-19T00:45:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:45:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.120+0000 7fba30ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:01.123 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":12,"quorum":[0,1,2,3,4,6,8],"quorum_names":["a","f","b","g","c","d","e"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:45:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 10 librados: watch_flush enter 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 10 librados: watch_flush exit 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 20 client.4447.objecter shutdown clearing up homeless session... 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 10 client.4447.objecter successfully canceled tick 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fba20038e00 msgr2=0x7fba2003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fba20038e00 0x7fba2003b2c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fba240099f0 tx=0x7fba24008040 comp rx=0 tx=0).stop 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 10 monclient: shutdown 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 msgr2=0x7fba341350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba341350a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0ac000 tx=0x7fba2c0195d0 comp rx=0 tx=0).stop 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 shutdown_connections 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fba20038e00 0x7fba2003b2c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fba34139370 0x7fba341350a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fba34138a50 0x7fba341310c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 --2- 172.21.15.43:0/223803574 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fba340a5bb0 0x7fba34130b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 >> 172.21.15.43:0/223803574 conn(0x7fba340a7cb0 msgr2=0x7fba34133b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 shutdown_connections 2024-04-19T00:45:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 -- 172.21.15.43:0/223803574 wait complete. 2024-04-19T00:45:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:01.124+0000 7fba3a3ff640 1 librados: shutdown 2024-04-19T00:45:04.141 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:45:04.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient: get_monmap_and_config 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient: build_initial_monmap 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient: monmap: 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:04.272540+0000 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:04.272540+0000 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:04.271 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:04.272 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 Processor -- start 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 -- start start 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient: init 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient: _add_conns ranks=[7,2,10,4,9,3,8,5,1,6,0] 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:04.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): picked mon.noname-i con 0x7ff764138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): picked mon.noname-b con 0x7ff7640a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): picked mon.noname-f con 0x7ff7641358b0 addr v2:172.21.15.161:3300/0 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff7640a4410 con 0x7ff7641358b0 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff764139b30 con 0x7ff7640a5b80 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff7640a3dc0 con 0x7ff764138cd0 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff76a535640 10 monclient(hunting): authenticate will time out at 974.399642s 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35710/0 (socket says 172.21.15.43:35710) 2024-04-19T00:45:04.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 learned_addr learned my addr 172.21.15.43:0/1159161774 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 10 monclient(hunting): get_auth_request con 0x7ff7640a5b80 auth_method 0 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.268+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff764138cd0 auth_method 0 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 10 monclient(hunting): get_auth_request con 0x7ff7641358b0 auth_method 0 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:04.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.272+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 10 monclient(hunting): get_auth_request con 0x7ff7640a5b80 auth_method 0 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff764138cd0 auth_method 0 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 10 monclient(hunting): get_auth_request con 0x7ff7641358b0 auth_method 0 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:45:04.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:45:04.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.476+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:45:04.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 10 monclient(hunting): get_auth_request con 0x7ff7640a5b80 auth_method 0 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff764138cd0 auth_method 0 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:04.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 10 monclient(hunting): get_auth_request con 0x7ff7641358b0 auth_method 0 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:04.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:04.880+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 10 monclient(hunting): get_auth_request con 0x7ff7641358b0 auth_method 0 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 10 monclient(hunting): get_auth_request con 0x7ff7640a5b80 auth_method 0 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:05.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff764138cd0 auth_method 0 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:45:05.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:05.684+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): tick 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 monclient(hunting): continuing hunt 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 msgr2=0x7ff7641390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff764138cd0 0x7ff7641390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 msgr2=0x7ff7641395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff7640a5b80 0x7ff7641395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 -- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 msgr2=0x7ff764135d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 --2- 172.21.15.43:0/1159161774 >> v2:172.21.15.161:3300/0 conn(0x7ff7641358b0 0x7ff764135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient: _add_conns ranks=[0,5,1,9,3,8,6,4] 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff74c003820 addr v1:172.21.15.43:6789/0 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 0x7ff74c009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): picked mon.noname-h con 0x7ff74c0073f0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff74c009f40 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 -- 172.21.15.43:0/1159161774 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7640a40a0 con 0x7ff74c003820 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 -- 172.21.15.43:0/1159161774 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff74c00ece0 con 0x7ff74c009f40 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 1 -- 172.21.15.43:0/1159161774 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ff7640a4410 con 0x7ff74c0073f0 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff761ffb640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 msgr2=0x7ff74c009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.272+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 0x7ff74c009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.476+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 msgr2=0x7ff74c009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:07.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.476+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 0x7ff74c009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:45:07.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.880+0000 7ff7637fe640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 msgr2=0x7ff74c009880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:07.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:07.880+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 0x7ff74c009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:45:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.420+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3509375678 0 0) 0x7ff74c00ece0 con 0x7ff74c009f40 2024-04-19T00:45:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.420+0000 7ff762ffd640 10 monclient(hunting): _negotiate 2024-04-19T00:45:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.420+0000 7ff762ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.420+0000 7ff762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.420+0000 7ff762ffd640 10 monclient(hunting): my global_id is 4456 2024-04-19T00:45:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.420+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff750003530 con 0x7ff74c009f40 2024-04-19T00:45:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.424+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2438871366 0 0) 0x7ff7640a40a0 con 0x7ff74c003820 2024-04-19T00:45:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.424+0000 7ff762ffd640 10 monclient(hunting): _negotiate 2024-04-19T00:45:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.424+0000 7ff762ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.424+0000 7ff762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.424+0000 7ff762ffd640 10 monclient(hunting): my global_id is 4527 2024-04-19T00:45:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.424+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff74c00ece0 con 0x7ff74c003820 2024-04-19T00:45:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 803630868 0 0) 0x7ff750003530 con 0x7ff74c009f40 2024-04-19T00:45:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 msgr2=0x7ff74c009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:08.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff74c0073f0 0x7ff74c009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 >> v1:172.21.15.43:6789/0 conn(0x7ff74c003820 legacy=0x7ff74c005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 reap_dead start 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 1 monclient: found mon.noname-f-legacy 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.161:6789/0 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff74c010030 con 0x7ff74c009f40 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.428+0000 7ff762ffd640 10 monclient: _finish_auth 0 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7ff75c009bf0 con 0x7ff74c009f40 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff76a535640 5 monclient: authenticate success, global_id 4456 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient: dump: 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.436 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] but i'm connected to v1:172.21.15.161:6789/0, reconnecting 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 >> v1:172.21.15.161:6789/0 conn(0x7ff74c009f40 legacy=0x7ff74c00c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient: _add_conns ranks=[8,7,3,4,1,2,5,6,0] 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff750004350 0x7ff750004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): picked mon.e con 0x7ff750004350 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff750004f00 0x7ff7500099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): picked mon.i con 0x7ff750004f00 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750009f00 0x7ff75000c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): picked mon.g con 0x7ff750009f00 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff7640a4410 con 0x7ff750009f00 2024-04-19T00:45:08.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff75000cac0 con 0x7ff750004f00 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ff75000cc70 con 0x7ff750004350 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff762ffd640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff76a535640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff750004350 0x7ff750004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff750004f00 0x7ff7500099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750009f00 0x7ff75000c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff750004350 auth_method 0 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:08.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): get_auth_request con 0x7ff750009f00 auth_method 0 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): get_auth_request con 0x7ff750004f00 auth_method 0 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:45:08.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient(hunting): handle_auth_done global_id 4456 payload 293 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff750004350 msgr2=0x7ff750004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff750004350 0x7ff750004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff750004f00 msgr2=0x7ff7500099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff750004f00 0x7ff7500099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient: _finish_hunting 0 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 monclient: found mon.g 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 1 -- 172.21.15.43:0/1159161774 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff750003530 con 0x7ff750009f00 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient: _finish_auth 0 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 10 monclient: _check_auth_tickets 2024-04-19T00:45:08.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff768ad2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:08.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.432+0000 7ff7637fe640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff750004f00 0x7ff7500099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff768ad2640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750009f00 0x7ff75000c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7640a7450 tx=0x7ff75c01d7a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=bf2f55bdfc91d438 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75c007350 con 0x7ff750009f00 2024-04-19T00:45:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: dump: 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75c0b1070 con 0x7ff750009f00 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff76a535640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff76a535640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff76a535640 4 set_mon_vals no callback set 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 1 -- 172.21.15.43:0/1159161774 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75c0acd10 con 0x7ff750009f00 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.436+0000 7ff762ffd640 10 monclient: dump: 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 10 monclient: shutdown 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 -- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750009f00 msgr2=0x7ff75000c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750009f00 0x7ff75000c2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7640a7450 tx=0x7ff75c01d7a0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 -- 172.21.15.43:0/1159161774 shutdown_connections 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750009f00 0x7ff75000c2f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff750004f00 0x7ff7500099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 --2- 172.21.15.43:0/1159161774 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff750004350 0x7ff750004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 -- 172.21.15.43:0/1159161774 >> 172.21.15.43:0/1159161774 conn(0x7ff7640ab2f0 msgr2=0x7ff7640ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 -- 172.21.15.43:0/1159161774 shutdown_connections 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 -- 172.21.15.43:0/1159161774 wait complete. 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 10 monclient: build_initial_monmap 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 10 monclient: monmap: 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:08.451 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 librados: starting msgr at 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 librados: starting objecter 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 Processor -- start 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 -- start start 2024-04-19T00:45:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 librados: setting wanted keys 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 1 librados: calling monclient init 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 10 monclient: init 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.440+0000 7ff76a535640 10 monclient: _add_conns ranks=[8,2,3,1,7,5,0,6,4] 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 0x7ff7641d0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): picked mon.noname-e con 0x7ff74c009f40 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff750004350 0x7ff764131840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): picked mon.noname-b con 0x7ff750004350 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750004f00 0x7ff764131d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): picked mon.noname-g con 0x7ff750004f00 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff764135cc0 con 0x7ff750004350 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff7640a20a0 con 0x7ff750004f00 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ff764135e40 con 0x7ff74c009f40 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient(hunting): authenticate will time out at 978.572155s 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 0x7ff7641d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff750004350 0x7ff764131840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 0x7ff7641d0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:34968/0 (socket says 172.21.15.43:34968) 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff750004350 0x7ff764131840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35746/0 (socket says 172.21.15.43:35746) 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 -- 172.21.15.43:0/399011831 learned_addr learned my addr 172.21.15.43:0/399011831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): get_auth_request con 0x7ff74c009f40 auth_method 0 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): get_auth_request con 0x7ff750004350 auth_method 0 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750004f00 0x7ff764131d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): get_auth_request con 0x7ff750004f00 auth_method 0 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff768ad2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient(hunting): handle_auth_done global_id 4517 payload 824 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 -- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750004f00 msgr2=0x7ff764131d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750004f00 0x7ff764131d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 -- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff750004350 msgr2=0x7ff764131840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff750004350 0x7ff764131840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient: _finish_hunting 0 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 monclient: found mon.noname-e 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 -- 172.21.15.43:0/399011831 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7640a1240 con 0x7ff74c009f40 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff763fff640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 0x7ff7641d0330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7580167f0 tx=0x7ff758016bf0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 client.?.objecter ms_handle_connect 0x7ff74c009f40 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff75800f040 con 0x7ff74c009f40 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: dump: 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: _finish_auth 0 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:45:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75801b030 con 0x7ff74c009f40 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 5 monclient: authenticate success, global_id 4517 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient: _renew_subs 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff764122c80 con 0x7ff74c009f40 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff758016ea0 con 0x7ff74c009f40 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 monclient: dump: 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient: _renew_subs 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7627fc640 4 set_mon_vals no callback set 2024-04-19T00:45:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7640a1920 con 0x7ff74c009f40 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 librados: init done 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff764068240 con 0x7ff74c009f40 2024-04-19T00:45:08.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff758004240 con 0x7ff74c009f40 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff74c03a270 0x7ff74c03c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7ff758049b90 con 0x7ff74c009f40 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 10 client.4517.objecter ms_dispatch 0x7ff764000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 3 client.4517.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 3 client.4517.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff760ff9640 20 client.4517.objecter dump_active .. 0 homeless 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff74c03a270 0x7ff74c03c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.444+0000 7ff7637fe640 10 monclient: get_auth_request con 0x7ff74c03a270 auth_method 0 2024-04-19T00:45:08.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.448+0000 7ff7637fe640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff74c03a270 0x7ff74c03c730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff754004660 tx=0x7ff754008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.448+0000 7ff760ff9640 10 client.4517.objecter ms_handle_connect 0x7ff74c03a270 2024-04-19T00:45:08.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.452+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff758014930 con 0x7ff74c009f40 2024-04-19T00:45:08.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.452+0000 7ff760ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:08.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.452+0000 7ff760ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff76a535640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:45:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff76a535640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:45:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff76a535640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:45:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff7640693b0 con 0x7ff74c009f40 2024-04-19T00:45:08.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff760ff9640 1 -- 172.21.15.43:0/399011831 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7ff758017d20 con 0x7ff74c009f40 2024-04-19T00:45:08.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff760ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:45:08.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff760ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:08.690 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":18,"quorum":[0,1,2,3,4,6,7,8],"quorum_names":["a","f","b","g","c","d","i","e"],"quorum_leader_name":"e","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.688+0000 7ff76a535640 10 librados: watch_flush enter 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 10 librados: watch_flush exit 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 20 client.4517.objecter shutdown clearing up homeless session... 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 10 client.4517.objecter successfully canceled tick 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff74c03a270 msgr2=0x7ff74c03c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff74c03a270 0x7ff74c03c730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff754004660 tx=0x7ff754008040 comp rx=0 tx=0).stop 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 10 monclient: shutdown 2024-04-19T00:45:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 msgr2=0x7ff7641d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 0x7ff7641d0330 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7580167f0 tx=0x7ff758016bf0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 shutdown_connections 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff750004f00 0x7ff764131d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff750004350 0x7ff764131840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff74c03a270 0x7ff74c03c730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 --2- 172.21.15.43:0/399011831 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff74c009f40 0x7ff7641d0330 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 >> 172.21.15.43:0/399011831 conn(0x7ff7640ab2f0 msgr2=0x7ff7641368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 shutdown_connections 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 -- 172.21.15.43:0/399011831 wait complete. 2024-04-19T00:45:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.692+0000 7ff76a535640 1 librados: shutdown 2024-04-19T00:45:08.710 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-04-19T00:45:08.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-04-19T00:45:08.710 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:45:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.836+0000 7f8dec5d4640 10 monclient: get_monmap_and_config 2024-04-19T00:45:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.836+0000 7f8dec5d4640 10 monclient: build_initial_monmap 2024-04-19T00:45:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.836+0000 7f8dec5d4640 10 monclient: monmap: 2024-04-19T00:45:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:08.841197+0000 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:08.841197+0000 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:08.841 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:08.841 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:08.841 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.836+0000 7f8dec5d4640 1 Processor -- start 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.836+0000 7f8dec5d4640 1 -- start start 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.836+0000 7f8dec5d4640 10 monclient: init 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient: _add_conns ranks=[2,6,5,0,9,4,3,8,7,10,1] 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 0x7f8de41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): picked mon.noname-b con 0x7f8de4138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 0x7f8de41395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): picked mon.noname-d con 0x7f8de40a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8de41358b0 0x7f8de4135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): picked mon.noname-h con 0x7f8de41358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:08.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8de40a4410 con 0x7f8de4138cd0 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8de4139b30 con 0x7f8de41358b0 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8de40a3dc0 con 0x7f8de40a5b80 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient(hunting): authenticate will time out at 978.968386s 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8deab71640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8de41358b0 msgr2=0x7f8de4135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 0x7f8de41395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8deab71640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8de41358b0 0x7f8de4135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 0x7f8de41390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 0x7f8de41390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35752/0 (socket says 172.21.15.43:35752) 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 0x7f8de41395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:41186/0 (socket says 172.21.15.43:41186) 2024-04-19T00:45:08.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 -- 172.21.15.43:0/2194861570 learned_addr learned my addr 172.21.15.43:0/2194861570 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): get_auth_request con 0x7f8de4138cd0 auth_method 0 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): get_auth_request con 0x7f8de40a5b80 auth_method 0 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient(hunting): handle_auth_done global_id 4466 payload 274 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 -- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 msgr2=0x7f8de41395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 0x7f8de41395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 -- 172.21.15.43:0/2194861570 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8de41358b0 msgr2=0x7f8de4135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8de41358b0 0x7f8de4135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient: _finish_hunting 0 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 monclient: found mon.noname-b 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 -- 172.21.15.43:0/2194861570 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de40a40a0 con 0x7f8de4138cd0 2024-04-19T00:45:08.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de9b6f640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 0x7f8de41395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:08.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dea370640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 0x7f8de41390b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8de00098a0 tx=0x7f8de001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83a4471d9de2f99a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 1 -- 172.21.15.43:0/2194861570 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8de00b0070 con 0x7f8de4138cd0 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: dump: 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.859 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: _finish_auth 0 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: _check_auth_tickets 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 1 -- 172.21.15.43:0/2194861570 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de00aa810 con 0x7f8de4138cd0 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 5 monclient: authenticate success, global_id 4466 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:08.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 1 -- 172.21.15.43:0/2194861570 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8de00a9300 con 0x7f8de4138cd0 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 4 set_mon_vals no callback set 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8de936e640 10 monclient: dump: 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.861 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 10 monclient: shutdown 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 -- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 msgr2=0x7f8de41390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 0x7f8de41390b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8de00098a0 tx=0x7f8de001c400 comp rx=0 tx=0).stop 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 -- 172.21.15.43:0/2194861570 shutdown_connections 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8de41358b0 0x7f8de4135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.840+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de40a5b80 0x7f8de41395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/2194861570 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8de4138cd0 0x7f8de41390b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- 172.21.15.43:0/2194861570 >> 172.21.15.43:0/2194861570 conn(0x7f8de40ab2f0 msgr2=0x7f8de40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- 172.21.15.43:0/2194861570 shutdown_connections 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- 172.21.15.43:0/2194861570 wait complete. 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: build_initial_monmap 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: monmap: 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 librados: starting msgr at 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 librados: starting objecter 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 Processor -- start 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- start start 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 librados: setting wanted keys 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 librados: calling monclient init 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: init 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: _add_conns ranks=[4,8,6,0,2,3,7,1,5] 2024-04-19T00:45:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8de40a5b80 0x7f8de4134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): picked mon.noname-c con 0x7f8de40a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8de41358b0 0x7f8de4135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): picked mon.noname-e con 0x7f8de41358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de4138cd0 0x7f8de41d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): picked mon.noname-d con 0x7f8de4138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8de40a2010 con 0x7f8de40a5b80 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8de40a1e90 con 0x7f8de4138cd0 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8de4142aa0 con 0x7f8de41358b0 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): authenticate will time out at 978.973511s 2024-04-19T00:45:08.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8de40a5b80 0x7f8de4134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8de40a5b80 0x7f8de4134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44586/0 (socket says 172.21.15.43:44586) 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 -- 172.21.15.43:0/237265266 learned_addr learned my addr 172.21.15.43:0/237265266 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): get_auth_request con 0x7f8de40a5b80 auth_method 0 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de4138cd0 0x7f8de41d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): get_auth_request con 0x7f8de4138cd0 auth_method 0 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8de41358b0 0x7f8de4135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): get_auth_request con 0x7f8de41358b0 auth_method 0 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8deab71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient(hunting): handle_auth_done global_id 4450 payload 824 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8de41358b0 msgr2=0x7f8de4135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8de41358b0 0x7f8de4135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de4138cd0 msgr2=0x7f8de41d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8de4138cd0 0x7f8de41d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient: _finish_hunting 0 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 monclient: found mon.noname-c 2024-04-19T00:45:08.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de413e8f0 con 0x7f8de40a5b80 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dea370640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8de40a5b80 0x7f8de4134bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8de001c230 tx=0x7f8de0019200 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 client.?.objecter ms_handle_connect 0x7f8de40a5b80 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8de00b0040 con 0x7f8de40a5b80 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: dump: 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.868 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: _finish_auth 0 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de00b9070 con 0x7f8de40a5b80 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8de00044a0 con 0x7f8de40a5b80 2024-04-19T00:45:08.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 5 monclient: authenticate success, global_id 4450 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: _renew_subs 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de40a3590 con 0x7f8de40a5b80 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 monclient: dump: 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.870 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de8b6d640 4 set_mon_vals no callback set 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: _renew_subs 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de413c600 con 0x7f8de40a5b80 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 librados: init done 2024-04-19T00:45:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 0x7f8de41d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8de406d140 con 0x7f8de413e470 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 0x7f8de41d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient: get_auth_request con 0x7f8de413e470 auth_method 0 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8de0004030 con 0x7f8de40a5b80 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_done global_id 4536 payload 824 2024-04-19T00:45:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8de9b6f640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8de4068240 con 0x7f8de413e470 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8dc8038e00 0x7f8dc803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f8de00e2910 con 0x7f8de40a5b80 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 10 client.4450.objecter ms_dispatch 0x7f8de4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 3 client.4450.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.844+0000 7f8ddaffd640 3 client.4450.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 20 client.4450.objecter dump_active .. 0 homeless 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 client.4450.objecter ms_handle_connect 0x7f8de413e470 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8de9b6f640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 0x7f8de41d1070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8de40a7360 tx=0x7f8dd4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 client.4450.objecter resend_mon_ops 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8deab71640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8dc8038e00 0x7f8dc803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8deab71640 10 monclient: get_auth_request con 0x7f8dc8038e00 auth_method 0 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dd400fb30 con 0x7f8de413e470 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 monclient: dump: 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.874 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8dd400f6c0 con 0x7f8de413e470 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 msgr2=0x7f8de41d1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 0x7f8de41d1070 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8de40a7360 tx=0x7f8dd4011a60 comp rx=0 tx=0).stop 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 msgr2=0x7f8de41d1070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de413e470 0x7f8de41d1070 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8de40a7360 tx=0x7f8dd4011a60 comp rx=0 tx=0).stop 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8deab71640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8dc8038e00 0x7f8dc803b2c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8de41d1bf0 tx=0x7f8ddc0095f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.848+0000 7f8ddaffd640 10 client.4450.objecter ms_handle_connect 0x7f8dc8038e00 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8dec5d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 0x7f8de409da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8dec5d4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8de41d19c0 con 0x7f8de41d15b0 2024-04-19T00:45:08.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 0x7f8de409da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient: get_auth_request con 0x7f8de41d15b0 auth_method 0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 10 monclient(hunting): handle_auth_done global_id 4545 payload 824 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 1 -- 172.21.15.43:0/237265266 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8de40693b0 con 0x7f8de41d15b0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 client.4450.objecter ms_handle_connect 0x7f8de41d15b0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 client.4450.objecter resend_mon_ops 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8de9b6f640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 0x7f8de409da80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8de40a7360 tx=0x7f8dd40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dd4026070 con 0x7f8de41d15b0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:08.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 monclient: dump: 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:08.877 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f8dd4022570 con 0x7f8de41d15b0 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.852+0000 7f8ddaffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 msgr2=0x7f8de409da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8ddaffd640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 0x7f8de409da80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8de40a7360 tx=0x7f8dd40046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8ddaffd640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 msgr2=0x7f8de409da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8ddaffd640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8de41d15b0 0x7f8de409da80 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8de40a7360 tx=0x7f8dd40046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.878 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 52347, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:08.879 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:08.880 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:08.881 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:08.882 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:08.883 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:08.885 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.888 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:08.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:08.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:08.897 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:08.898 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:08.899 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:08.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 10 librados: watch_flush enter 2024-04-19T00:45:08.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 10 librados: watch_flush exit 2024-04-19T00:45:08.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 20 client.4450.objecter shutdown clearing up homeless session... 2024-04-19T00:45:08.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 10 client.4450.objecter successfully canceled tick 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8dc8038e00 msgr2=0x7f8dc803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8dc8038e00 0x7f8dc803b2c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8de41d1bf0 tx=0x7f8ddc0095f0 comp rx=0 tx=0).stop 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 10 monclient: shutdown 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8de40a5b80 msgr2=0x7f8de4134bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 --2- 172.21.15.43:0/237265266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8de40a5b80 0x7f8de4134bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8de001c230 tx=0x7f8de0019200 comp rx=0 tx=0).stop 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8deab71640 1 -- 172.21.15.43:0/237265266 reap_dead start 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8deab71640 1 -- 172.21.15.43:0/237265266 reap_dead start 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 shutdown_connections 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 >> 172.21.15.43:0/237265266 conn(0x7f8de40ab2f0 msgr2=0x7f8de41368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 shutdown_connections 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 -- 172.21.15.43:0/237265266 wait complete. 2024-04-19T00:45:08.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:08.856+0000 7f8dec5d4640 1 librados: shutdown 2024-04-19T00:45:08.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:08.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-04-19T00:45:08.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient: get_monmap_and_config 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient: monmap: 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:09.026861+0000 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:09.026861+0000 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.025 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:09.026 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 Processor -- start 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 -- start start 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient: init 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient: _add_conns ranks=[7,3,1,10,8,4,5,6,2,9,0] 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 0x7f32a4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): picked mon.noname-i con 0x7f32a4138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 0x7f32a4139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): picked mon.noname-g con 0x7f32a40a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f32a4137960 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32a40a0d40 con 0x7f32a4137960 2024-04-19T00:45:09.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f32a40a1020 con 0x7f32a40a5bb0 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f32a41398b0 con 0x7f32a4138a50 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a91fa640 10 monclient(hunting): authenticate will time out at 979.154181s 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 0x7f32a4139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 0x7f32a4138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 0x7f32a4138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:59900/0 (socket says 172.21.15.43:59900) 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 0x7f32a4139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:35792/0 (socket says 172.21.15.43:35792) 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 -- 172.21.15.43:0/2901477716 learned_addr learned my addr 172.21.15.43:0/2901477716 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): get_auth_request con 0x7f32a4138a50 auth_method 0 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a359e640 1 --1- 172.21.15.43:0/2901477716 >> v1:172.21.15.161:6789/0 conn(0x7f32a4137960 0x7f32a4137dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:38962/0 (socket says 172.21.15.43:38962) 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): get_auth_request con 0x7f32a40a5bb0 auth_method 0 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_done global_id 4552 payload 274 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 -- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 msgr2=0x7f32a4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 --2- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 0x7f32a4139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 -- 172.21.15.43:0/2901477716 >> v1:172.21.15.161:6789/0 conn(0x7f32a4137960 legacy=0x7f32a4137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 monclient: found mon.noname-i 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a2d9d640 1 -- 172.21.15.43:0/2901477716 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a40a0a60 con 0x7f32a4138a50 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a1d9b640 1 -- 172.21.15.43:0/2901477716 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313285987 0 0) 0x7f32a40a0d40 con 0x7f32a4137960 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a259c640 1 --2- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 0x7f32a4139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.024+0000 7f32a1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:45:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a2d9d640 1 --2- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 0x7f32a4138e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f329c009e80 tx=0x7f329c01c910 comp rx=0 tx=0).ready entity=mon.7 client_cookie=282183ad40124378 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 1 -- 172.21.15.43:0/2901477716 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f329c0090d0 con 0x7f32a4138a50 2024-04-19T00:45:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:45:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: dump: 2024-04-19T00:45:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 1 -- 172.21.15.43:0/2901477716 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329c0a5b10 con 0x7f32a4138a50 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 5 monclient: authenticate success, global_id 4552 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 4 set_mon_vals no callback set 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 1 -- 172.21.15.43:0/2901477716 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f329c0a5cc0 con 0x7f32a4138a50 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a1d9b640 10 monclient: dump: 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.041 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: shutdown 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 msgr2=0x7f32a4138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 --2- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 0x7f32a4138e30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f329c009e80 tx=0x7f329c01c910 comp rx=0 tx=0).stop 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- 172.21.15.43:0/2901477716 shutdown_connections 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 --2- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32a40a5bb0 0x7f32a4139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 --2- 172.21.15.43:0/2901477716 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f32a4138a50 0x7f32a4138e30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- 172.21.15.43:0/2901477716 >> 172.21.15.43:0/2901477716 conn(0x7f32a40a7cb0 msgr2=0x7f32a40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- 172.21.15.43:0/2901477716 shutdown_connections 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- 172.21.15.43:0/2901477716 wait complete. 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: monmap: 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 librados: starting msgr at 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 librados: starting objecter 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 Processor -- start 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- start start 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 librados: setting wanted keys 2024-04-19T00:45:09.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 librados: calling monclient init 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: init 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient: _add_conns ranks=[6,8,1,3,5,0,4,7,2] 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32a40a5bb0 0x7f32a41367f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): picked mon.noname-d con 0x7f32a40a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f32a4137960 0x7f32a412e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): picked mon.noname-e con 0x7f32a4137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a4138a50 0x7f32a412ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): picked mon.noname-f con 0x7f32a4138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32a4143bb0 con 0x7f32a4138a50 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f32a4143a30 con 0x7f32a40a5bb0 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f32a4143d30 con 0x7f32a4137960 2024-04-19T00:45:09.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.028+0000 7f32a91fa640 10 monclient(hunting): authenticate will time out at 979.159952s 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32a40a5bb0 0x7f32a41367f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f32a4137960 0x7f32a412e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32a40a5bb0 0x7f32a41367f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:41200/0 (socket says 172.21.15.43:41200) 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 -- 172.21.15.43:0/227223704 learned_addr learned my addr 172.21.15.43:0/227223704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): get_auth_request con 0x7f32a40a5bb0 auth_method 0 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): get_auth_request con 0x7f32a4137960 auth_method 0 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a4138a50 0x7f32a412ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): get_auth_request con 0x7f32a4138a50 auth_method 0 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f32a4137960 msgr2=0x7f32a412e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f32a4137960 0x7f32a412e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a4138a50 msgr2=0x7f32a412ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32a4138a50 0x7f32a412ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 monclient: found mon.noname-d 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32a4135d60 con 0x7f32a40a5bb0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a2d9d640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32a40a5bb0 0x7f32a41367f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f329c009040 tx=0x7f329c0196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 client.?.objecter ms_handle_connect 0x7f32a40a5bb0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f329c0bc020 con 0x7f32a40a5bb0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: dump: 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f329c009210 con 0x7f32a40a5bb0 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 5 monclient: authenticate success, global_id 4371 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 monclient: _renew_subs 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32a41348b0 con 0x7f32a40a5bb0 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f329c003bb0 con 0x7f32a40a5bb0 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a0d99640 4 set_mon_vals no callback set 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: dump: 2024-04-19T00:45:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.051 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:09.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 monclient: _renew_subs 2024-04-19T00:45:09.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f32a409fc90 con 0x7f32a40a5bb0 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 1 librados: init done 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f329c0bf020 con 0x7f32a40a5bb0 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 0x7f32a40a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32a409e990 con 0x7f32a412f350 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 0x7f32a40a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient: get_auth_request con 0x7f32a412f350 auth_method 0 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3290038e10 0x7f329003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f329c0b3040 con 0x7f32a40a5bb0 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 client.4371.objecter ms_dispatch 0x7f32a4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 3 client.4371.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 3 client.4371.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 20 client.4371.objecter dump_active .. 0 homeless 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3290038e10 0x7f329003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 10 monclient: get_auth_request con 0x7f3290038e10 auth_method 0 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 10 monclient(hunting): handle_auth_done global_id 4475 payload 824 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32a412f730 con 0x7f32a412f350 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 client.4371.objecter ms_handle_connect 0x7f32a412f350 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 client.4371.objecter resend_mon_ops 2024-04-19T00:45:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a259c640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 0x7f32a40a3820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32980119a0 tx=0x7f3298011e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f329800fc20 con 0x7f32a412f350 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: dump: 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 client.4371.objecter ms_handle_connect 0x7f3290038e10 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f32a359e640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3290038e10 0x7f329003b2d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3294007fd0 tx=0x7f3294007e50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f329800fc20 con 0x7f32a412f350 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 msgr2=0x7f32a40a3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 0x7f32a40a3820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32980119a0 tx=0x7f3298011e60 comp rx=0 tx=0).stop 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 msgr2=0x7f32a40a3820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.032+0000 7f328b7fe640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a412f350 0x7f32a40a3820 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32980119a0 tx=0x7f3298011e60 comp rx=0 tx=0).stop 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 0x7f32a40a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32a406d140 con 0x7f32a40a3d60 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 0x7f32a40a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient: get_auth_request con 0x7f32a40a3d60 auth_method 0 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 10 monclient(hunting): handle_auth_done global_id 4484 payload 824 2024-04-19T00:45:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 1 -- 172.21.15.43:0/227223704 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f32a409ee60 con 0x7f32a40a3d60 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 client.4371.objecter ms_handle_connect 0x7f32a40a3d60 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 client.4371.objecter resend_mon_ops 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a259c640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 0x7f32a40a4170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32980119a0 tx=0x7f32980046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f329800fe90 con 0x7f32a40a3d60 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 monclient: dump: 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f3298022140 con 0x7f32a40a3d60 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 msgr2=0x7f32a40a4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 0x7f32a40a4170 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32980119a0 tx=0x7f32980046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 msgr2=0x7f32a40a4170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f328b7fe640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32a40a3d60 0x7f32a40a4170 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32980119a0 tx=0x7f32980046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 52539, 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:09.062 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.063 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.064 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:09.065 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.066 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.067 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.071 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:09.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.081 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:09.082 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 10 librados: watch_flush enter 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 10 librados: watch_flush exit 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 20 client.4371.objecter shutdown clearing up homeless session... 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 10 client.4371.objecter successfully canceled tick 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3290038e10 msgr2=0x7f329003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3290038e10 0x7f329003b2d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3294007fd0 tx=0x7f3294007e50 comp rx=0 tx=0).stop 2024-04-19T00:45:09.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 10 monclient: shutdown 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32a40a5bb0 msgr2=0x7f32a41367f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 --2- 172.21.15.43:0/227223704 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32a40a5bb0 0x7f32a41367f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f329c009040 tx=0x7f329c0196b0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a359e640 1 -- 172.21.15.43:0/227223704 reap_dead start 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a359e640 1 -- 172.21.15.43:0/227223704 reap_dead start 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 shutdown_connections 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 >> 172.21.15.43:0/227223704 conn(0x7f32a40a7cb0 msgr2=0x7f32a40a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 shutdown_connections 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 -- 172.21.15.43:0/227223704 wait complete. 2024-04-19T00:45:09.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.040+0000 7f32a91fa640 1 librados: shutdown 2024-04-19T00:45:09.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:09.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-04-19T00:45:09.086 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:45:09.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient: get_monmap_and_config 2024-04-19T00:45:09.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient: monmap: 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:09.215250+0000 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:09.215250+0000 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.214 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:09.215 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:09.215 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 Processor -- start 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 -- start start 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient: init 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient: _add_conns ranks=[1,9,8,10,3,6,2,0,4,7,5] 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa610138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 0x7fa61013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fa610139540 addr v2:172.21.15.43:3300/0 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6100a5bb0 0x7fa610140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): picked mon.noname-e con 0x7fa6100a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:09.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6100a0d40 con 0x7fa610138a50 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fa6100a1020 con 0x7fa610139540 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa6100a0780 con 0x7fa6100a5bb0 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa614ab3640 10 monclient(hunting): authenticate will time out at 979.343273s 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 0x7fa61013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6100a5bb0 0x7fa610140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 0x7fa61013dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39016/0 (socket says 172.21.15.43:39016) 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 1 -- 172.21.15.43:0/2416306759 learned_addr learned my addr 172.21.15.43:0/2416306759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): get_auth_request con 0x7fa610139540 auth_method 0 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): get_auth_request con 0x7fa6100a5bb0 auth_method 0 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_done global_id 4544 payload 274 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 1 -- 172.21.15.43:0/2416306759 >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 msgr2=0x7fa61013dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 1 --2- 172.21.15.43:0/2416306759 >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 0x7fa61013dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 1 -- 172.21.15.43:0/2416306759 >> v1:172.21.15.161:6789/0 conn(0x7fa610138a50 legacy=0x7fa610138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 1 monclient: found mon.noname-e 2024-04-19T00:45:09.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:45:09.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60ed9d640 1 -- 172.21.15.43:0/2416306759 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6100a0a60 con 0x7fa6100a5bb0 2024-04-19T00:45:09.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.212+0000 7fa60dd9b640 1 --2- 172.21.15.43:0/2416306759 >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 0x7fa61013dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:09.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60ed9d640 1 --2- 172.21.15.43:0/2416306759 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6100a5bb0 0x7fa610140530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097c0 tx=0x7fa60801c3f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3e40a3a7126ec813 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 1 -- 172.21.15.43:0/2416306759 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6080b0070 con 0x7fa6100a5bb0 2024-04-19T00:45:09.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: dump: 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 1 -- 172.21.15.43:0/2416306759 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6080aa810 con 0x7fa6100a5bb0 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 5 monclient: authenticate success, global_id 4544 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 1 -- 172.21.15.43:0/2416306759 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6080a9300 con 0x7fa6100a5bb0 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 4 set_mon_vals no callback set 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa60d59a640 10 monclient: dump: 2024-04-19T00:45:09.228 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 10 monclient: shutdown 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 -- 172.21.15.43:0/2416306759 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6100a5bb0 msgr2=0x7fa610140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 --2- 172.21.15.43:0/2416306759 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6100a5bb0 0x7fa610140530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097c0 tx=0x7fa60801c3f0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 -- 172.21.15.43:0/2416306759 shutdown_connections 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 --2- 172.21.15.43:0/2416306759 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6100a5bb0 0x7fa610140530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 --2- 172.21.15.43:0/2416306759 >> v2:172.21.15.43:3300/0 conn(0x7fa610139540 0x7fa61013dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 -- 172.21.15.43:0/2416306759 >> 172.21.15.43:0/2416306759 conn(0x7fa6100a7cb0 msgr2=0x7fa6100a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 -- 172.21.15.43:0/2416306759 shutdown_connections 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 -- 172.21.15.43:0/2416306759 wait complete. 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 10 monclient: monmap: 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.230 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.231 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 librados: starting msgr at 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 librados: starting objecter 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 Processor -- start 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 -- start start 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 librados: setting wanted keys 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 1 librados: calling monclient init 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.216+0000 7fa614ab3640 10 monclient: init 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient: _add_conns ranks=[0,5,6,2,8,3,4,1,7] 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6100a5bb0 0x7fa61013ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): picked mon.noname-a con 0x7fa6100a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa610138a50 0x7fa61013d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): picked mon.noname-h con 0x7fa610138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa610139540 0x7fa6101d0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): picked mon.noname-d con 0x7fa610139540 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa610143620 con 0x7fa6100a5bb0 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa6101434a0 con 0x7fa610138a50 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa6101437a0 con 0x7fa610139540 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): authenticate will time out at 979.348376s 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa610138a50 msgr2=0x7fa61013d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6100a5bb0 0x7fa61013ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa610138a50 0x7fa61013d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6100a5bb0 0x7fa61013ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39030/0 (socket says 172.21.15.43:39030) 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa610139540 0x7fa6101d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 -- 172.21.15.43:0/4020825001 learned_addr learned my addr 172.21.15.43:0/4020825001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): get_auth_request con 0x7fa6100a5bb0 auth_method 0 2024-04-19T00:45:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): get_auth_request con 0x7fa610139540 auth_method 0 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient(hunting): handle_auth_done global_id 4563 payload 824 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa610139540 msgr2=0x7fa6101d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa610139540 0x7fa6101d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa610138a50 msgr2=0x7fa61013d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa610138a50 0x7fa61013d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 monclient: found mon.noname-a 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa61009ee60 con 0x7fa6100a5bb0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60e59c640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6100a5bb0 0x7fa61013ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa60000c870 tx=0x7fa60000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 client.?.objecter ms_handle_connect 0x7fa6100a5bb0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6000018a0 con 0x7fa6100a5bb0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: dump: 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.235 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa600003df0 con 0x7fa6100a5bb0 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa60000acc0 con 0x7fa6100a5bb0 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 5 monclient: authenticate success, global_id 4563 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient: _renew_subs 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa61013e3a0 con 0x7fa6100a5bb0 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60cd99640 4 set_mon_vals no callback set 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: dump: 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient: _renew_subs 2024-04-19T00:45:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa610121a30 con 0x7fa6100a5bb0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 librados: init done 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa60000a850 con 0x7fa6100a5bb0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 0x7fa61002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa610121c00 con 0x7fa61013d9c0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa5f0038e40 0x7fa5f003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 0x7fa61002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient: get_auth_request con 0x7fa61013d9c0 auth_method 0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fa600048ad0 con 0x7fa6100a5bb0 2024-04-19T00:45:09.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 client.4563.objecter ms_dispatch 0x7fa610000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 3 client.4563.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 3 client.4563.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 20 client.4563.objecter dump_active .. 0 homeless 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa5f0038e40 0x7fa5f003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 10 monclient: get_auth_request con 0x7fa5f0038e40 auth_method 0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_done global_id 4459 payload 824 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa61009e190 con 0x7fa61013d9c0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 client.4563.objecter ms_handle_connect 0x7fa61013d9c0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60dd9b640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 0x7fa61002c450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa61013c800 tx=0x7fa60400ec00 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 client.4563.objecter resend_mon_ops 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6040044a0 con 0x7fa61013d9c0 2024-04-19T00:45:09.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: dump: 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.241 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 client.4563.objecter ms_handle_connect 0x7fa5f0038e40 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa60ed9d640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa5f0038e40 0x7fa5f003b300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097c0 tx=0x7fa6080a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa604004030 con 0x7fa61013d9c0 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 msgr2=0x7fa61002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 0x7fa61002c450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa61013c800 tx=0x7fa60400ec00 comp rx=0 tx=0).stop 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 msgr2=0x7fa61002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.220+0000 7fa5feffd640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61013d9c0 0x7fa61002c450 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa61013c800 tx=0x7fa60400ec00 comp rx=0 tx=0).stop 2024-04-19T00:45:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 0x7fa61002cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa61006d140 con 0x7fa61002c990 2024-04-19T00:45:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 0x7fa61002cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient: get_auth_request con 0x7fa61002c990 auth_method 0 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 10 monclient(hunting): handle_auth_done global_id 4468 payload 824 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 1 -- 172.21.15.43:0/4020825001 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa610076190 con 0x7fa61002c990 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60dd9b640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 0x7fa61002cda0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa604018000 tx=0x7fa604004740 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 client.4563.objecter ms_handle_connect 0x7fa61002c990 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 client.4563.objecter resend_mon_ops 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa604003af0 con 0x7fa61002c990 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:45:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 monclient: dump: 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7fa604003680 con 0x7fa61002c990 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 msgr2=0x7fa61002cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 0x7fa61002cda0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa604018000 tx=0x7fa604004740 comp rx=0 tx=0).stop 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 msgr2=0x7fa61002cda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa5feffd640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa61002c990 0x7fa61002cda0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa604018000 tx=0x7fa604004740 comp rx=0 tx=0).stop 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.247 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 52747, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.248 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:09.249 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:09.250 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:09.251 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.252 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:09.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.255 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:09.261 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.262 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.265 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.268 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.269 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.270 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.270 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:09.270 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:09.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 10 librados: watch_flush enter 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 10 librados: watch_flush exit 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 20 client.4563.objecter shutdown clearing up homeless session... 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 10 client.4563.objecter successfully canceled tick 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa5f0038e40 msgr2=0x7fa5f003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa5f0038e40 0x7fa5f003b300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097c0 tx=0x7fa6080a4f90 comp rx=0 tx=0).stop 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 10 monclient: shutdown 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6100a5bb0 msgr2=0x7fa61013ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 --2- 172.21.15.43:0/4020825001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6100a5bb0 0x7fa61013ef00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa60000c870 tx=0x7fa60000cd30 comp rx=0 tx=0).stop 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60ed9d640 1 -- 172.21.15.43:0/4020825001 reap_dead start 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa60ed9d640 1 -- 172.21.15.43:0/4020825001 reap_dead start 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 shutdown_connections 2024-04-19T00:45:09.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 >> 172.21.15.43:0/4020825001 conn(0x7fa6100a7cb0 msgr2=0x7fa6100a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 shutdown_connections 2024-04-19T00:45:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 -- 172.21.15.43:0/4020825001 wait complete. 2024-04-19T00:45:09.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.228+0000 7fa614ab3640 1 librados: shutdown 2024-04-19T00:45:09.273 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:09.273 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-04-19T00:45:09.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient: get_monmap_and_config 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient: monmap: 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:09.394872+0000 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:09.394872+0000 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:09.393 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:09.394 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 Processor -- start 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 -- start start 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient: init 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient: _add_conns ranks=[7,1,9,4,6,8,0,5,3,10,2] 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd598138a50 0x7fd598138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): picked mon.noname-i con 0x7fd598138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd5980a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:09.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd598139370 0x7fd598138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): picked mon.noname-a con 0x7fd598139370 addr v2:172.21.15.43:3300/0 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5980a0d40 con 0x7fd5980a5bb0 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fd5980a1020 con 0x7fd598139370 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd5980a0780 con 0x7fd598138a50 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd59de6b640 10 monclient(hunting): authenticate will time out at 979.522061s 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd598139370 0x7fd598138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd598139370 0x7fd598138070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39036/0 (socket says 172.21.15.43:39036) 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 -- 172.21.15.43:0/36979421 learned_addr learned my addr 172.21.15.43:0/36979421 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): get_auth_request con 0x7fd598139370 auth_method 0 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd5977fe640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd598138a50 0x7fd598138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd5977fe640 10 monclient(hunting): get_auth_request con 0x7fd598138a50 auth_method 0 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd5977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd5977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd5977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient(hunting): handle_auth_done global_id 4572 payload 274 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 -- 172.21.15.43:0/36979421 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd598138a50 msgr2=0x7fd598138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd598138a50 0x7fd598138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 -- 172.21.15.43:0/36979421 >> v1:172.21.15.161:6789/0 conn(0x7fd5980a5bb0 legacy=0x7fd598137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 monclient: found mon.noname-a 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd597fff640 1 -- 172.21.15.43:0/36979421 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5980a0a60 con 0x7fd598139370 2024-04-19T00:45:09.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.392+0000 7fd5977fe640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd598138a50 0x7fd598138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598138070 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd5900098a0 tx=0x7fd59001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399e5d0d31658cce server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 1 -- 172.21.15.43:0/36979421 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5900b0070 con 0x7fd598139370 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: dump: 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.405 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 1 -- 172.21.15.43:0/36979421 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5900aa810 con 0x7fd598139370 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 1 -- 172.21.15.43:0/36979421 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5900a9300 con 0x7fd598139370 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 5 monclient: authenticate success, global_id 4572 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 4 set_mon_vals no callback set 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5967fc640 10 monclient: dump: 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.407 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: shutdown 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- 172.21.15.43:0/36979421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 msgr2=0x7fd598138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598138070 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd5900098a0 tx=0x7fd59001c400 comp rx=0 tx=0).stop 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- 172.21.15.43:0/36979421 shutdown_connections 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598138070 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 --2- 172.21.15.43:0/36979421 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd598138a50 0x7fd598138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- 172.21.15.43:0/36979421 >> 172.21.15.43:0/36979421 conn(0x7fd5980a7cb0 msgr2=0x7fd5980a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- 172.21.15.43:0/36979421 shutdown_connections 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- 172.21.15.43:0/36979421 wait complete. 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: monmap: 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:09.409 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 librados: starting msgr at 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 librados: starting objecter 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 Processor -- start 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- start start 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 librados: setting wanted keys 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 librados: calling monclient init 2024-04-19T00:45:09.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: init 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient: _add_conns ranks=[1,4,0,6,7,8,2,5,3] 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5980a5bb0 0x7fd598130ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): picked mon.noname-f con 0x7fd5980a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd598138a50 0x7fd598135020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): picked mon.noname-c con 0x7fd598138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598135560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): picked mon.noname-a con 0x7fd598139370 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd598138730 con 0x7fd598139370 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5981385b0 con 0x7fd5980a5bb0 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd598143820 con 0x7fd598138a50 2024-04-19T00:45:09.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd59de6b640 10 monclient(hunting): authenticate will time out at 979.527230s 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd598138a50 0x7fd598135020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598135560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5980a5bb0 0x7fd598130ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598135560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39050/0 (socket says 172.21.15.43:39050) 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 -- 172.21.15.43:0/3882822531 learned_addr learned my addr 172.21.15.43:0/3882822531 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): get_auth_request con 0x7fd598139370 auth_method 0 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd598138a50 0x7fd598135020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44618/0 (socket says 172.21.15.43:44618) 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): get_auth_request con 0x7fd598138a50 auth_method 0 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): get_auth_request con 0x7fd5980a5bb0 auth_method 0 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd596ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient(hunting): handle_auth_done global_id 4581 payload 824 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd598138a50 msgr2=0x7fd598135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd598138a50 0x7fd598135020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5980a5bb0 msgr2=0x7fd598130ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd5980a5bb0 0x7fd598130ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 monclient: found mon.noname-a 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd598139c90 con 0x7fd598139370 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd597fff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598135560 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5900ae040 tx=0x7fd5900191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd58ffff640 10 client.?.objecter ms_handle_connect 0x7fd598139370 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.396+0000 7fd58ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:09.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5900b9070 con 0x7fd598139370 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: dump: 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.415 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5900044b0 con 0x7fd598139370 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 5 monclient: authenticate success, global_id 4581 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 monclient: _renew_subs 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59809fc90 con 0x7fd598139370 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5900b0040 con 0x7fd598139370 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: dump: 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.417 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd595ffb640 4 set_mon_vals no callback set 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 monclient: _renew_subs 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59813a250 con 0x7fd598139370 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 1 librados: init done 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 0x7fd59802c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd59806d140 con 0x7fd598131410 2024-04-19T00:45:09.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd590004040 con 0x7fd598139370 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 0x7fd59802c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient: get_auth_request con 0x7fd598131410 auth_method 0 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd578038e10 0x7fd57803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd598137510 con 0x7fd598131410 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fd5900e1da0 con 0x7fd598139370 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 client.4581.objecter ms_dispatch 0x7fd598000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 3 client.4581.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 3 client.4581.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:09.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 20 client.4581.objecter dump_active .. 0 homeless 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd596ffd640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd578038e10 0x7fd57803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd596ffd640 10 monclient: get_auth_request con 0x7fd578038e10 auth_method 0 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 client.4581.objecter ms_handle_connect 0x7fd598131410 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 client.4581.objecter resend_mon_ops 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd5977fe640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 0x7fd59802c450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd5880115b0 tx=0x7fd588011a70 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd588019070 con 0x7fd598131410 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: dump: 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.420 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 client.4581.objecter ms_handle_connect 0x7fd578038e10 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd596ffd640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd578038e10 0x7fd57803b2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd59806e8c0 tx=0x7fd580009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd58800f7c0 con 0x7fd598131410 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:09.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 msgr2=0x7fd59802c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 0x7fd59802c450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd5880115b0 tx=0x7fd588011a70 comp rx=0 tx=0).stop 2024-04-19T00:45:09.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 msgr2=0x7fd59802c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.400+0000 7fd58ffff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd598131410 0x7fd59802c450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:09.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 0x7fd59802cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd598076190 con 0x7fd59802c990 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 0x7fd59802cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient: get_auth_request con 0x7fd59802c990 auth_method 0 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 1 -- 172.21.15.43:0/3882822531 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd598136c30 con 0x7fd59802c990 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd5977fe640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 0x7fd59802cda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd5880115b0 tx=0x7fd588004740 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 client.4581.objecter ms_handle_connect 0x7fd59802c990 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 client.4581.objecter resend_mon_ops 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd58800f040 con 0x7fd59802c990 2024-04-19T00:45:09.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 monclient: dump: 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.425 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7fd588015e30 con 0x7fd59802c990 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 msgr2=0x7fd59802cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 0x7fd59802cda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd5880115b0 tx=0x7fd588004740 comp rx=0 tx=0).stop 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 msgr2=0x7fd59802cda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd58ffff640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd59802c990 0x7fd59802cda0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd5880115b0 tx=0x7fd588004740 comp rx=0 tx=0).stop 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 52923, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:09.427 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.428 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.429 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:09.430 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.431 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.433 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.436 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.439 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.442 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:09.445 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:09.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:09.447 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 10 librados: watch_flush enter 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 10 librados: watch_flush exit 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 20 client.4581.objecter shutdown clearing up homeless session... 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 10 client.4581.objecter successfully canceled tick 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd578038e10 msgr2=0x7fd57803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd578038e10 0x7fd57803b2d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd59806e8c0 tx=0x7fd580009040 comp rx=0 tx=0).stop 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 10 monclient: shutdown 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 msgr2=0x7fd598135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 --2- 172.21.15.43:0/3882822531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd598139370 0x7fd598135560 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd5900ae040 tx=0x7fd5900191a0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd597fff640 1 -- 172.21.15.43:0/3882822531 reap_dead start 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd597fff640 1 -- 172.21.15.43:0/3882822531 reap_dead start 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 shutdown_connections 2024-04-19T00:45:09.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 >> 172.21.15.43:0/3882822531 conn(0x7fd5980a7cb0 msgr2=0x7fd5980aa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 shutdown_connections 2024-04-19T00:45:09.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 -- 172.21.15.43:0/3882822531 wait complete. 2024-04-19T00:45:09.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.408+0000 7fd59de6b640 1 librados: shutdown 2024-04-19T00:45:09.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:09.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-04-19T00:45:09.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:45:09.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient: get_monmap_and_config 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient: monmap: 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:09.570649+0000 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:09.570649+0000 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.570 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 Processor -- start 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 -- start start 2024-04-19T00:45:09.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient: init 2024-04-19T00:45:09.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient: _add_conns ranks=[6,7,0,2,4,1,8,5,3,9,10] 2024-04-19T00:45:09.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 0x7f29d4136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): picked mon.noname-d con 0x7f29d409e4c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29d409e8a0 0x7f29d40a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): picked mon.noname-i con 0x7f29d409e8a0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29d409eca0 addr v1:172.21.15.43:6789/0 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29d40a1e20 con 0x7f29d409eca0 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f29d40a1c50 con 0x7f29d409e4c0 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f29d406c760 con 0x7f29d409e8a0 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29dc1b1640 10 monclient(hunting): authenticate will time out at 979.698003s 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 0x7f29d4136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 0x7f29d4136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:41232/0 (socket says 172.21.15.43:41232) 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29da74e640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f29d409eca0 0x7f29d40a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:55538/0 (socket says 172.21.15.43:55538) 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4137750273 learned_addr learned my addr 172.21.15.43:0/4137750273 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): get_auth_request con 0x7f29d409e4c0 auth_method 0 2024-04-19T00:45:09.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 1 --2- 172.21.15.43:0/4137750273 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29d409e8a0 0x7f29d40a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): get_auth_request con 0x7f29d409e8a0 auth_method 0 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d8f4b640 1 -- 172.21.15.43:0/4137750273 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272204669 0 0) 0x7f29d40a1e20 con 0x7f29d409eca0 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d8f4b640 10 monclient(hunting): _negotiate 2024-04-19T00:45:09.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d8f4b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d8f4b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d8f4b640 10 monclient(hunting): my global_id is 4590 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d8f4b640 1 -- 172.21.15.43:0/4137750273 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c0003530 con 0x7f29d409eca0 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_done global_id 4407 payload 274 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4137750273 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29d409e8a0 msgr2=0x7f29d40a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 --2- 172.21.15.43:0/4137750273 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29d409e8a0 0x7f29d40a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4137750273 >> v1:172.21.15.43:6789/0 conn(0x7f29d409eca0 legacy=0x7f29d40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 monclient: found mon.noname-d 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.568+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4137750273 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d40a5510 con 0x7f29d409e4c0 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 --2- 172.21.15.43:0/4137750273 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 0x7f29d4136e60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e80 tx=0x7f29cc01c9a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7143a3dcda69314f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 1 -- 172.21.15.43:0/4137750273 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29cc0090d0 con 0x7f29d409e4c0 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: dump: 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.583 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 1 -- 172.21.15.43:0/4137750273 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29cc0a5a90 con 0x7f29d409e4c0 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 1 -- 172.21.15.43:0/4137750273 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29cc0a5c40 con 0x7f29d409e4c0 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 5 monclient: authenticate success, global_id 4407 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 4 set_mon_vals no callback set 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d8f4b640 10 monclient: dump: 2024-04-19T00:45:09.584 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: shutdown 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4137750273 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 msgr2=0x7f29d4136e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4137750273 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 0x7f29d4136e60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e80 tx=0x7f29cc01c9a0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4137750273 shutdown_connections 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4137750273 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29d409e8a0 0x7f29d40a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4137750273 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f29d409e4c0 0x7f29d4136e60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4137750273 >> 172.21.15.43:0/4137750273 conn(0x7f29d40a85c0 msgr2=0x7f29d40a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4137750273 shutdown_connections 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4137750273 wait complete. 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: monmap: 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:09.586 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 librados: starting msgr at 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 librados: starting objecter 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 Processor -- start 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- start start 2024-04-19T00:45:09.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 librados: setting wanted keys 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 librados: calling monclient init 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: init 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient: _add_conns ranks=[2,0,8,7,5,1,4,6,3] 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29d409e4c0 0x7f29d4130f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): picked mon.noname-b con 0x7f29d409e4c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29d409e8a0 0x7f29d41314d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): picked mon.noname-a con 0x7f29d409e8a0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d409eca0 0x7f29d4131a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): picked mon.noname-e con 0x7f29d409eca0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f29d41416f0 con 0x7f29d409e8a0 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f29d40a47a0 con 0x7f29d409e4c0 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f29d4141870 con 0x7f29d409eca0 2024-04-19T00:45:09.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29dc1b1640 10 monclient(hunting): authenticate will time out at 979.703178s 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29d409e8a0 0x7f29d41314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29d409e4c0 0x7f29d4130f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29d409e4c0 0x7f29d4130f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35786/0 (socket says 172.21.15.43:35786) 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29d409e8a0 0x7f29d41314d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39052/0 (socket says 172.21.15.43:39052) 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d409eca0 0x7f29d4131a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4039616796 learned_addr learned my addr 172.21.15.43:0/4039616796 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): get_auth_request con 0x7f29d409e4c0 auth_method 0 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): get_auth_request con 0x7f29d409e8a0 auth_method 0 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): get_auth_request con 0x7f29d409eca0 auth_method 0 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d409eca0 msgr2=0x7f29d4131a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d409eca0 0x7f29d4131a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29d409e8a0 msgr2=0x7f29d41314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29d409e8a0 0x7f29d41314d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 monclient: found mon.noname-b 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d9f4d640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d4132580 con 0x7f29d409e4c0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29da74e640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d409eca0 0x7f29d4131a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.572+0000 7f29d974c640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f29d409e8a0 0x7f29d41314d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d9f4d640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29d409e4c0 0x7f29d4130f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e20 tx=0x7f29cc019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 client.?.objecter ms_handle_connect 0x7f29d409e4c0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29cc0c4070 con 0x7f29d409e4c0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: dump: 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.592 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29cc0039a0 con 0x7f29d409e4c0 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 5 monclient: authenticate success, global_id 4493 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 monclient: _renew_subs 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d406b9e0 con 0x7f29d409e4c0 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29cc004040 con 0x7f29d409e4c0 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:45:09.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29cbfff640 4 set_mon_vals no callback set 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: dump: 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.594 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 monclient: _renew_subs 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d4133b80 con 0x7f29d409e4c0 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 1 librados: init done 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 0x7f29d41ce780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f29d41340a0 con 0x7f29d4132bd0 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f29cc003b50 con 0x7f29d409e4c0 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 0x7f29d41ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient: get_auth_request con 0x7f29d4132bd0 auth_method 0 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f29b8038e10 0x7f29b803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f29cc0e6ca0 con 0x7f29d409e4c0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 client.4493.objecter ms_dispatch 0x7f29d4000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 3 client.4493.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 3 client.4493.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 20 client.4493.objecter dump_active .. 0 homeless 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 10 monclient(hunting): handle_auth_done global_id 4562 payload 824 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29d40a1440 con 0x7f29d4132bd0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29da74e640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f29b8038e10 0x7f29b803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29da74e640 10 monclient: get_auth_request con 0x7f29b8038e10 auth_method 0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 client.4493.objecter ms_handle_connect 0x7f29d4132bd0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 client.4493.objecter resend_mon_ops 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29d974c640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 0x7f29d41ce780 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009fd0 tx=0x7f29c400f610 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29c40044a0 con 0x7f29d4132bd0 2024-04-19T00:45:09.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: dump: 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.597 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f29c4004030 con 0x7f29d4132bd0 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29da74e640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f29b8038e10 0x7f29b803b2d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f29d4072b30 tx=0x7f29d0007f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 msgr2=0x7f29d41ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 0x7f29d41ce780 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009fd0 tx=0x7f29c400f610 comp rx=0 tx=0).stop 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 msgr2=0x7f29d41ce780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d4132bd0 0x7f29d41ce780 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009fd0 tx=0x7f29c400f610 comp rx=0 tx=0).stop 2024-04-19T00:45:09.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.576+0000 7f29ca7fc640 10 client.4493.objecter ms_handle_connect 0x7f29b8038e10 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 0x7f29d41cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f29d41cf610 con 0x7f29d41cecc0 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 0x7f29d41cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient: get_auth_request con 0x7f29d41cecc0 auth_method 0 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 10 monclient(hunting): handle_auth_done global_id 4571 payload 824 2024-04-19T00:45:09.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 1 -- 172.21.15.43:0/4039616796 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29d4076500 con 0x7f29d41cecc0 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 client.4493.objecter ms_handle_connect 0x7f29d41cecc0 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29d974c640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 0x7f29d41cf0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29c4000c10 tx=0x7f29c40046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 client.4493.objecter resend_mon_ops 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29c4014ac0 con 0x7f29d41cecc0 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 monclient: dump: 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.601 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9603 (secure 0 0 0) 0x7f29c40273a0 con 0x7f29d41cecc0 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 msgr2=0x7f29d41cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 0x7f29d41cf0d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29c4000c10 tx=0x7f29c40046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 msgr2=0x7f29d41cf0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29ca7fc640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f29d41cecc0 0x7f29d41cf0d0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29c4000c10 tx=0x7f29c40046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 53095, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:09.603 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.604 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.605 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:09.606 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.610 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.613 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:09.614 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.619 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.620 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:45:09.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.622 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:09.623 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:09.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 10 librados: watch_flush enter 2024-04-19T00:45:09.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 10 librados: watch_flush exit 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 20 client.4493.objecter shutdown clearing up homeless session... 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 10 client.4493.objecter successfully canceled tick 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f29b8038e10 msgr2=0x7f29b803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f29b8038e10 0x7f29b803b2d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f29d4072b30 tx=0x7f29d0007f30 comp rx=0 tx=0).stop 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 10 monclient: shutdown 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29d409e4c0 msgr2=0x7f29d4130f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 --2- 172.21.15.43:0/4039616796 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29d409e4c0 0x7f29d4130f90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f29cc009e20 tx=0x7f29cc019740 comp rx=0 tx=0).stop 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29da74e640 1 -- 172.21.15.43:0/4039616796 reap_dead start 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29da74e640 1 -- 172.21.15.43:0/4039616796 reap_dead start 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 shutdown_connections 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 >> 172.21.15.43:0/4039616796 conn(0x7f29d40a85c0 msgr2=0x7f29d412dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 shutdown_connections 2024-04-19T00:45:09.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 -- 172.21.15.43:0/4039616796 wait complete. 2024-04-19T00:45:09.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.584+0000 7f29dc1b1640 1 librados: shutdown 2024-04-19T00:45:09.626 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:09.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-04-19T00:45:09.626 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient: get_monmap_and_config 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient: monmap: 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:09.746765+0000 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:09.746765+0000 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:09.745 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:09.746 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 Processor -- start 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 -- start start 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient: init 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient: _add_conns ranks=[9,7,1,2,4,3,6,5,8,0,10] 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f41b806c240 0x7f41b80a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): picked mon.noname-a con 0x7f41b806c240 addr v2:172.21.15.43:3300/0 2024-04-19T00:45:09.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b80a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): picked mon.noname-i con 0x7f41b80a2f90 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f41b80a4f70 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41b81374f0 con 0x7f41b80a4f70 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f41b806c620 con 0x7f41b806c240 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f41b80722d0 con 0x7f41b80a2f90 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bf096640 10 monclient(hunting): authenticate will time out at 979.873956s 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f41b806c240 0x7f41b80a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f41b806c240 0x7f41b80a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39054/0 (socket says 172.21.15.43:39054) 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 -- 172.21.15.43:0/339523072 learned_addr learned my addr 172.21.15.43:0/339523072 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): get_auth_request con 0x7f41b806c240 auth_method 0 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41b7fff640 1 --2- 172.21.15.43:0/339523072 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b80a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41b7fff640 10 monclient(hunting): get_auth_request con 0x7f41b80a2f90 auth_method 0 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient(hunting): handle_auth_done global_id 4608 payload 274 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 -- 172.21.15.43:0/339523072 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 msgr2=0x7f41b80a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 --2- 172.21.15.43:0/339523072 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b80a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 -- 172.21.15.43:0/339523072 >> v1:172.21.15.161:6789/0 conn(0x7f41b80a4f70 legacy=0x7f41b813dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 monclient: found mon.noname-a 2024-04-19T00:45:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:45:09.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.744+0000 7f41bce32640 1 -- 172.21.15.43:0/339523072 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b8072f50 con 0x7f41b806c240 2024-04-19T00:45:09.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bce32640 1 --2- 172.21.15.43:0/339523072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b806c240 0x7f41b80a2a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009e80 tx=0x7f41b001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff6fb7b6016afb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 1 -- 172.21.15.43:0/339523072 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41b00090d0 con 0x7f41b806c240 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: dump: 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.758 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 1 -- 172.21.15.43:0/339523072 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b00a5b40 con 0x7f41b806c240 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 5 monclient: authenticate success, global_id 4608 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:09.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 1 -- 172.21.15.43:0/339523072 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41b00a5d30 con 0x7f41b806c240 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 4 set_mon_vals no callback set 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41b77fe640 10 monclient: dump: 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.760 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: shutdown 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 -- 172.21.15.43:0/339523072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b806c240 msgr2=0x7f41b80a2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 --2- 172.21.15.43:0/339523072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b806c240 0x7f41b80a2a50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009e80 tx=0x7f41b001c900 comp rx=0 tx=0).stop 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 -- 172.21.15.43:0/339523072 shutdown_connections 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 --2- 172.21.15.43:0/339523072 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b80a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 --2- 172.21.15.43:0/339523072 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b806c240 0x7f41b80a2a50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 -- 172.21.15.43:0/339523072 >> 172.21.15.43:0/339523072 conn(0x7f41b80a62e0 msgr2=0x7f41b806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 -- 172.21.15.43:0/339523072 shutdown_connections 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 -- 172.21.15.43:0/339523072 wait complete. 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: monmap: 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:09.762 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 librados: starting msgr at 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 librados: starting objecter 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 Processor -- start 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 -- start start 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 librados: setting wanted keys 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 librados: calling monclient init 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: init 2024-04-19T00:45:09.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient: _add_conns ranks=[8,7,6,4,3,2,5,0,1] 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b806c240 0x7f41b809e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient(hunting): picked mon.noname-e con 0x7f41b806c240 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b81357c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 10 monclient(hunting): picked mon.noname-i con 0x7f41b80a2f90 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.748+0000 7f41bf096640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f41b80a4f70 0x7f41b811f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): picked mon.noname-d con 0x7f41b80a4f70 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f41b8071d00 con 0x7f41b80a4f70 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f41b8071b80 con 0x7f41b80a2f90 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f41b80a3370 con 0x7f41b806c240 2024-04-19T00:45:09.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): authenticate will time out at 979.880029s 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f41b80a4f70 0x7f41b811f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f41b80a4f70 0x7f41b811f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:41246/0 (socket says 172.21.15.43:41246) 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 1 -- 172.21.15.43:0/951972721 learned_addr learned my addr 172.21.15.43:0/951972721 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): get_auth_request con 0x7f41b80a4f70 auth_method 0 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b81357c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): get_auth_request con 0x7f41b80a2f90 auth_method 0 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b806c240 0x7f41b809e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): get_auth_request con 0x7f41b806c240 auth_method 0 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient(hunting): handle_auth_done global_id 4580 payload 824 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 msgr2=0x7f41b81357c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b80a2f90 0x7f41b81357c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f41b80a4f70 msgr2=0x7f41b811f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f41b80a4f70 0x7f41b811f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 monclient: found mon.noname-e 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b8121bb0 con 0x7f41b806c240 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f41b80a4f70 0x7f41b811f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bce32640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b806c240 0x7f41b809e670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41b00ab000 tx=0x7f41b0019700 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 client.?.objecter ms_handle_connect 0x7f41b806c240 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:09.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41b00bc020 con 0x7f41b806c240 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: dump: 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.768 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 5 monclient: authenticate success, global_id 4580 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient: _renew_subs 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41b0003e10 con 0x7f41b806c240 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b8121e90 con 0x7f41b806c240 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41b00044b0 con 0x7f41b806c240 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b6ffd640 4 set_mon_vals no callback set 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: dump: 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.770 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient: _renew_subs 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b80724f0 con 0x7f41b806c240 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 librados: init done 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 0x7f41b811ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f41b806d140 con 0x7f41b811fbb0 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f41b00bf020 con 0x7f41b806c240 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 0x7f41b811ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient: get_auth_request con 0x7f41b811fbb0 auth_method 0 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4198038e10 0x7f419803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f41b00e4120 con 0x7f41b806c240 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 client.4580.objecter ms_dispatch 0x7f41b8000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 3 client.4580.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 3 client.4580.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 10 monclient(hunting): handle_auth_done global_id 4519 payload 824 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41b81204d0 con 0x7f41b811fbb0 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 20 client.4580.objecter dump_active .. 0 homeless 2024-04-19T00:45:09.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4198038e10 0x7f419803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 10 monclient: get_auth_request con 0x7f4198038e10 auth_method 0 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 client.4580.objecter ms_handle_connect 0x7f41b811fbb0 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 client.4580.objecter resend_mon_ops 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b7fff640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 0x7f41b811ff90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a0010820 tx=0x7f41a0010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a0018020 con 0x7f41b811fbb0 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: dump: 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.773 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 client.4580.objecter ms_handle_connect 0x7f4198038e10 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41bd633640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4198038e10 0x7f419803b2d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41b809edc0 tx=0x7f41a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41a0004240 con 0x7f41b811fbb0 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 msgr2=0x7f41b811ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 0x7f41b811ff90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a0010820 tx=0x7f41a0010ce0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 msgr2=0x7f41b811ff90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.752+0000 7f41b57fa640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b811fbb0 0x7f41b811ff90 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f41a0010820 tx=0x7f41a0010ce0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41bf096640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41bf096640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 0x7f41b806dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41bf096640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f41b81d1850 con 0x7f41b806d7e0 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 0x7f41b806dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient: get_auth_request con 0x7f41b806d7e0 auth_method 0 2024-04-19T00:45:09.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 10 monclient(hunting): handle_auth_done global_id 4528 payload 824 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 1 -- 172.21.15.43:0/951972721 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41b809f240 con 0x7f41b806d7e0 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b57fa640 10 client.4580.objecter ms_handle_connect 0x7f41b806d7e0 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b57fa640 10 client.4580.objecter resend_mon_ops 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b7fff640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 0x7f41b806dbc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41a0010820 tx=0x7f41a00046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a0018040 con 0x7f41b806d7e0 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.760+0000 7f41b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 10 monclient: dump: 2024-04-19T00:45:09.777 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7f41a000ee00 con 0x7f41b806d7e0 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 msgr2=0x7f41b806dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 0x7f41b806dbc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41a0010820 tx=0x7f41a00046d0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 msgr2=0x7f41b806dbc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41b57fa640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f41b806d7e0 0x7f41b806dbc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:09.779 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 53231, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.780 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.781 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:09.782 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:09.783 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:09.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.787 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:09.788 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.790 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:09.792 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.795 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:09.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.798 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.799 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.801 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:09.801 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:09.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 10 librados: watch_flush enter 2024-04-19T00:45:09.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 10 librados: watch_flush exit 2024-04-19T00:45:09.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 20 client.4580.objecter shutdown clearing up homeless session... 2024-04-19T00:45:09.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 10 client.4580.objecter successfully canceled tick 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4198038e10 msgr2=0x7f419803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4198038e10 0x7f419803b2d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f41b809edc0 tx=0x7f41a8009040 comp rx=0 tx=0).stop 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 10 monclient: shutdown 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b806c240 msgr2=0x7f41b809e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b806c240 0x7f41b809e670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f41b00ab000 tx=0x7f41b0019700 comp rx=0 tx=0).stop 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bd633640 1 -- 172.21.15.43:0/951972721 reap_dead start 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 shutdown_connections 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 --2- 172.21.15.43:0/951972721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b806c240 0x7f41b809e670 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 >> 172.21.15.43:0/951972721 conn(0x7f41b80a62e0 msgr2=0x7f41b80a46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 shutdown_connections 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 -- 172.21.15.43:0/951972721 wait complete. 2024-04-19T00:45:09.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.764+0000 7f41bf096640 1 librados: shutdown 2024-04-19T00:45:09.804 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:09.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-04-19T00:45:09.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:45:09.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient: get_monmap_and_config 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient: monmap: 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:09.916597+0000 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:09.916597+0000 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:09.915 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:09.916 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 Processor -- start 2024-04-19T00:45:09.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 -- start start 2024-04-19T00:45:09.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient: init 2024-04-19T00:45:09.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient: _add_conns ranks=[1,3,5,6,9,2,7,10,8,4,0] 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6068138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6068139540 0x7f606813dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): picked mon.noname-g con 0x7f6068139540 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60680a5bb0 0x7f6068140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): picked mon.noname-h con 0x7f60680a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60680a0d40 con 0x7f6068138a50 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f60680a1020 con 0x7f6068139540 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f60680a0780 con 0x7f60680a5bb0 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f606d716640 10 monclient(hunting): authenticate will time out at 980.043689s 2024-04-19T00:45:09.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60677fe640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60680a5bb0 msgr2=0x7f6068140530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60677fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60680a5bb0 0x7f6068140530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60667fc640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6068139540 0x7f606813dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f6066ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6068138a50 0x7f6068138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:38988/0 (socket says 172.21.15.43:38988) 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f6066ffd640 1 -- 172.21.15.43:0/1193306126 learned_addr learned my addr 172.21.15.43:0/1193306126 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60667fc640 10 monclient(hunting): get_auth_request con 0x7f6068139540 auth_method 0 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60667fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.912+0000 7f60667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 1 -- 172.21.15.43:0/1193306126 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204452679 0 0) 0x7f60680a0d40 con 0x7f6068138a50 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient(hunting): my global_id is 4537 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 1 -- 172.21.15.43:0/1193306126 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6054003530 con 0x7f6068138a50 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 10 monclient(hunting): handle_auth_done global_id 4494 payload 274 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 1 -- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60680a5bb0 msgr2=0x7f6068140530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 1 --2- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60680a5bb0 0x7f6068140530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 1 -- 172.21.15.43:0/1193306126 >> v1:172.21.15.161:6789/0 conn(0x7f6068138a50 legacy=0x7f6068138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 1 monclient: found mon.noname-g 2024-04-19T00:45:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:45:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 1 -- 172.21.15.43:0/1193306126 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60680a0a60 con 0x7f6068139540 2024-04-19T00:45:09.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f60667fc640 1 --2- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6068139540 0x7f606813dfe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60500097f0 tx=0x7f605001c3c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=7a576adb18f4fbb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 1 -- 172.21.15.43:0/1193306126 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60500b0070 con 0x7f6068139540 2024-04-19T00:45:09.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:45:09.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: dump: 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 1 -- 172.21.15.43:0/1193306126 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60500aa810 con 0x7f6068139540 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 5 monclient: authenticate success, global_id 4494 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 1 -- 172.21.15.43:0/1193306126 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60500aae30 con 0x7f6068139540 2024-04-19T00:45:09.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 4 set_mon_vals no callback set 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f6065ffb640 10 monclient: dump: 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.932 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 10 monclient: shutdown 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 -- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6068139540 msgr2=0x7f606813dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 --2- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6068139540 0x7f606813dfe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60500097f0 tx=0x7f605001c3c0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 -- 172.21.15.43:0/1193306126 shutdown_connections 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 --2- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f60680a5bb0 0x7f6068140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 --2- 172.21.15.43:0/1193306126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6068139540 0x7f606813dfe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 -- 172.21.15.43:0/1193306126 >> 172.21.15.43:0/1193306126 conn(0x7f60680a7cb0 msgr2=0x7f60680a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 -- 172.21.15.43:0/1193306126 shutdown_connections 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.916+0000 7f606d716640 1 -- 172.21.15.43:0/1193306126 wait complete. 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: build_initial_monmap 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: monmap: 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:09.934 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 librados: starting msgr at 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 librados: starting objecter 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 Processor -- start 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- start start 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 librados: setting wanted keys 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 librados: calling monclient init 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: init 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:09.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: _add_conns ranks=[3,1,7,8,5,0,6,4,2] 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60680a5bb0 0x7f606813ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): picked mon.noname-g con 0x7f60680a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6068138a50 0x7f606813d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): picked mon.noname-f con 0x7f6068138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6068139540 0x7f606813d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): picked mon.noname-i con 0x7f6068139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6068143620 con 0x7f6068138a50 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f60681434a0 con 0x7f60680a5bb0 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f60681437a0 con 0x7f6068139540 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:09.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): authenticate will time out at 980.049061s 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60680a5bb0 0x7f606813ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6068138a50 0x7f606813d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6068138a50 0x7f606813d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:47428/0 (socket says 172.21.15.43:47428) 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 -- 172.21.15.43:0/3177109910 learned_addr learned my addr 172.21.15.43:0/3177109910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): get_auth_request con 0x7f60680a5bb0 auth_method 0 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): get_auth_request con 0x7f6068138a50 auth_method 0 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6068139540 0x7f606813d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): get_auth_request con 0x7f6068139540 auth_method 0 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient(hunting): handle_auth_done global_id 4503 payload 824 2024-04-19T00:45:09.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6068139540 msgr2=0x7f606813d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6068139540 0x7f606813d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6068138a50 msgr2=0x7f606813d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6068138a50 0x7f606813d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 monclient: found mon.noname-g 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606813f970 con 0x7f60680a5bb0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6068138a50 0x7f606813d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f6066ffd640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60680a5bb0 0x7f606813ef40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f605800c400 tx=0x7f605800c8c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 client.?.objecter ms_handle_connect 0x7f60680a5bb0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6058017070 con 0x7f60680a5bb0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:45:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: dump: 2024-04-19T00:45:09.940 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.941 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:09.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 5 monclient: authenticate success, global_id 4503 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: _renew_subs 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60580070d0 con 0x7f60680a5bb0 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606809ee60 con 0x7f60680a5bb0 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6058007e30 con 0x7f60680a5bb0 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60657fa640 4 set_mon_vals no callback set 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: dump: 2024-04-19T00:45:09.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.944 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: _renew_subs 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f606813e3e0 con 0x7f60680a5bb0 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 librados: init done 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 0x7f60681d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f606806d140 con 0x7f606813cca0 2024-04-19T00:45:09.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 0x7f60681d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient: get_auth_request con 0x7f606813cca0 auth_method 0 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6058007300 con 0x7f60680a5bb0 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6048038e10 0x7f604803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f6058048070 con 0x7f60680a5bb0 2024-04-19T00:45:09.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 client.4503.objecter ms_dispatch 0x7f6068000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 3 client.4503.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 3 client.4503.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 10 monclient(hunting): handle_auth_done global_id 4512 payload 824 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6068121c90 con 0x7f606813cca0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 20 client.4503.objecter dump_active .. 0 homeless 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6048038e10 0x7f604803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60677fe640 10 monclient: get_auth_request con 0x7f6048038e10 auth_method 0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f60667fc640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 0x7f60681d0890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6050009710 tx=0x7f6050019160 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 client.4503.objecter ms_handle_connect 0x7f606813cca0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 client.4503.objecter resend_mon_ops 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60500b0040 con 0x7f606813cca0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.920+0000 7f605f7fe640 10 monclient: dump: 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 10 client.4503.objecter ms_handle_connect 0x7f6048038e10 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f60677fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6048038e10 0x7f604803b2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6060003cc0 tx=0x7f60600095f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60500b1040 con 0x7f606813cca0 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:09.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 msgr2=0x7f60681d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 0x7f60681d0890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6050009710 tx=0x7f6050019160 comp rx=0 tx=0).stop 2024-04-19T00:45:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 msgr2=0x7f60681d0890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.924+0000 7f605f7fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f606813cca0 0x7f60681d0890 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6050009710 tx=0x7f6050019160 comp rx=0 tx=0).stop 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f606d716640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f606d716640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 0x7f60681d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f606d716640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f60681d1720 con 0x7f60681d0dd0 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 0x7f60681d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient: get_auth_request con 0x7f60681d0dd0 auth_method 0 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:09.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 10 monclient(hunting): handle_auth_done global_id 4521 payload 824 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 1 -- 172.21.15.43:0/3177109910 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60680008e0 con 0x7f60681d0dd0 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f60667fc640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 0x7f60681d11e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f605001cc60 tx=0x7f60500a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f605f7fe640 10 client.4503.objecter ms_handle_connect 0x7f60681d0dd0 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f605f7fe640 10 client.4503.objecter resend_mon_ops 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f60500b0040 con 0x7f60681d0dd0 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f605f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:09.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f605f7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.928+0000 7f605f7fe640 10 monclient: dump: 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:09.954 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7f60500c6070 con 0x7f60681d0dd0 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 msgr2=0x7f60681d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 0x7f60681d11e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f605001cc60 tx=0x7f60500a8000 comp rx=0 tx=0).stop 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 msgr2=0x7f60681d11e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:09.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f605f7fe640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60681d0dd0 0x7f60681d11e0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f605001cc60 tx=0x7f60500a8000 comp rx=0 tx=0).stop 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 53411, 2024-04-19T00:45:09.956 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:09.957 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.958 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.959 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:09.960 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:09.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:09.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:09.964 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.965 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.967 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:09.969 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.971 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.972 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.973 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.974 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:09.975 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.976 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:09.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.978 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:09.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:09.980 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:09.981 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 10 librados: watch_flush enter 2024-04-19T00:45:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 10 librados: watch_flush exit 2024-04-19T00:45:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 20 client.4503.objecter shutdown clearing up homeless session... 2024-04-19T00:45:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 10 client.4503.objecter successfully canceled tick 2024-04-19T00:45:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6048038e10 msgr2=0x7f604803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6048038e10 0x7f604803b2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6060003cc0 tx=0x7f60600095f0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 10 monclient: shutdown 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60680a5bb0 msgr2=0x7f606813ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 --2- 172.21.15.43:0/3177109910 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f60680a5bb0 0x7f606813ef40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f605800c400 tx=0x7f605800c8c0 comp rx=0 tx=0).stop 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f60677fe640 1 -- 172.21.15.43:0/3177109910 reap_dead start 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f60677fe640 1 -- 172.21.15.43:0/3177109910 reap_dead start 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 shutdown_connections 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 >> 172.21.15.43:0/3177109910 conn(0x7f60680a7cb0 msgr2=0x7f60680a2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 shutdown_connections 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 -- 172.21.15.43:0/3177109910 wait complete. 2024-04-19T00:45:09.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:09.932+0000 7f606d716640 1 librados: shutdown 2024-04-19T00:45:09.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:09.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-04-19T00:45:09.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient: get_monmap_and_config 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient: build_initial_monmap 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient: monmap: 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:45:10.114812+0000 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:45:10.114812+0000 2024-04-19T00:45:10.113 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:45:10.114 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 Processor -- start 2024-04-19T00:45:10.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 -- start start 2024-04-19T00:45:10.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient: init 2024-04-19T00:45:10.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient: _add_conns ranks=[9,10,3,0,6,8,4,2,1,5,7] 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdb34138a50 0x7fdb34138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): picked mon.noname-a con 0x7fdb34138a50 addr v2:172.21.15.43:3300/0 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdb340a5bb0 0x7fdb34139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): picked mon.noname-f con 0x7fdb340a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb34137960 0x7fdb34137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): picked mon.noname-g con 0x7fdb34137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fdb340a1020 con 0x7fdb34138a50 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdb341398b0 con 0x7fdb340a5bb0 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdb340a0780 con 0x7fdb34137960 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb39568640 10 monclient(hunting): authenticate will time out at 980.241971s 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdb34138a50 0x7fdb34138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdb34138a50 0x7fdb34138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39058/0 (socket says 172.21.15.43:39058) 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 -- 172.21.15.43:0/496361276 learned_addr learned my addr 172.21.15.43:0/496361276 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): get_auth_request con 0x7fdb34138a50 auth_method 0 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb327fc640 1 --2- 172.21.15.43:0/496361276 >> v2:172.21.15.161:3300/0 conn(0x7fdb340a5bb0 0x7fdb34139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:10.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb327fc640 10 monclient(hunting): get_auth_request con 0x7fdb340a5bb0 auth_method 0 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb327fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb327fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb337fe640 1 --2- 172.21.15.43:0/496361276 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb34137960 0x7fdb34137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb337fe640 10 monclient(hunting): get_auth_request con 0x7fdb34137960 auth_method 0 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb337fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_done global_id 4617 payload 274 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 -- 172.21.15.43:0/496361276 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb34137960 msgr2=0x7fdb34137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/496361276 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb34137960 0x7fdb34137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 -- 172.21.15.43:0/496361276 >> v2:172.21.15.161:3300/0 conn(0x7fdb340a5bb0 msgr2=0x7fdb34139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/496361276 >> v2:172.21.15.161:3300/0 conn(0x7fdb340a5bb0 0x7fdb34139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 monclient: found mon.noname-a 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb32ffd640 1 -- 172.21.15.43:0/496361276 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb340a0d40 con 0x7fdb34138a50 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.112+0000 7fdb327fc640 1 --2- 172.21.15.43:0/496361276 >> v2:172.21.15.161:3300/0 conn(0x7fdb340a5bb0 0x7fdb34139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/496361276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb34138a50 0x7fdb34138e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb3406e8c0 tx=0x7fdb1c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60070279d84c8e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 1 -- 172.21.15.43:0/496361276 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c0b0070 con 0x7fdb34138a50 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: dump: 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.121 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: _finish_auth 0 2024-04-19T00:45:10.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 5 monclient: authenticate success, global_id 4617 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 1 -- 172.21.15.43:0/496361276 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb1c0aa810 con 0x7fdb34138a50 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: get_monmap_and_config success 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 4 set_mon_vals no callback set 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 1 -- 172.21.15.43:0/496361276 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c0a9300 con 0x7fdb34138a50 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb31ffb640 10 monclient: dump: 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.123 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:10.124 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: shutdown 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- 172.21.15.43:0/496361276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb34138a50 msgr2=0x7fdb34138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- 172.21.15.43:0/496361276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb34138a50 0x7fdb34138e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb3406e8c0 tx=0x7fdb1c01c3e0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- 172.21.15.43:0/496361276 shutdown_connections 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- 172.21.15.43:0/496361276 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb34137960 0x7fdb34137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- 172.21.15.43:0/496361276 >> v2:172.21.15.161:3300/0 conn(0x7fdb340a5bb0 0x7fdb34139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- 172.21.15.43:0/496361276 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb34138a50 0x7fdb34138e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- 172.21.15.43:0/496361276 >> 172.21.15.43:0/496361276 conn(0x7fdb340a7cb0 msgr2=0x7fdb340a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- 172.21.15.43:0/496361276 shutdown_connections 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- 172.21.15.43:0/496361276 wait complete. 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: build_initial_monmap 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: monmap: 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:10.133 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 librados: starting msgr at 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 librados: starting objecter 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 Processor -- start 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- start start 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 librados: setting wanted keys 2024-04-19T00:45:10.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 librados: calling monclient init 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: init 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: _reopen_session rank -1 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient: _add_conns ranks=[0,2,4,7,5,6,3,8,1] 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb340a5bb0 0x7fdb341d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): picked mon.noname-a con 0x7fdb340a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb34137960 0x7fdb34136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): picked mon.noname-b con 0x7fdb34137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb34138a50 0x7fdb3413c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): picked mon.noname-c con 0x7fdb34138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdb340a2100 con 0x7fdb340a5bb0 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdb340a1f80 con 0x7fdb34137960 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdb34142810 con 0x7fdb34138a50 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): _renew_subs 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb39568640 10 monclient(hunting): authenticate will time out at 980.247103s 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb340a5bb0 0x7fdb341d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb34137960 0x7fdb34136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb34137960 0x7fdb34136a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35794/0 (socket says 172.21.15.43:35794) 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 1 -- 172.21.15.43:0/641331624 learned_addr learned my addr 172.21.15.43:0/641331624 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb340a5bb0 0x7fdb341d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39062/0 (socket says 172.21.15.43:39062) 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): get_auth_request con 0x7fdb34137960 auth_method 0 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb34138a50 0x7fdb3413c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): get_auth_request con 0x7fdb340a5bb0 auth_method 0 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): get_auth_request con 0x7fdb34138a50 auth_method 0 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:10.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient(hunting): handle_auth_done global_id 4486 payload 824 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb34137960 msgr2=0x7fdb34136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb34137960 0x7fdb34136a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb340a5bb0 msgr2=0x7fdb341d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb340a5bb0 0x7fdb341d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient: _finish_hunting 0 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 monclient: found mon.noname-c 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:45:10.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3409ee60 con 0x7fdb34138a50 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb327fc640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb34137960 0x7fdb34136a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb340a5bb0 0x7fdb341d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb337fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb34138a50 0x7fdb3413c850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00d530 tx=0x7fdb2c00d9f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb2b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb34138a50 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.116+0000 7fdb2b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb2c012a60 con 0x7fdb34138a50 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: dump: 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.139 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2c019070 con 0x7fdb34138a50 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 5 monclient: authenticate success, global_id 4486 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 monclient: _renew_subs 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3413e4d0 con 0x7fdb34138a50 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb2c012c40 con 0x7fdb34138a50 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:45:10.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb317fa640 4 set_mon_vals no callback set 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: dump: 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:10.142 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 monclient: _renew_subs 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb34121b70 con 0x7fdb34138a50 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 1 librados: init done 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 0x7fdb3413cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdb3406d140 con 0x7fdb34136fb0 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdb2c0125f0 con 0x7fdb34138a50 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 0x7fdb3413cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb14038e10 0x7fdb1403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient: get_auth_request con 0x7fdb34136fb0 auth_method 0 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fdb2c00f040 con 0x7fdb34138a50 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 client.4486.objecter ms_dispatch 0x7fdb34000ba0 osd_map(15..15 src has 1..15) 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 3 client.4486.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 3 client.4486.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:10.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 20 client.4486.objecter dump_active .. 0 homeless 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb327fc640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb14038e10 0x7fdb1403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb327fc640 10 monclient: get_auth_request con 0x7fdb14038e10 auth_method 0 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_done global_id 4606 payload 824 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb340008e0 con 0x7fdb34136fb0 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 0x7fdb3413cd90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdb340a72d0 tx=0x7fdb1c019180 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 client.4486.objecter ms_handle_connect 0x7fdb34136fb0 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 client.4486.objecter resend_mon_ops 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 client.4486.objecter ms_handle_connect 0x7fdb14038e10 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb327fc640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb14038e10 0x7fdb1403b2d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdb3413dfb0 tx=0x7fdb240099d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:10.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c0ba070 con 0x7fdb34136fb0 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: dump: 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.146 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb1c0aa750 con 0x7fdb34136fb0 2024-04-19T00:45:10.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 msgr2=0x7fdb3413cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 0x7fdb3413cd90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdb340a72d0 tx=0x7fdb1c019180 comp rx=0 tx=0).stop 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 msgr2=0x7fdb3413cd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.120+0000 7fdb2b7fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34136fb0 0x7fdb3413cd90 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdb340a72d0 tx=0x7fdb1c019180 comp rx=0 tx=0).stop 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 0x7fdb3413d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdb34121800 con 0x7fdb34121d70 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 0x7fdb3413d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient: get_auth_request con 0x7fdb34121d70 auth_method 0 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:10.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 10 monclient(hunting): handle_auth_done global_id 4615 payload 824 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 1 -- 172.21.15.43:0/641331624 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdb3413f030 con 0x7fdb34121d70 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb32ffd640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 0x7fdb3413d2d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdb340a7a50 tx=0x7fdb1c0a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 client.4486.objecter ms_handle_connect 0x7fdb34121d70 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 client.4486.objecter resend_mon_ops 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb1c0b0040 con 0x7fdb34121d70 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 monclient: dump: 2024-04-19T00:45:10.149 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:10.150 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9431 (secure 0 0 0) 0x7fdb1c003830 con 0x7fdb34121d70 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 msgr2=0x7fdb3413d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 0x7fdb3413d2d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdb340a7a50 tx=0x7fdb1c0a8000 comp rx=0 tx=0).stop 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 msgr2=0x7fdb3413d2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb2b7fe640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb34121d70 0x7fdb3413d2d0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdb340a7a50 tx=0x7fdb1c0a8000 comp rx=0 tx=0).stop 2024-04-19T00:45:10.151 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 18, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 53587, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:45:10.152 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:10.153 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:10.154 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:10.155 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:45:10.156 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:45:10.157 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.160 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:45:10.162 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.164 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.165 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.168 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:45:10.170 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.173 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:45:10.174 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 10 librados: watch_flush enter 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 10 librados: watch_flush exit 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 20 client.4486.objecter shutdown clearing up homeless session... 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 10 client.4486.objecter successfully canceled tick 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb14038e10 msgr2=0x7fdb1403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb14038e10 0x7fdb1403b2d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdb3413dfb0 tx=0x7fdb240099d0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 10 monclient: shutdown 2024-04-19T00:45:10.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb34138a50 msgr2=0x7fdb3413c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 --2- 172.21.15.43:0/641331624 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb34138a50 0x7fdb3413c850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00d530 tx=0x7fdb2c00d9f0 comp rx=0 tx=0).stop 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb337fe640 1 -- 172.21.15.43:0/641331624 reap_dead start 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb337fe640 1 -- 172.21.15.43:0/641331624 reap_dead start 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 shutdown_connections 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 >> 172.21.15.43:0/641331624 conn(0x7fdb340a7cb0 msgr2=0x7fdb3409ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 shutdown_connections 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 -- 172.21.15.43:0/641331624 wait complete. 2024-04-19T00:45:10.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:45:10.128+0000 7fdb39568640 1 librados: shutdown 2024-04-19T00:45:10.177 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:45:10.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T00:45:25.183 INFO:tasks.workunit.client.0.smithi161.stderr:Note: switching to '8b4ab94f9c30da63487fa2f241ce12f483378860'. 2024-04-19T00:45:25.183 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr:state without impacting any branches by switching back to a branch. 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr: git switch -c 2024-04-19T00:45:25.184 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr:Or undo this operation with: 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr: git switch - 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr: 2024-04-19T00:45:25.185 INFO:tasks.workunit.client.0.smithi161.stderr:HEAD is now at 8b4ab94f9c3 Merge branch 'wip-ectransaction-truncate-fix' of https://github.com/markhpc/ceph into wip-yuri5-testing-2024-04-17-1400 2024-04-19T00:45:25.189 DEBUG:teuthology.orchestra.run.smithi161:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-19T00:45:25.196 INFO:tasks.workunit.client.0.smithi161.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-19T00:45:25.198 INFO:tasks.workunit.client.0.smithi161.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-19T00:45:25.198 INFO:tasks.workunit.client.0.smithi161.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-19T00:45:25.960 INFO:tasks.workunit.client.0.smithi161.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-19T00:45:26.028 INFO:tasks.workunit.client.0.smithi161.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-19T00:45:26.111 INFO:tasks.workunit.client.0.smithi161.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-19T00:45:26.112 INFO:tasks.workunit.client.0.smithi161.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-19T00:45:26.112 INFO:tasks.workunit.client.0.smithi161.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-19T00:45:26.160 INFO:tasks.workunit.client.0.smithi161.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-19T00:45:26.164 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-04-19T00:45:26.164 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-19T00:45:26.211 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-19T00:45:26.212 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-19T00:45:26.212 DEBUG:teuthology.orchestra.run.smithi161:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b4ab94f9c30da63487fa2f241ce12f483378860 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-19T00:45:26.263 INFO:tasks.workunit.client.0.smithi161.stderr:+ parallel=1 2024-04-19T00:45:26.263 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' '' = --serial ']' 2024-04-19T00:45:26.263 INFO:tasks.workunit.client.0.smithi161.stderr:+ crimson=0 2024-04-19T00:45:26.264 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' '' = --crimson ']' 2024-04-19T00:45:26.264 INFO:tasks.workunit.client.0.smithi161.stderr:+ color= 2024-04-19T00:45:26.264 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -t 1 ']' 2024-04-19T00:45:26.264 INFO:tasks.workunit.client.0.smithi161.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-19T00:45:26.264 INFO:tasks.workunit.client.0.smithi161.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-19T00:45:26.264 INFO:tasks.workunit.client.0.smithi161.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-19T00:45:26.265 INFO:tasks.workunit.client.0.smithi161.stderr:+ declare -A pids 2024-04-19T00:45:26.265 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.265 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.265 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_aio 2024-04-19T00:45:26.265 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_aio' 2024-04-19T00:45:26.266 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_aio 2024-04-19T00:45:26.266 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.268 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_aio 2024-04-19T00:45:26.268 INFO:tasks.workunit.client.0.smithi161.stdout:test api_aio on pid 28712 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28712 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_aio on pid 28712' 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28712 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_aio_pp 2024-04-19T00:45:26.269 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_aio_pp' 2024-04-19T00:45:26.270 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.270 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.270 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_aio_pp 2024-04-19T00:45:26.270 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.270 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.270 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.271 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-19T00:45:26.271 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-19T00:45:26.271 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_aio: /' 2024-04-19T00:45:26.271 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_aio_pp 2024-04-19T00:45:26.271 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28720 2024-04-19T00:45:26.271 INFO:tasks.workunit.client.0.smithi161.stdout:test api_aio_pp on pid 28720 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_aio_pp on pid 28720' 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28720 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_io 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_io' 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_io 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.273 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.274 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.274 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-19T00:45:26.274 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-19T00:45:26.274 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-19T00:45:26.274 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_io 2024-04-19T00:45:26.275 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28728 2024-04-19T00:45:26.275 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_io on pid 28728' 2024-04-19T00:45:26.275 INFO:tasks.workunit.client.0.smithi161.stdout:test api_io on pid 28728 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28728 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_io_pp 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_io_pp' 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.276 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_io_pp 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_io.log 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_io: /' 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_io_pp 2024-04-19T00:45:26.277 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28736 2024-04-19T00:45:26.278 INFO:tasks.workunit.client.0.smithi161.stdout:test api_io_pp on pid 28736 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_io_pp on pid 28736' 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28736 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_asio 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_asio' 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.279 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.280 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_asio 2024-04-19T00:45:26.280 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.280 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-19T00:45:26.280 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-19T00:45:26.281 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_asio 2024-04-19T00:45:26.281 INFO:tasks.workunit.client.0.smithi161.stdout:test api_asio on pid 28744 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28744 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_asio on pid 28744' 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28744 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-19T00:45:26.282 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_list 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_list' 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_list 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-19T00:45:26.284 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-19T00:45:26.285 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_asio: /' 2024-04-19T00:45:26.285 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_list 2024-04-19T00:45:26.285 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28752 2024-04-19T00:45:26.285 INFO:tasks.workunit.client.0.smithi161.stdout:test api_list on pid 28752 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_list on pid 28752' 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28752 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_lock 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_lock' 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_lock 2024-04-19T00:45:26.287 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.288 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.288 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.288 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.288 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-19T00:45:26.288 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_list.log 2024-04-19T00:45:26.288 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_list: /' 2024-04-19T00:45:26.289 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.291 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_lock 2024-04-19T00:45:26.291 INFO:tasks.workunit.client.0.smithi161.stdout:test api_lock on pid 28760 2024-04-19T00:45:26.292 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28760 2024-04-19T00:45:26.292 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_lock on pid 28760' 2024-04-19T00:45:26.292 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28760 2024-04-19T00:45:26.292 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.292 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-19T00:45:26.292 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.293 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.293 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.293 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.293 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.293 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_lock_pp 2024-04-19T00:45:26.294 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-19T00:45:26.294 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_lock_pp' 2024-04-19T00:45:26.294 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-19T00:45:26.294 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.297 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_lock_pp 2024-04-19T00:45:26.298 INFO:tasks.workunit.client.0.smithi161.stdout:test api_lock_pp on pid 28768 2024-04-19T00:45:26.298 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_lock_pp 2024-04-19T00:45:26.299 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28768 2024-04-19T00:45:26.299 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_lock_pp on pid 28768' 2024-04-19T00:45:26.299 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28768 2024-04-19T00:45:26.299 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.299 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.306 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_lock: /' 2024-04-19T00:45:26.306 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_misc 2024-04-19T00:45:26.306 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_misc' 2024-04-19T00:45:26.307 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.310 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-19T00:45:26.311 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_misc 2024-04-19T00:45:26.311 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.312 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.312 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.312 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.313 INFO:tasks.workunit.client.0.smithi161.stdout:test api_misc on pid 28793 2024-04-19T00:45:26.313 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_misc 2024-04-19T00:45:26.313 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-19T00:45:26.313 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28793 2024-04-19T00:45:26.313 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_misc on pid 28793' 2024-04-19T00:45:26.314 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28793 2024-04-19T00:45:26.314 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.314 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.314 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-19T00:45:26.314 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-19T00:45:26.315 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_misc_pp 2024-04-19T00:45:26.315 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_misc_pp' 2024-04-19T00:45:26.315 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.315 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.315 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.315 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.316 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_misc: /' 2024-04-19T00:45:26.316 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_misc_pp 2024-04-19T00:45:26.316 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-19T00:45:26.316 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-19T00:45:26.316 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.319 INFO:tasks.workunit.client.0.smithi161.stdout:test api_misc_pp on pid 28822 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_misc_pp 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28822 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_misc_pp on pid 28822' 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28822 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-19T00:45:26.320 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_tier_pp 2024-04-19T00:45:26.321 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_tier_pp' 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_tier_pp 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-19T00:45:26.322 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-19T00:45:26.323 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-19T00:45:26.326 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_tier_pp 2024-04-19T00:45:26.326 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28848 2024-04-19T00:45:26.326 INFO:tasks.workunit.client.0.smithi161.stdout:test api_tier_pp on pid 28848 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_tier_pp on pid 28848' 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28848 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_pool 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_pool' 2024-04-19T00:45:26.327 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.328 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.328 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.328 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.328 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.329 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-19T00:45:26.330 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_pool 2024-04-19T00:45:26.331 INFO:tasks.workunit.client.0.smithi161.stdout:test api_pool on pid 28865 2024-04-19T00:45:26.331 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_pool 2024-04-19T00:45:26.331 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28865 2024-04-19T00:45:26.332 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_pool on pid 28865' 2024-04-19T00:45:26.332 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28865 2024-04-19T00:45:26.332 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.332 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.334 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-19T00:45:26.334 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-19T00:45:26.335 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.335 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.336 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_snapshots 2024-04-19T00:45:26.336 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.336 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.336 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_snapshots' 2024-04-19T00:45:26.336 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_pool: /' 2024-04-19T00:45:26.337 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-19T00:45:26.337 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-19T00:45:26.338 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-19T00:45:26.339 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_snapshots 2024-04-19T00:45:26.339 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.340 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_snapshots 2024-04-19T00:45:26.341 INFO:tasks.workunit.client.0.smithi161.stdout:test api_snapshots on pid 28886 2024-04-19T00:45:26.341 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28886 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_snapshots on pid 28886' 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28886 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_snapshots_pp 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_snapshots_pp' 2024-04-19T00:45:26.342 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.343 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-19T00:45:26.343 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [----------] Global test environment set-up. 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient: get_monmap_and_config 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient: build_initial_monmap 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient: monmap: 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: epoch 0 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: last_changed 0.000000 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: created 0.000000 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: min_mon_release 0 (unknown) 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: election_strategy: 1 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:26.344 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 Processor -- start 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 -- start start 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient: init 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient: _reopen_session rank -1 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient: _add_conns ranks=[5,7,3,8,1,6,4,2,0] 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x562cab691440 0x562cab7e3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.345 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): picked mon.noname-h con 0x562cab691440 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691d60 0x562cab7e3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): picked mon.noname-i con 0x562cab691d60 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 0x562cab7e4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): picked mon.noname-g con 0x562cab6a44c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x562cab6a1f30 con 0x562cab6a44c0 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:26.346 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x562cab6a2140 con 0x562cab691440 2024-04-19T00:45:26.347 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.347 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.347 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.347 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x562cab6a48a0 con 0x562cab691d60 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): _renew_subs 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a3d16880 10 monclient(hunting): authenticate will time out at 1029.289742s 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1835640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x562cab691440 msgr2=0x562cab7e3610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1835640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x562cab691440 0x562cab7e3610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 0x562cab7e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 0x562cab7e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.161:32850/0 (socket says 172.21.15.161:32850) 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 -- 172.21.15.161:0/90048660 learned_addr learned my addr 172.21.15.161:0/90048660 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:45:26.348 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): get_auth_request con 0x562cab6a44c0 auth_method 0 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691d60 0x562cab7e3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): get_auth_request con 0x562cab691d60 auth_method 0 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:26.349 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a1034640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient(hunting): handle_auth_done global_id 4557 payload 274 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 -- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691d60 msgr2=0x562cab7e3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691d60 0x562cab7e3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 -- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x562cab691440 msgr2=0x562cab7e3610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x562cab691440 0x562cab7e3610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient: _finish_hunting 0 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 monclient: found mon.noname-g 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:45:26.350 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.341+0000 7fc3a2036640 1 -- 172.21.15.161:0/90048660 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cab5f6640 con 0x562cab6a44c0 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a2036640 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 0x562cab7e4090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0096e0 tx=0x7fc39c009ba0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=fcdd35284a28ecb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 1 -- 172.21.15.161:0/90048660 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc39c014070 con 0x562cab6a44c0 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: dump: 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: epoch 1 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: min_mon_release 19 (squid) 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: election_strategy: 3 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:26.351 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.352 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: _finish_auth 0 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: _check_auth_tickets 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 1 -- 172.21.15.161:0/90048660 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc39c00bba0 con 0x562cab6a44c0 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 5 monclient: authenticate success, global_id 4557 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 1 -- 172.21.15.161:0/90048660 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc39c00bd50 con 0x562cab6a44c0 2024-04-19T00:45:26.353 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc393fff640 10 monclient: dump: 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: epoch 1 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: min_mon_release 19 (squid) 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: election_strategy: 3 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:26.354 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 10 monclient: get_monmap_and_config success 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 4 set_mon_vals no callback set 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 10 monclient: shutdown 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 -- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 msgr2=0x562cab7e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 0x562cab7e4090 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc39c0096e0 tx=0x7fc39c009ba0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 -- 172.21.15.161:0/90048660 shutdown_connections 2024-04-19T00:45:26.355 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x562cab6a44c0 0x562cab7e4090 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691d60 0x562cab7e3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/90048660 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x562cab691440 0x562cab7e3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 -- 172.21.15.161:0/90048660 >> 172.21.15.161:0/90048660 conn(0x562cab6a7ff0 msgr2=0x562cab72e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 -- 172.21.15.161:0/90048660 shutdown_connections 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 -- 172.21.15.161:0/90048660 wait complete. 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 10 monclient: build_initial_monmap 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 10 monclient: monmap: 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: epoch 0 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: last_changed 0.000000 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: created 0.000000 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: min_mon_release 0 (unknown) 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: election_strategy: 1 2024-04-19T00:45:26.356 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 librados: starting msgr at 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 librados: starting objecter 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 Processor -- start 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 -- start start 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 librados: setting wanted keys 2024-04-19T00:45:26.357 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 1 librados: calling monclient init 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.345+0000 7fc3a3d16880 10 monclient: init 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient: _reopen_session rank -1 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient: _add_conns ranks=[7,1,4,5,2,8,6,3,0] 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691440 0x562cab67a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): picked mon.noname-i con 0x562cab691440 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cab691d60 0x562cab7eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): picked mon.noname-f con 0x562cab691d60 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x562cab6a44c0 0x562cab737bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): picked mon.noname-c con 0x562cab6a44c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:45:26.358 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x562cab6799e0 con 0x562cab691d60 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x562cab72d720 con 0x562cab6a44c0 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): start opening mon connection 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x562cab6a59f0 con 0x562cab691440 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): _renew_subs 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a3d16880 10 monclient(hunting): authenticate will time out at 1029.296712s 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cab691d60 0x562cab7eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cab691d60 0x562cab7eabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37998/0 (socket says 172.21.15.161:37998) 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 -- 172.21.15.161:0/4143917193 learned_addr learned my addr 172.21.15.161:0/4143917193 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): get_auth_request con 0x562cab691d60 auth_method 0 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:26.359 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691440 0x562cab67a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): get_auth_request con 0x562cab691440 auth_method 0 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x562cab6a44c0 0x562cab737bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:26.360 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-19T00:45:26.361 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-19T00:45:26.361 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): get_auth_request con 0x562cab6a44c0 auth_method 0 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1835640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:26.362 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a2036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient(hunting): handle_auth_done global_id 4600 payload 824 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 -- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691440 msgr2=0x562cab67a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x562cab691440 0x562cab67a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 -- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x562cab6a44c0 msgr2=0x562cab737bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x562cab6a44c0 0x562cab737bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient: _finish_hunting 0 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 monclient: found mon.noname-f 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562cab734f10 con 0x562cab691d60 2024-04-19T00:45:26.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.349+0000 7fc3a1034640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cab691d60 0x562cab7eabc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc398006700 tx=0x7fc398006bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:26.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 client.?.objecter ms_handle_connect 0x562cab691d60 2024-04-19T00:45:26.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:45:26.364 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_snapshots_pp 2024-04-19T00:45:26.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 -- 172.21.15.161:0/414391test api_snapshots_pp on pid 28932 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_snapshots_pp 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28932 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_snapshots_pp on pid 28932' 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28932 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-19T00:45:26.365 INFO:tasks.workunit.client.0.smithi161.stdout:7193 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3980092e0 con 0x562cab691d60 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: dump: 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: epoch 1 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: min_mon_release 19 (squid) 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: election_strategy: 3 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:26.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: _finish_auth 0 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 5 monclient: authenticate success, global_id 4600 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 monclient: _renew_subs 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:26.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562cab7351b0 con 0x562cab691d60 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 monclient: _renew_subs 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562cab7356f0 con 0x562cab691d60 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 1 librados: init done 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 librados: wait_for_osdmap waiting 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc398019070 con 0x562cab691d60 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc398006e10 con 0x562cab691d60 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:45:26.368 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: dump: 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: epoch 1 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: min_mon_release 19 (squid) 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: election_strategy: 3 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:45:26.369 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3937fe640 4 set_mon_vals no callback set 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc398012050 con 0x562cab691d60 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc384036e70 0x7fc384039330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fc39804c170 con 0x562cab691d60 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_map(15..15 src has 1..15) 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 3 client.4600.objecter handle_osd_map got epochs [15,15] > 0 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 3 client.4600.objecter handle_osd_map decoding full epoch 15 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 20 client.4600.objecter dump_active .. 0 homeless 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a1835640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc384036e70 0x7fc384039330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:26.370 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 librados: wait_for_osdmap done waiting 2024-04-19T00:45:26.371 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:26.371 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc39c00bba0 con 0x562cab691d60 2024-04-19T00:45:26.371 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a1835640 10 monclient: get_auth_request con 0x7fc384036e70 auth_method 0 2024-04-19T00:45:26.371 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fc39804d060 con 0x562cab691d60 2024-04-19T00:45:26.371 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_stat 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_stat' 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_stat 2024-04-19T00:45:26.372 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-19T00:45:26.374 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-19T00:45:26.375 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.379 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a1835640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:6824/2963482test api_stat on pid 28960 2024-04-19T00:45:26.379 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_stat 2024-04-19T00:45:26.379 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28960 2024-04-19T00:45:26.379 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_stat on pid 28960' 2024-04-19T00:45:26.380 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=28960 2024-04-19T00:45:26.380 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.380 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.380 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-19T00:45:26.382 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.382 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.382 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_stat_pp 2024-04-19T00:45:26.383 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.383 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.383 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_stat_pp' 2024-04-19T00:45:26.383 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_stat: /' 2024-04-19T00:45:26.384 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-19T00:45:26.384 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-19T00:45:26.393 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_stat_pp 2024-04-19T00:45:26.393 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.395 INFO:tasks.workunit.client.0.smithi161.stdout:test api_stat_pp on pid 29010 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_stat_pp 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29010 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_stat_pp on pid 29010' 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29010 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_watch_notify 2024-04-19T00:45:26.396 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_watch_notify' 2024-04-19T00:45:26.397 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_watch_notify 2024-04-19T00:45:26.397 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.397 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.398 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.398 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.398 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.399 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-19T00:45:26.400 INFO:tasks.workunit.client.0.smithi161.stdout:test api_watch_notify on pid 29035 2024-04-19T00:45:26.400 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_watch_notify 2024-04-19T00:45:26.400 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29035 2024-04-19T00:45:26.400 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_watch_notify on pid 29035' 2024-04-19T00:45:26.400 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29035 2024-04-19T00:45:26.400 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.401 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.401 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_watch_notify_pp 2024-04-19T00:45:26.401 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-19T00:45:26.401 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_watch_notify_pp' 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_watch_notify_pp 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.402 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-19T00:45:26.403 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-19T00:45:26.403 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-19T00:45:26.404 INFO:tasks.workunit.client.0.smithi161.stdout:test api_watch_notify_pp on pid 29046 2024-04-19T00:45:26.404 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_watch_notify_pp 2024-04-19T00:45:26.404 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29046 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_watch_notify_pp on pid 29046' 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29046 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_cmd 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_cmd' 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-19T00:45:26.405 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_cmd 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.406 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-19T00:45:26.411 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-19T00:45:26.412 INFO:tasks.workunit.client.0.smithi161.stdout:test api_cmd on pid 29080 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_cmd 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29080 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_cmd on pid 29080' 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29080 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_cmd_pp 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_cmd_pp' 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_cmd_pp 2024-04-19T00:45:26.413 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.414 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_cmd: /' 2024-04-19T00:45:26.415 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_cmd_pp 2024-04-19T00:45:26.416 INFO:tasks.workunit.client.0.smithi161.stdout:test api_cmd_pp on pid 29088 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29088 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_cmd_pp on pid 29088' 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29088 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_service 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_service' 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.417 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-19T00:45:26.418 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_service 2024-04-19T00:45:26.429 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.432 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-19T00:45:26.432 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_service 2024-04-19T00:45:26.432 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-19T00:45:26.432 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29108 2024-04-19T00:45:26.432 INFO:tasks.workunit.client.0.smithi161.stdout:test api_service on pid 29108 2024-04-19T00:45:26.433 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_service on pid 29108' 2024-04-19T00:45:26.433 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29108 2024-04-19T00:45:26.433 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.433 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.433 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-19T00:45:26.434 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.434 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.434 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.434 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.434 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-19T00:45:26.442 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_service: /' 2024-04-19T00:45:26.448 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_service.log 2024-04-19T00:45:26.451 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_service_pp 2024-04-19T00:45:26.451 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_service_pp' 2024-04-19T00:45:26.452 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.452 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_service_pp 2024-04-19T00:45:26.455 INFO:tasks.workunit.client.0.smithi161.stdout:test api_service_pp on pid 29175 2024-04-19T00:45:26.456 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_service_pp 2024-04-19T00:45:26.456 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29175 2024-04-19T00:45:26.456 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_service_pp on pid 29175' 2024-04-19T00:45:26.456 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29175 2024-04-19T00:45:26.456 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.457 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.457 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_c_write_operations 2024-04-19T00:45:26.457 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_c_write_operations' 2024-04-19T00:45:26.457 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-19T00:45:26.457 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_c_write_operations 2024-04-19T00:45:26.458 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.458 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.459 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.459 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.459 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-19T00:45:26.459 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-19T00:45:26.460 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-19T00:45:26.460 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.464 INFO:tasks.workunit.client.0.smithi161.stdout:test api_c_write_operations on pid 29213 2024-04-19T00:45:26.464 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_c_write_operations 2024-04-19T00:45:26.464 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29213 2024-04-19T00:45:26.464 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_c_write_operations on pid 29213' 2024-04-19T00:45:26.464 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29213 2024-04-19T00:45:26.465 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.465 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.465 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-19T00:45:26.465 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_c_read_operations 2024-04-19T00:45:26.465 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_c_read_operations' 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-19T00:45:26.466 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-19T00:45:26.467 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_c_read_operations 2024-04-19T00:45:26.467 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.468 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_c_read_operations 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stdout:test api_c_read_operations on pid 29238 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29238 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_c_read_operations on pid 29238' 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29238 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-19T00:45:26.469 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s api_cls_remote_reads 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' api_cls_remote_reads' 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo api_cls_remote_reads 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.471 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-19T00:45:26.473 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=api_cls_remote_reads 2024-04-19T00:45:26.473 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29247 2024-04-19T00:45:26.473 INFO:tasks.workunit.client.0.smithi161.stdout:test api_cls_remote_reads on pid 29247 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test api_cls_remote_reads on pid 29247' 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29247 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s list_parallel 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' list_parallel' 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo list_parallel 2024-04-19T00:45:26.475 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.476 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-19T00:45:26.476 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-19T00:45:26.476 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-19T00:45:26.476 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=list_parallel 2024-04-19T00:45:26.477 INFO:tasks.workunit.client.0.smithi161.stdout:test list_parallel on pid 29255 2024-04-19T00:45:26.477 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29255 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test list_parallel on pid 29255' 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29255 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-19T00:45:26.478 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ list_parallel: /' 2024-04-19T00:45:26.479 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s open_pools_parallel 2024-04-19T00:45:26.479 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' open_pools_parallel' 2024-04-19T00:45:26.479 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo open_pools_parallel 2024-04-19T00:45:26.479 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stdout:test open_pools_parallel on pid 29263 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=open_pools_parallel 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29263 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test open_pools_parallel on pid 29263' 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29263 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-19T00:45:26.481 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.482 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s delete_pools_parallel 2024-04-19T00:45:26.482 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.482 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.483 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.483 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.483 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' delete_pools_parallel' 2024-04-19T00:45:26.483 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-19T00:45:26.483 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-19T00:45:26.497 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.502 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-19T00:45:26.504 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo delete_pools_parallel 2024-04-19T00:45:26.505 INFO:tasks.workunit.client.0.smithi161.stdout:test delete_pools_parallel on pid 29282 2024-04-19T00:45:26.505 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=delete_pools_parallel 2024-04-19T00:45:26.505 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29282 2024-04-19T00:45:26.505 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test delete_pools_parallel on pid 29282' 2024-04-19T00:45:26.505 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29282 2024-04-19T00:45:26.506 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.506 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.510 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-19T00:45:26.511 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.511 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.511 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.511 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.512 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-19T00:45:26.513 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-19T00:45:26.516 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s cls 2024-04-19T00:45:26.516 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' cls' 2024-04-19T00:45:26.520 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-19T00:45:26.521 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo cls 2024-04-19T00:45:26.522 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.530 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=cls 2024-04-19T00:45:26.531 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29357 2024-04-19T00:45:26.531 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test cls on pid 29357' 2024-04-19T00:45:26.531 INFO:tasks.workunit.client.0.smithi161.stdout:test cls on pid 29357 2024-04-19T00:45:26.532 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29357 2024-04-19T00:45:26.532 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.532 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.532 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s cmd 2024-04-19T00:45:26.532 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' cmd' 2024-04-19T00:45:26.532 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-19T00:45:26.533 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.533 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.534 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.534 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.534 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_cls 2024-04-19T00:45:26.542 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ cls: /' 2024-04-19T00:45:26.542 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.542 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_cls.log 2024-04-19T00:45:26.548 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo cmd 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stdout:test cmd on pid 29433 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=cmd 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29433 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test cmd on pid 29433' 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29433 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.550 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.553 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s handler_error 2024-04-19T00:45:26.553 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' handler_error' 2024-04-19T00:45:26.555 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo handler_error 2024-04-19T00:45:26.555 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.560 INFO:tasks.workunit.client.0.smithi161.stdout:test handler_error on pid 29450 2024-04-19T00:45:26.560 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=handler_error 2024-04-19T00:45:26.560 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29450 2024-04-19T00:45:26.560 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test handler_error on pid 29450' 2024-04-19T00:45:26.560 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29450 2024-04-19T00:45:26.561 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.561 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.561 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s io 2024-04-19T00:45:26.561 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' io' 2024-04-19T00:45:26.561 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-19T00:45:26.562 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.562 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.562 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.562 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.563 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ handler_error: /' 2024-04-19T00:45:26.564 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-19T00:45:26.564 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-19T00:45:26.566 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.567 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo io 2024-04-19T00:45:26.569 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_handler_error 2024-04-19T00:45:26.572 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.572 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.572 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.572 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.575 INFO:tasks.workunit.client.0.smithi161.stdout:test io on pid 29476 2024-04-19T00:45:26.575 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=io 2024-04-19T00:45:26.575 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29476 2024-04-19T00:45:26.575 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test io on pid 29476' 2024-04-19T00:45:26.575 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29476 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stdout:743,v1:172.21.15.161:6825/2963482743] conn(0x7fc384036e70 0x7fc384039330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc39400a5f0 tx=0x7fc3940a5070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc393fff640 10 client.4600.objecter _get_latest_version latest 15, have it 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc392ffd640 10 client.4600.objecter ms_handle_connect 0x7fc384036e70 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 client.4600.objecter create_pool name=ceph_test_rados_api_asio 2024-04-19T00:45:26.576 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 client.4600.objecter pool_op_submit 1 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.361+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x562cab62faf0 con 0x562cab691d60 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v16) ==== 43+0+0 (secure 0 0 0) 0x7fc398010040 con 0x562cab691d60 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 client.4600.objecter ms_dispatch 0x562cab600170 pool_op_reply(tid 1 (0) Success v16) 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 client.4600.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v16) 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 client.4600.objecter have request 1 at 0x562cab734080 Op: create 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 20 client.4600.objecter waiting for client to reach epoch 16 before calling back 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 monclient: _renew_subs 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=16}) -- 0x7fc38403c5d0 con 0x562cab691d60 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.569+0000 7fc392ffd640 10 client.4600.objecter done 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== mon.1 v2:172.21.15.161:3300/0 8 ==== osd_map(16..16 src has 1..16) ==== 6911+0+0 (secure 0 0 0) 0x7fc3980149a0 con 0x562cab691d60 2024-04-19T00:45:26.577 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc392ffd640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_map(16..16 src has 1..16) 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc392ffd640 3 client.4600.objecter handle_osd_map got epochs [16,16] > 15 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc392ffd640 3 client.4600.objecter handle_osd_map decoding incremental epoch 16 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc392ffd640 20 client.4600.objecter dump_active .. 0 homeless 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 10 librados: set snap read head -> 1 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 10 librados: writefull oid=exist nspace= 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7380f0 2024-04-19T00:45:26.578 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 16 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:26.582 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_cmd 2024-04-19T00:45:26.582 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s ec_io 2024-04-19T00:45:26.582 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-19T00:45:26.582 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' ec_io' 2024-04-19T00:45:26.582 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-19T00:45:26.584 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.584 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.584 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.584 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.584 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_io.log 2024-04-19T00:45:26.594 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_io 2024-04-19T00:45:26.597 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ io: /' 2024-04-19T00:45:26.618 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.618 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo ec_io 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59test ec_io on pid 29543 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=ec_io 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29543 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test ec_io on pid 29543' 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29543 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.621 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.627 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-19T00:45:26.628 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.629 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s list 2024-04-19T00:45:26.630 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ cmd: /' 2024-04-19T00:45:26.631 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' list' 2024-04-19T00:45:26.631 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.631 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.631 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.636 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_ec_io 2024-04-19T00:45:26.642 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ ec_io: /' 2024-04-19T00:45:26.642 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-19T00:45:26.642 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.642 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo list 2024-04-19T00:45:26.648 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=list 2024-04-19T00:45:26.648 INFO:tasks.workunit.client.0.smithi161.stdout:test list on pid 29603 2024-04-19T00:45:26.648 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29603 2024-04-19T00:45:26.649 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test list on pid 29603' 2024-04-19T00:45:26.649 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29603 2024-04-19T00:45:26.649 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.649 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.652 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s ec_list 2024-04-19T00:45:26.653 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' ec_list' 2024-04-19T00:45:26.653 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo ec_list 2024-04-19T00:45:26.653 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stdout:test ec_list on pid 29620 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=ec_list 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29620 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test ec_list on pid 29620' 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29620 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.655 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-19T00:45:26.656 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.656 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.656 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.656 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.656 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_ec_list 2024-04-19T00:45:26.657 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ ec_list: /' 2024-04-19T00:45:26.657 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-19T00:45:26.657 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-19T00:45:26.658 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s misc 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' misc' 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo misc 2024-04-19T00:45:26.659 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.660 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_list 2024-04-19T00:45:26.660 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_list.log 2024-04-19T00:45:26.660 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ list: /' 2024-04-19T00:45:26.661 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=misc 2024-04-19T00:45:26.661 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29634 2024-04-19T00:45:26.661 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test misc on pid 29634' 2024-04-19T00:45:26.661 INFO:tasks.workunit.client.0.smithi161.stdout:test misc on pid 29634 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29634 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s pool 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' pool' 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo pool 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.663 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_misc 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_misc.log 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ misc: /' 2024-04-19T00:45:26.664 INFO:tasks.workunit.client.0.smithi161.stdout:test pool on pid 29647 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=pool 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29647 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test pool on pid 29647' 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29647 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.665 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-19T00:45:26.666 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.666 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.666 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.666 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.667 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s read_operations 2024-04-19T00:45:26.667 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_pool 2024-04-19T00:45:26.667 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' read_operations' 2024-04-19T00:45:26.667 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_pool.log 2024-04-19T00:45:26.667 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo read_operations 2024-04-19T00:45:26.670 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.670 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ pool: /' 2024-04-19T00:45:26.672 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=read_operations 2024-04-19T00:45:26.672 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29661 2024-04-19T00:45:26.672 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test read_operations on pid 29661' 2024-04-19T00:45:26.673 INFO:tasks.workunit.client.0.smithi161.stdout:test read_operations on pid 29661 2024-04-19T00:45:26.675 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29661 2024-04-19T00:45:26.675 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.675 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.675 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s snapshots 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' snapshots' 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_read_operations 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-19T00:45:26.676 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.678 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo snapshots 2024-04-19T00:45:26.678 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ read_operations: /' 2024-04-19T00:45:26.680 INFO:tasks.workunit.client.0.smithi161.stdout:test snapshots on pid 29670 2024-04-19T00:45:26.680 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=snapshots 2024-04-19T00:45:26.680 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29670 2024-04-19T00:45:26.681 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test snapshots on pid 29670' 2024-04-19T00:45:26.681 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29670 2024-04-19T00:45:26.681 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.681 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.681 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-19T00:45:26.681 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s watch_notify 2024-04-19T00:45:26.682 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' watch_notify' 2024-04-19T00:45:26.682 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.682 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.682 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.682 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo watch_notify 2024-04-19T00:45:26.683 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.683 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.683 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ snapshots: /' 2024-04-19T00:45:26.684 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-19T00:45:26.684 INFO:tasks.workunit.client.0.smithi161.stdout:test watch_notify on pid 29686 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=watch_notify 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29686 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test watch_notify on pid 29686' 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29686 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_snapshots 2024-04-19T00:45:26.685 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-19T00:45:26.686 INFO:tasks.workunit.client.0.smithi161.stderr:++ printf %25s write_operations 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ r=' write_operations' 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:++ echo write_operations 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_watch_notify 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-19T00:45:26.687 INFO:tasks.workunit.client.0.smithi161.stderr:++ awk '{print $1}' 2024-04-19T00:45:26.689 INFO:tasks.workunit.client.0.smithi161.stderr:+ ff=write_operations 2024-04-19T00:45:26.689 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29703 2024-04-19T00:45:26.689 INFO:tasks.workunit.client.0.smithi161.stdout:test write_operations on pid 29703 2024-04-19T00:45:26.690 INFO:tasks.workunit.client.0.smithi161.stderr:+ echo 'test write_operations on pid 29703' 2024-04-19T00:45:26.690 INFO:tasks.workunit.client.0.smithi161.stderr:+ pids[$f]=29703 2024-04-19T00:45:26.690 INFO:tasks.workunit.client.0.smithi161.stderr:+ ret=0 2024-04-19T00:45:26.690 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' 1 -eq 1 ']' 2024-04-19T00:45:26.690 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T00:45:26.690 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28848 2024-04-19T00:45:26.691 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28848 2024-04-19T00:45:26.691 INFO:tasks.workunit.client.0.smithi161.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-19T00:45:26.691 INFO:tasks.workunit.client.0.smithi161.stderr:+ '[' -z '' ']' 2024-04-19T00:45:26.691 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.692 INFO:tasks.workunit.client.0.smithi161.stderr:+ case $- in 2024-04-19T00:45:26.692 INFO:tasks.workunit.client.0.smithi161.stderr:+ return 2024-04-19T00:45:26.692 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ watch_notify: /' 2024-04-19T00:45:26.692 INFO:tasks.workunit.client.0.smithi161.stderr:+ sed 's/^/ write_operations: /' 2024-04-19T00:45:26.692 INFO:tasks.workunit.client.0.smithi161.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-19T00:45:26.693 INFO:tasks.workunit.client.0.smithi161.stderr:+ ceph_test_neorados_write_operations 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout:201 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x562cab7f1f30 0x562cab7f4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 2 osd.1 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 2 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 15 client.4600.objecter _send_op 2 to 8.1 on osd.1 2024-04-19T00:45:27.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:2 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x562cab7f4860 con 0x562cab7f1f30 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.573+0000 7fc3a2036640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x562cab7f1f30 0x562cab7f4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.577+0000 7fc3a2036640 10 monclient: get_auth_request con 0x562cab7f1f30 auth_method 0 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.613+0000 7fc3a2036640 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x562cab7f1f30 0x562cab7f4320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:26.613+0000 7fc392ffd640 10 client.4600.objecter ms_handle_connect 0x562cab7f1f30 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 1 ==== osd_map(17..17 src has 1..17) ==== 14078+0+0 (crc 0 0 0) 0x7fc39c00b730 con 0x562cab7f1f30 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_map(17..17 src has 1..17) 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 3 client.4600.objecter handle_osd_map got epochs [17,17] > 16 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 3 client.4600.objecter handle_osd_map decoding incremental epoch 17 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 10 client.4600.objecter checking op 2 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 20 client.4600.objecter dump_active .. 0 homeless 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.809+0000 7fc392ffd640 20 client.4600.objecter 2 8.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 2 ==== osd_op_reply(2 exist [writefull 0~5] v17'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(2 exist [writefull 0~5] v17'1 uv1 ondisk = 0) 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 2 ondisk uv 1 in 8.1 attempt 0 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 2 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 15 client.4600.objecter _finish_op 2 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 2 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.840 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 librados: Objecter returned from writefull r=0 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7380f0 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [read 0~256] tid 3 osd.1 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 3 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 15 client.4600.objecter _send_op 3 to 8.1 on osd.1 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:3 8.1 8:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab7329c0 con 0x562cab7f1f30 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.841 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab732d60 2024-04-19T00:45:27.842 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.842 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2024-04-19T00:45:27.842 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.842 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.842 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid noexist '@8' '@8' [read 0~256] tid 4 osd.1 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 4 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 15 client.4600.objecter _send_op 4 to 8.1 on osd.1 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:4 8.1 8:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab733210 con 0x562cab7f1f30 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.833+0000 7fc3a3d16880 5 client.4600.objecter 2 in flight 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.843 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 3 ondisk uv 1 in 8.1 attempt 0 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 5 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 3 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _finish_op 3 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 3 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-19T00:45:27.844 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 4 ondisk uv 0 in 8.1 attempt 0 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 4 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _finish_op 4 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.845 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 4 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab732d60 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [read 0~256] tid 5 osd.1 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.846 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 5 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _send_op 5 to 8.1 on osd.1 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:5 8.1 8:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab733600 con 0x562cab7f1f30 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7380f0 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.847 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid noexist '@8' '@8' [read 0~256] tid 6 osd.1 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 6 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _send_op 6 to 8.1 on osd.1 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:6 8.1 8:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f4c00 con 0x562cab7f1f30 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 5 client.4600.objecter 2 in flight 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 5 ondisk uv 1 in 8.1 attempt 0 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 5 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 5 2024-04-19T00:45:27.848 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _finish_op 5 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 5 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 6 ondisk uv 0 in 8.1 attempt 0 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 6 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _finish_op 6 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 6 2024-04-19T00:45:27.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab732d60 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [read 0~256] tid 7 osd.1 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 7 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _send_op 7 to 8.1 on osd.1 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:7 8.1 8:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab7380f0 con 0x562cab7f1f30 2024-04-19T00:45:27.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab835630 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid noexist '@8' '@8' [read 0~256] tid 8 osd.1 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 8 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _send_op 8 to 8.1 on osd.1 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:8 8.1 8:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab835ae0 con 0x562cab7f1f30 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 5 client.4600.objecter 2 in flight 2024-04-19T00:45:27.851 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 7 ondisk uv 1 in 8.1 attempt 0 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 5 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 7 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _finish_op 7 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 7 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.852 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 8 ondisk uv 0 in 8.1 attempt 0 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 8 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _finish_op 8 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 8 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.854 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 librados: queue_aio_write 0x562cab5e92f0 completion 0x562cab5c1300 write_seq 1 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab732d60 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [write 0~5 in=5b] tid 9 osd.1 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 9 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 15 client.4600.objecter _send_op 9 to 8.1 on osd.1 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:9 8.1 8:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f5fb0 con 0x562cab7f1f30 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.855 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.837+0000 7fc3a3d16880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 9 ==== osd_op_reply(9 exist [write 0~5] v17'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(9 exist [write 0~5] v17'2 uv2 ondisk = 0) 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 9 ondisk uv 2 in 8.1 attempt 0 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 9 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 15 client.4600.objecter _finish_op 9 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 9 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 20 librados: complete_aio_write 0x562cab5c1300 2024-04-19T00:45:27.856 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 librados: queue_aio_write 0x562cab5e92f0 completion 0x562cab5ce330 write_seq 2 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab732d60 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [write 0~5 in=5b] tid 10 osd.1 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 10 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 15 client.4600.objecter _send_op 10 to 8.1 on osd.1 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:10 8.1 8:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f5fb0 con 0x562cab7f1f30 2024-04-19T00:45:27.857 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 10 ==== osd_op_reply(10 exist [write 0~5] v17'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(10 exist [write 0~5] v17'3 uv3 ondisk = 0) 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 10 ondisk uv 3 in 8.1 attempt 0 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 10 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 15 client.4600.objecter _finish_op 10 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 10 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.858 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a2036640 20 librados: complete_aio_write 0x562cab5ce330 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (3 ms) 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.841+0000 7fc3a3d16880 20 librados: queue_aio_write 0x562cab5e92f0 completion 0x562cab732d60 write_seq 3 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab818800 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [write 0~5 in=5b] tid 11 osd.1 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 11 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 15 client.4600.objecter _send_op 11 to 8.1 on osd.1 2024-04-19T00:45:27.859 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:11 8.1 8:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x562cab818cb0 con 0x562cab7f1f30 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 11 ==== osd_op_reply(11 exist [write 0~5] v17'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(11 exist [write 0~5] v17'4 uv4 ondisk = 0) 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 11 ondisk uv 4 in 8.1 attempt 0 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 11 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 15 client.4600.objecter _finish_op 11 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 11 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a2036640 20 librados: complete_aio_write 0x562cab732d60 2024-04-19T00:45:27.860 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f4fa0 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [read 0~0] tid 12 osd.1 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 12 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 15 client.4600.objecter _send_op 12 to 8.1 on osd.1 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:12 8.1 8:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f5450 con 0x562cab7f1f30 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.861 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f57f0 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid noexist '@8' '@8' [read 0~0] tid 13 osd.1 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 13 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 15 client.4600.objecter _send_op 13 to 8.1 on osd.1 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:13 8.1 8:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab819100 con 0x562cab7f1f30 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.845+0000 7fc3a3d16880 5 client.4600.objecter 2 in flight 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.862 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 12 ondisk uv 4 in 8.1 attempt 0 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 5 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 12 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _finish_op 12 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 12 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 13 ondisk uv 0 in 8.1 attempt 0 2024-04-19T00:45:27.863 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 13 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _finish_op 13 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 13 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f50b0 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.864 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [read 0~0] tid 14 osd.1 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 14 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _send_op 14 to 8.1 on osd.1 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:14 8.1 8:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f5560 con 0x562cab7f1f30 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f5ae0 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid noexist '@8' '@8' [read 0~0] tid 15 osd.1 2024-04-19T00:45:27.865 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 15 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _send_op 15 to 8.1 on osd.1 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:15 8.1 8:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab819100 con 0x562cab7f1f30 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 5 client.4600.objecter 2 in flight 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 14 ondisk uv 4 in 8.1 attempt 0 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 5 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 14 2024-04-19T00:45:27.866 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _finish_op 14 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 14 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 15 ondisk uv 0 in 8.1 attempt 0 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 15 2024-04-19T00:45:27.867 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _finish_op 15 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 15 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f5ae0 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [read 0~0] tid 16 osd.1 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 16 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _send_op 16 to 8.1 on osd.1 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:16 8.1 8:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab818190 con 0x562cab7f1f30 2024-04-19T00:45:27.868 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab8185f0 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base noexist @8 precalc_pgid 0 pgid 0.0 is_read 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target noexist @8 -> pgid 8.a7bbaa1 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.a7bbaa1 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid noexist '@8' '@8' [read 0~0] tid 17 osd.1 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 17 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _send_op 17 to 8.1 on osd.1 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:17 8.1 8:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x562cab818aa0 con 0x562cab7f1f30 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.869 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 5 client.4600.objecter 2 in flight 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 16 ondisk uv 4 in 8.1 attempt 0 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 5 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 16 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _finish_op 16 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 5 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 16 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-04-19T00:45:27.870 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.871 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 17 ondisk uv 0 in 8.1 attempt 0 2024-04-19T00:45:27.871 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 17 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _finish_op 17 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 17 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 librados: queue_aio_write 0x562cab5e92f0 completion 0x562cab732ed0 write_seq 4 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f5ae0 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.872 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 18 osd.1 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 18 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 15 client.4600.objecter _send_op 18 to 8.1 on osd.1 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:18 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f50b0 con 0x562cab7f1f30 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.849+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 18 ==== osd_op_reply(18 exist [writefull 0~5] v17'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(18 exist [writefull 0~5] v17'5 uv5 ondisk = 0) 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 18 ondisk uv 5 in 8.1 attempt 0 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 18 2024-04-19T00:45:27.873 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 15 client.4600.objecter _finish_op 18 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 18 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 20 librados: complete_aio_write 0x562cab732ed0 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (3 ms) 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 librados: queue_aio_write 0x562cab5e92f0 completion 0x562cab7f4fa0 write_seq 5 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f5ae0 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.874 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 19 osd.1 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 19 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 15 client.4600.objecter _send_op 19 to 8.1 on osd.1 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:19 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f50b0 con 0x562cab7f1f30 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 19 ==== osd_op_reply(19 exist [writefull 0~5] v17'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(19 exist [writefull 0~5] v17'6 uv6 ondisk = 0) 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 19 ondisk uv 6 in 8.1 attempt 0 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 19 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 15 client.4600.objecter _finish_op 19 2024-04-19T00:45:27.875 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 19 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a2036640 20 librados: complete_aio_write 0x562cab7f4fa0 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 librados: queue_aio_write 0x562cab5e92f0 completion 0x562cab7f5830 write_seq 6 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 10 client.4600.objecter _op_submit op 0x562cab7f5ae0 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter _calc_target epoch 17 base exist @8 precalc_pgid 0 pgid 0.0 is_write 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter _calc_target target exist @8 -> pgid 8.d3a59201 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 10 client.4600.objecter _calc_target raw pgid 8.d3a59201 -> actual 8.1 acting [1,0] primary 1 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter _get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 10 client.4600.objecter _op_submit oid exist '@8' '@8' [writefull 0~5 in=5b] tid 20 osd.1 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter get_session s=0x562cab72cee0 osd=1 3 2024-04-19T00:45:27.876 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 15 client.4600.objecter _session_op_assign 1 20 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 15 client.4600.objecter _send_op 20 to 8.1 on osd.1 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- osd_op(unknown.0.0:20 8.1 8:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x562cab7f50b0 con 0x562cab7f1f30 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.853+0000 7fc3a3d16880 5 client.4600.objecter 1 in flight 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 <== osd.1 v2:172.21.15.43:6800/1380723486 20 ==== osd_op_reply(20 exist [writefull 0~5] v17'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc39c016030 con 0x562cab7f1f30 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 10 client.4600.objecter ms_dispatch 0x562cab600170 osd_op_reply(20 exist [writefull 0~5] v17'7 uv7 ondisk = 0) 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 10 client.4600.objecter in handle_osd_op_reply 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 7 client.4600.objecter handle_osd_op_reply 20 ondisk uv 7 in 8.1 attempt 0 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 10 client.4600.objecter op 0 rval 0 len 0 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 15 client.4600.objecter handle_osd_op_reply completed tid 20 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 15 client.4600.objecter _finish_op 20 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 4 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 15 client.4600.objecter _session_op_remove 1 20 2024-04-19T00:45:27.877 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 5 client.4600.objecter 0 in flight 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 20 librados: complete_aio_write 0x562cab7f5830 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (3 ms) 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 10 librados: watch_flush enter 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 10 librados: watch_flush exit 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 10 client.4600.objecter close_session for osd.1 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x562cab7f1f30 msgr2=0x562cab7f4320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x562cab7f1f30 0x562cab7f4320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 20 client.4600.objecter put_session s=0x562cab72cee0 osd=1 1 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 20 client.4600.objecter shutdown clearing up homeless session... 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 10 client.4600.objecter successfully canceled tick 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc384036e70 msgr2=0x7fc384039330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc384036e70 0x7fc384039330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc39400a5f0 tx=0x7fc3940a5070 comp rx=0 tx=0).stop 2024-04-19T00:45:27.878 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 10 monclient: shutdown 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cab691d60 msgr2=0x562cab7eabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 --2- 172.21.15.161:0/4143917193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x562cab691d60 0x562cab7eabc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc398006700 tx=0x7fc398006bc0 comp rx=0 tx=0).stop 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a2036640 1 -- 172.21.15.161:0/4143917193 reap_dead start 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 shutdown_connections 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 >> 172.21.15.161:0/4143917193 conn(0x562cab6a7ff0 msgr2=0x562cab72b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 shutdown_connections 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 -- 172.21.15.161:0/4143917193 wait complete. 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.857+0000 7fc3a3d16880 1 librados: shutdown 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [----------] 12 tests from AsioRados (23 ms total) 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [----------] Global test environment tear-down 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1519 ms total) 2024-04-19T00:45:27.879 INFO:tasks.workunit.client.0.smithi161.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-19T00:45:28.216 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-04-19T00:45:28.216 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] Global test environment set-up. 2024-04-19T00:45:28.216 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-19T00:45:28.216 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.ListObjects (251 ms) 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (25 ms) 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset foo1,foo2,foo3 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo1 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo2 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo3 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset foo1,foo4,foo5 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo4 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo5 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo1 2024-04-19T00:45:28.217 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset foo6,foo7 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo7 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo6 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns1:foo4 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns1:foo5 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns2:foo7 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns2:foo6 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns1:foo1 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: :foo1 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: :foo2 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: :foo3 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (86 ms) 2024-04-19T00:45:28.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 1 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 10 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 13 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 7 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 14 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 0 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 15 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 11 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 5 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 8 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 6 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 3 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 4 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 12 0 2024-04-19T00:45:28.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 9 0 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 2 0 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (102 ms) 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: x cursor=MIN 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=1 cursor=9:02547ec2:::1:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=10 cursor=9:52ea6a34:::10:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=13 cursor=9:566253c9:::13:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=7 cursor=9:5c6b0b28:::7:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=14 cursor=9:62a1935d:::14:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=0 cursor=9:6cac518f:::0:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=15 cursor=9:863748b0:::15:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=11 cursor=9:89d3ae78:::11:head 2024-04-19T00:45:28.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=5 cursor=9:b29083e3:::5:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=8 cursor=9:bd63b0f1:::8:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=6 cursor=9:c4fdafeb:::6:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=3 cursor=9:cfc208b3:::3:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=4 cursor=9:d83876eb:::4:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=12 cursor=9:de5d7c5f:::12:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=9 cursor=9:e960b815:::9:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > oid=2 cursor=9:f905c69b:::2:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=1 cursor=9:02547ec2:::1:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=10 cursor=9:52ea6a34:::10:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-04-19T00:45:28.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-04-19T00:45:28.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=13 cursor=9:566253c9:::13:head 2024-04-19T00:45:28.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:566253c9:::13:head 2024-04-19T00:45:28.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-04-19T00:45:28.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=7 cursor=9:5c6b0b28:::7:head 2024-04-19T00:45:28.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-04-19T00:45:28.223 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-04-19T00:45:28.223 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=14 cursor=9:62a1935d:::14:head 2024-04-19T00:45:28.304 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-04-19T00:45:28.304 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-04-19T00:45:28.304 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=0 cursor=9:6cac518f:::0:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=15 cursor=9:863748b0:::15:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:863748b0:::15:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=11 cursor=9:89d3ae78:::11:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=5 cursor=9:b29083e3:::5:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=8 cursor=9:bd63b0f1:::8:head 2024-04-19T00:45:28.305 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=6 cursor=9:c4fdafeb:::6:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=3 cursor=9:cfc208b3:::3:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=4 cursor=9:d83876eb:::4:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=12 cursor=9:de5d7c5f:::12:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-04-19T00:45:28.306 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=9 cursor=9:e960b815:::9:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:e960b815:::9:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : oid=2 cursor=9:f905c69b:::2:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:52ea6a34:::10:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:52ea6a34:::10:head expected=9:52ea6a34:::10:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:52ea6a34:::10:head -> 10 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=10 expected=10 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:f905c69b:::2:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:f905c69b:::2:head expected=9:f905c69b:::2:head 2024-04-19T00:45:28.307 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:f905c69b:::2:head -> 2 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=2 expected=2 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:de5d7c5f:::12:head 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:de5d7c5f:::12:head expected=9:de5d7c5f:::12:head 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:de5d7c5f:::12:head -> 12 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=12 expected=12 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:d83876eb:::4:head 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:d83876eb:::4:head expected=9:d83876eb:::4:head 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:d83876eb:::4:head -> 4 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=4 expected=4 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:cfc208b3:::3:head 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:cfc208b3:::3:head expected=9:cfc208b3:::3:head 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:cfc208b3:::3:head -> 3 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=3 expected=3 2024-04-19T00:45:28.308 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:c4fdafeb:::6:head 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:c4fdafeb:::6:head expected=9:c4fdafeb:::6:head 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:c4fdafeb:::6:head -> 6 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=6 expected=6 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:bd63b0f1:::8:head 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:bd63b0f1:::8:head expected=9:bd63b0f1:::8:head 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:bd63b0f1:::8:head -> 8 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=8 expected=8 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:b29083e3:::5:head 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:b29083e3:::5:head expected=9:b29083e3:::5:head 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:b29083e3:::5:head -> 5 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=5 expected=5 2024-04-19T00:45:28.309 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:89d3ae78:::11:head 2024-04-19T00:45:29.673 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: api_c_read_operations: Running main() from gmock_main.cc 2024-04-19T00:45:29.674 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-19T00:45:29.674 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-19T00:45:29.674 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-19T00:45:29.674 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-19T00:45:29.674 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (257 ms) 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (17 ms) 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (9 ms) 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-19T00:45:29.675 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (40 ms) 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (11 ms) 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2024-04-19T00:45:29.676 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (5 ms) 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-19T00:45:29.677 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (10 ms) 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (12 ms) 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (20 ms) 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (21 ms) 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-19T00:45:29.678 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (14 ms) 2024-04-19T00:45:29.679 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (444 ms total) 2024-04-19T00:45:29.679 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: 2024-04-19T00:45:29.679 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-19T00:45:29.679 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3186 ms total) 2024-04-19T00:45:29.679 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-19T00:45:29.680 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: Running main() from gmock_main.cc 2024-04-19T00:45:29.680 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-19T00:45:29.680 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [----------] Global test environment set-up. 2024-04-19T00:45:29.680 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (3077 ms) 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [----------] 1 test from neocls_handler_error (3077 ms total) 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [----------] Global test environment tear-down 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [==========] 1 test from 1 test suite ran. (3077 ms total) 2024-04-19T00:45:29.681 INFO:tasks.workunit.client.0.smithi161.stdout: handler_error: [ PASSED ] 1 test. 2024-04-19T00:45:29.682 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-19T00:45:29.682 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-19T00:45:29.682 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-19T00:45:29.682 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-19T00:45:29.682 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (3154 ms) 2024-04-19T00:45:29.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (3154 ms total) 2024-04-19T00:45:29.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: 2024-04-19T00:45:29.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-19T00:45:29.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (3155 ms total) 2024-04-19T00:45:29.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-19T00:45:29.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-04-19T00:45:29.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-19T00:45:29.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-19T00:45:29.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-19T00:45:29.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-19T00:45:29.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (55 ms) 2024-04-19T00:45:29.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-19T00:45:29.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (85 ms) 2024-04-19T00:45:29.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-19T00:45:29.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (3106 ms) 2024-04-19T00:45:29.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3246 ms total) 2024-04-19T00:45:29.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: 2024-04-19T00:45:29.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-19T00:45:29.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3246 ms total) 2024-04-19T00:45:29.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (58 ms) 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (74 ms) 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (3116 ms) 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-19T00:45:29.776 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.615479+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi161-29055-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi161-29055-1"}]': finished 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.615730+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29093-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.616111+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29085-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.616370+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi161-29241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.616639+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29252-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.657720+0000 mon.i [INF] from='client.? 172.21.15.161:0/3470186504' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28857-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.660447+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28857-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.734397+0000 mon.f [INF] from='client.? 172.21.15.161:0/2500183297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.737858+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.741246+0000 mon.f [INF] from='client.? 172.21.15.161:0/2500183297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.743225+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.748723+0000 mon.f [INF] from='client.? 172.21.15.161:0/2500183297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.749210+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:29.777 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.807384+0000 mon.i [INF] from='client.? 172.21.15.161:0/2675316709' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.815046+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.816643+0000 mon.i [INF] from='client.? 172.21.15.161:0/2675316709' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.817024+0000 mon.c [INF] from='client.? 172.21.15.161:0/2535289777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.836175+0000 mon.b [INF] from='client.? 172.21.15.161:0/1619523334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.837944+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.839437+0000 mon.c [INF] from='client.? 172.21.15.161:0/2535289777' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.840940+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.843221+0000 mon.i [INF] from='client.? 172.21.15.161:0/2675316709' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.894926+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.895669+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.896260+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:29.778 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.896571+0000 mon.b [INF] from='client.? 172.21.15.161:0/1619523334' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:29.779 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.897355+0000 mon.c [INF] from='client.? 172.21.15.161:0/2535289777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi161-28733-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:29.779 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.898140+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi161-28733-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:29.779 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.899254+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:29.976 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04- list_parallel: process_1_[29269]: starting. 2024-04-19T00:45:29.976 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_1_[29269]: creating pool ceph_test_rados_list_parallel.smithi161-29257 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_1_[29269]: created object 0... 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_1_[29269]: created object 25... 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_1_[29269]: created object 49... 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_1_[29269]: finishing. 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_1_[29269]: shutting down. 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_2_[29270]: starting. 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_2_[29270]: listing objects. 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_2_[29270]: listed object 0... 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_2_[29270]: listed object 25... 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_2_[29270]: saw 50 objects 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_2_[29270]: shutting down. 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: starting. 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: creating pool ceph_test_rados_list_parallel.smithi161-29257 2024-04-19T00:45:29.977 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: created object 0... 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: created object 25... 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: created object 49... 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: finishing. 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_3_[30053]: shutting down. 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_4_[30054]: starting. 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_4_[30054]: listing objects. 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_4_[30054]: listed object 0... 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_4_[30054]: listed object 25... 2024-04-19T00:45:29.978 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_4_[30054]: saw 43 objects 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_4_[30054]: shutting down. 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_5_[30055]: starting. 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_5_[30055]: removed 25 objects... 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_5_[30055]: removed half of the objects 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_5_[30055]: removed 50 objects... 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_5_[30055]: removed 50 objects 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_5_[30055]: shutting down. 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.979 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: starting. 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: creating pool ceph_test_rados_list_parallel.smithi161-29257 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: created object 0... 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: created object 25... 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: created object 49... 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: finishing. 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_6_[30104]: shutting down. 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: starting. 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: listing objects. 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: listed object 0... 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: listed object 25... 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: listed object 50... 2024-04-19T00:45:29.980 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: saw 56 objects 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_7_[30105]: shutting down. 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_8_[30106]: starting. 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_8_[30106]: added 25 objects... 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_8_[30106]: added half of the objects 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_8_[30106]: added 50 objects... 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_8_[30106]: added 50 objects 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_8_[30106]: shutting down. 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:29.981 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.299 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.299 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: starting. 2024-04-19T00:45:30.299 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: creating pool ceph_test_rados_list_parallel.smithi161-29257 2024-04-19T00:45:30.299 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: created object 0... 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: created object 25... 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: created object 49... 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: finishing. 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_9_[30155]: shutting down. 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.300 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: starting. 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: listing objects. 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: listed object 0... 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: listed object 25... 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: listed object 50... 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: listed object 75... 2024-04-19T00:45:30.301 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: saw 98 objects 2024-04-19T00:45:30.302 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_10_[30156]: shutting down. 2024-04-19T00:45:30.302 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.302 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.302 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.302 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_13_[30159]: starting. 2024-04-19T00:45:30.302 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_13_[30159]: removed 25 objects... 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_13_[30159]: removed half of the objects 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_13_[30159]: removed 50 objects... 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_13_[30159]: removed 50 objects 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_13_[30159]: shutting down. 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.303 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_12_[30158]: starting. 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_12_[30158]: added 25 objects... 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_12_[30158]: added half of the objects 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_12_[30158]: added 50 objects... 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_12_[30158]: added 50 objects 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_12_[30158]: shutting down. 2024-04-19T00:45:30.304 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.305 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.305 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.305 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_11_[30157]: starting. 2024-04-19T00:45:30.305 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_11_[30157]: added 25 objects... 2024-04-19T00:45:30.306 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_11_[30157]: added half of the objects 2024-04-19T00:45:30.306 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_11_[30157]: added 50 objects... 2024-04-19T00:45:30.306 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_11_[30157]: added 50 objects 2024-04-19T00:45:30.306 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_11_[30157]: shutting down. 2024-04-19T00:45:30.306 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: starting. 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: creating pool ceph_test_rados_list_parallel.smithi161-29257 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: created object 0... 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: created object 25... 2024-04-19T00:45:30.307 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: created object 49... 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: finishing. 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_14_[30419]: shutting down. 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: starting. 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listing objects. 2024-04-19T00:45:30.308 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listed object 0... 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listed object 25... 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listed object 50... 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listed object 75... 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listed object 100... 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: listed object 125... 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: saw 150 objects 2024-04-19T00:45:30.692 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_15_[30420]: shutting down. 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_16_[30421]: starting. 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_16_[30421]: added 25 objects... 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_16_[30421]: added half of the objects 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_16_[30421]: added 50 objects... 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_16_[30421]: added 50 objects 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: process_16_[30421]: shutting down. 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.693 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******************************* 2024-04-19T00:45:30.694 INFO:tasks.workunit.client.0.smithi161.stdout: list_parallel: ******* SUCCESS ********** 2024-04-19T00:45:30.703 INFO:tasks.workunit.client.0.smithi161.stdout:19T00:45:28.900207+0000 mon.b [INF] from='client.? 172.21.15.161:0/1619523334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi161-28974-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:30.703 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:28.901040+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi161-28974-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:30.703 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.634324+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi161-29622-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi161-29622-1"}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.634624+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi161-29557-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi161-29557-1"}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.634859+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28857-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.635158+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.635411+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.635601+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi161-28733-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.635754+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi161-28974-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.651700+0000 mon.c [INF] from='client.? 172.21.15.161:0/2535289777' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi161-28733-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.652151+0000 mon.b [INF] from='client.? 172.21.15.161:0/1619523334' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi161-28974-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.657555+0000 mon.i [INF] from='client.? 172.21.15.161:0/2675316709' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi161-28741-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:30.704 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.658630+0000 mon.c [INF] from='client.? 172.21.15.161:0/2224501540' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28879-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:32.730 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.661410+0000 mon.b [INF] from='client.? 172.21.15.161:0/2741865878' entity='client.admin' cmd=[{"pr cls: Running main() from gmock_main.cc 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [----------] Global test environment set-up. 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [ OK ] NeoRadosCls.DNE (3113 ms) 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (3054 ms) 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [----------] 2 tests from NeoRadosCls (6167 ms total) 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [----------] Global test environment tear-down 2024-04-19T00:45:32.731 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [==========] 2 tests from 1 test suite ran. (6167 ms total) 2024-04-19T00:45:32.732 INFO:tasks.workunit.client.0.smithi161.stdout: cls: [ PASSED ] 2 tests. 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: starting. 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: creating pool ceph_test_rados_open_pools_parallel.smithi161-29265 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: created object 0... 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: created object 25... 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: created object 49... 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: finishing. 2024-04-19T00:45:32.745 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_1_[29276]: shutting down. 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_2_[29277]: starting. 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_2_[29277]: rados_pool_create. 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_2_[29277]: rados_ioctx_create. 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_2_[29277]: shutting down. 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: ******************************* 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: starting. 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: creating pool ceph_test_rados_open_pools_parallel.smithi161-29265 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: created object 0... 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: created object 25... 2024-04-19T00:45:32.746 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: created object 49... 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: finishing. 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_3_[30229]: shutting down. 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: ******************************* 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_4_[30230]: starting. 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_4_[30230]: rados_pool_create. 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_4_[30230]: rados_ioctx_create. 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: process_4_[30230]: shutting down. 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: ******************************* 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: ******************************* 2024-04-19T00:45:32.747 INFO:tasks.workunit.client.0.smithi161.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: starting. 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: creating pool ceph_test_rados_delete_pools_parallel.smithi161-29288 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: created object 0... 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: created object 25... 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: created object 49... 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: finishing. 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_1_[29350]: shutting down. 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_2_[29352]: starting. 2024-04-19T00:45:32.748 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_2_[29352]: deleting pool ceph_test_rados_delete_pools_parallel.smithi161-29288 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_2_[29352]: shutting down. 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: ******************************* 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: starting. 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: creating pool ceph_test_rados_delete_pools_parallel.smithi161-29288 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: created object 0... 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: created object 25... 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: created object 49... 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: finishing. 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_3_[30231]: shutting down. 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: ******************************* 2024-04-19T00:45:32.749 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_5_[30233]: starting. 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_5_[30233]: listing objects. 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_5_[30233]: listed object 0... 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_5_[30233]: listed object 25... 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_5_[30233]: saw 50 objects 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_5_[30233]: shutting down. 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: ******************************* 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_4_[30232]: starting. 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_4_[30232]: deleting pool ceph_test_rados_delete_pools_parallel.smithi161-29288 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: process_4_[30232]: shutting down. 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: ******************************* 2024-04-19T00:45:32.750 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: ******************************* 2024-04-19T00:45:32.751 INFO:tasks.workunit.client.0.smithi161.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-19T00:45:32.914 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: Running main() from gmock_main.cc 2024-04-19T00:45:32.914 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [==========] Running 38 tests from 2 test suites. 2024-04-19T00:45:32.914 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: seed 28741 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (201 ms) 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (26 ms) 2024-04-19T00:45:32.915 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (503 ms) 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (3 ms) 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5 ms) 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-04-19T00:45:32.916 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (6 ms) 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (7 ms) 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-04-19T00:45:32.917 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (14 ms) 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5 ms) 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-19T00:45:32.918 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (9 ms) 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (27 ms) 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (8 ms) 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (8 ms) 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (3 ms) 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (6 ms) 2024-04-19T00:45:32.919 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP (853 ms total) 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1193 ms) 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (25 ms) 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-19T00:45:32.920 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-04-19T00:45:32.921 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-19T00:45:32.921 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-04-19T00:45:33.708 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: Running main() from gmock_main.cc 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [----------] Global test environment set-up. 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (2049 ms) 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2030 ms) 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3018 ms) 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [----------] 3 tests from NeoRadosCmd (7097 ms total) 2024-04-19T00:45:33.709 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: 2024-04-19T00:45:33.710 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [----------] Global test environment tear-down 2024-04-19T00:45:33.710 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [==========] 3 tests from 1 test suite ran. (7097 ms total) 2024-04-19T00:45:33.710 INFO:tasks.workunit.client.0.smithi161.stdout: cmd: [ PASSED ] 3 tests. 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout:efix": "osd pool application enable","pool": "SimpleWrite_smithi161-28717-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.662526+0000 mon.c [INF] from='client.? 172.21.15.161:0/2782106932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29225-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.664913+0000 mon.f [INF] from='client.? 172.21.15.161:0/2500183297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi161-29028-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.670390+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi161-28974-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.671113+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi161-28733-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.671672+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi161-28741-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.672291+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi161-28717-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.672649+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28879-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.677795+0000 mon.i [INF] from='client.? 172.21.15.161:0/740421580' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi161-28725-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.685061+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29225-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.685345+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi161-28725-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.685696+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi161-29028-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.774190+0000 mon.c [INF] from='client.? 172.21.15.161:0/1474493328' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.774398+0000 client.admin [INF] onexx 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:29.817827+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:31.732195+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.128744+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.0"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.129452+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.1"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.130115+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.2"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.130725+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.3"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.131293+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.4"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.131861+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.5"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.132421+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.6"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.132928+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.7"}]: dispatch 2024-04-19T00:45:33.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.133386+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.8"}]: dispatch 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.133909+0000 mon.i [INF] from='client.? 172.21.15.161:0/830043360' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "12.9"}]: dispatch 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.670275+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi161-29557-1"}]': finished 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.670453+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi161-29055-1"}]': finished 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.670663+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi161-29622-1"}]': finished 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.670843+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi161-28857-4", "tierpool": "test-rados-api-smithi161-28857-4-cache"}]': finished 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.671064+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi161-28725-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.682158+0000 mon.b [INF] from='client.? 172.21.15.161:0/2154768207' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi161-29622-1"}]: dispatch 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.692132+0000 mon.i [INF] from='client.? 172.21.15.161:0/3470186504' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi161-28857-4", "overlaypool": "test-rados-api-smithi161-28857-4-cache"}]: dispatch 2024-04-19T00:45:33.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.707369+0000 mon.g [INF] from='client.? 172.21.15.161:0/879997174' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28879-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.826 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.708707+0000 mon.b [INF] from='client.? 172.21.15.161:0/2671289021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi161-28717-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.826 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.709100+0000 mon.b [INF] from='client.? 172.21.15.161:0/1503042565' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29225-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.826 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.712833+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi161-29622-1"}]: dispatch 2024-04-19T00:45:33.826 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.713828+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28879-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.826 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.714530+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi161-28857-4", "overlaypool": "test-rados-api-smithi161-28857-4-cache"}]: dispatch 2024-04-19T00:45:33.826 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.715557+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi161-28717-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.716817+0000 mon.i [INF] from='client.? 172.21.15.161:0/740421580' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi161-28725-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.726685+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29225-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.735393+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi161-28725-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.825765+0000 mon.c [INF] from='client.? 172.21.15.161:0/3324398573' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi161-29557-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:32.863973+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi161-29557-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.680315+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi161-28785-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi161-28785-10"}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.680659+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi161-28762-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi161-28762-10"}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.680933+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi161-29622-1"}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.681144+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-28879-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.681370+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi161-28857-4", "overlaypool": "test-rados-api-smithi161-28857-4-cache"}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.681543+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi161-28717-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.681774+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi161-29225-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-19T00:45:33.827 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.681977+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi161-28725-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.682189+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi161-29557-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.692174+0000 mon.c [INF] from='client.? 172.21.15.161:0/3324398573' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi161-29557-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi161-29557-2"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.694398+0000 mon.i [INF] from='client.? 172.21.15.161:0/3470186504' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi161-28857-4-cache", "mode": "writeback"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.707036+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi161-29557-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi161-29557-2"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.707167+0000 mon.f [INF] from='client.? 172.21.15.161:0/2500183297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.708420+0000 mon.c [INF] from='client.? 172.21.15.161:0/2535289777' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.708500+0000 mon.b [INF] from='client.? 172.21.15.161:0/1619523334' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.708975+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi161-28857-4-cache", "mode": "writeback"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.714534+0000 mon.b [INF] from='client.? 172.21.15.161:0/694744327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi161-29055-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.714553+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi161-29028-7"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.715837+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi161-28733-16"}]: dispatch 2024-04-19T00:45:33.828 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.720731+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi161-28974-7"}]: dispatch 2024-04-19T00:45:35.735 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.721077+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool appl api_stat_pp: Running main() from gmock_main.cc 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: seed 29028 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (208 ms) 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (5 ms) 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (7 ms) 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-19T00:45:35.736 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (219 ms) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (439 ms total) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1181 ms) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (2 ms) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (16 ms) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (22 ms) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1221 ms total) 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: 2024-04-19T00:45:35.737 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-19T00:45:35.738 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (9289 ms total) 2024-04-19T00:45:35.738 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-19T00:45:35.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-19T00:45:35.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-04-19T00:45:35.747 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (2 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (9 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (7 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-19T00:45:35.748 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (11 ms) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (5 ms) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1294 ms total) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [==========] 38 tests from 2 test suites ran. (9439 ms total) 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_io_pp: [ PASSED ] 38 tests. 2024-04-19T00:45:35.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: Running main() from gmock_main.cc 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [----------] Global test environment set-up. 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStat.Stat (186 ms) 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (21 ms) 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStat.StatNS (217 ms) 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-19T00:45:35.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [----------] 5 tests from LibRadosStat (427 ms total) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1155 ms) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (48 ms) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-04-19T00:45:35.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1207 ms total) 2024-04-19T00:45:35.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: 2024-04-19T00:45:35.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [----------] Global test environment tear-down 2024-04-19T00:45:35.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (9326 ms total) 2024-04-19T00:45:35.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: Running main() from gmock_main.cc 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [----------] Global test environment set-up. 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (192 ms) 2024-04-19T00:45:35.762 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: no timeout :/ 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: no timeout :/ 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: no timeout :/ 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: no timeout :/ 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: no timeout :/ 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (55 ms) 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5 ms) 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.Checksum (3 ms) 2024-04-19T00:45:35.763 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (7 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-04-19T00:45:35.764 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.RmXattr (32 ms) 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIo.XattrIter (4 ms) 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [----------] 14 tests from LibRadosIo (321 ms total) 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1179 ms) 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (27 ms) 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-19T00:45:35.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (6 ms) 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (10 ms) 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (5 ms) 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-19T00:45:35.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (11 ms) 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1257 ms total) 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [----------] Global test environment tear-down 2024-04-19T00:45:35.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9469 ms total) 2024-04-19T00:45:35.768 INFO:tasks.workunit.client.0.smithi161.stdout: api_io: [ PASSED ] 24 tests. 2024-04-19T00:45:36.549 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: handle_notify cookie 94318822106272 notify_id 77309411328 notifier_gid 4729 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2957 ms) 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: Trying... 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: handle_notify cookie 94318834541344 notify_id 90194313220 notifier_gid 4895 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: Waiting for 3.000000000s 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: Timed out. 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: Flushing... 2024-04-19T00:45:36.550 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: Flushed... 2024-04-19T00:45:36.551 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (6884 ms) 2024-04-19T00:45:36.551 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9841 ms total) 2024-04-19T00:45:36.551 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: 2024-04-19T00:45:36.551 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-19T00:45:36.551 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9841 ms total) 2024-04-19T00:45:36.551 INFO:tasks.workunit.client.0.smithi161.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-19T00:45:36.571 INFO:tasks.workunit.client.0.smithi161.stdout: list: Running main() from gmock_main.cc 2024-04-19T00:45:36.571 INFO:tasks.workunit.client.0.smithi161.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-19T00:45:36.571 INFO:tasks.workunit.client.0.smithi161.stdout: list: [----------] Global test environment set-up. 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [----------] 3 tests from NeoradosList 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ OK ] NeoradosList.ListObjects (3001 ms) 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3045 ms) 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3837 ms) 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [----------] 3 tests from NeoradosList (9883 ms total) 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [----------] Global test environment tear-down 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [==========] 3 tests from 1 test suite ran. (9883 ms total) 2024-04-19T00:45:36.572 INFO:tasks.workunit.client.0.smithi161.stdout: list: [ PASSED ] 3 tests. 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout:ication enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi161-29055-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.722035+0000 mon.i [INF] from='client.? 172.21.15.161:0/2675316709' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.725316+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi161-28741-23"}]: dispatch 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.748243+0000 mon.c [INF] from='client.? 172.21.15.161:0/1474493328' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.748881+0000 client.admin [INF] threexx 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: got: 2024-04-19T00:45:33.750849+0000 mon.e [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7154 ms) 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10402 ms total) 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: 2024-04-19T00:45:36.838 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-19T00:45:36.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10403 ms total) 2024-04-19T00:45:36.839 INFO:tasks.workunit.client.0.smithi161.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify 2024-04-19T00:45:38.749 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1236 ms) 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (9 ms) 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1245 ms total) 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (144 ms) 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3817 ms) 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-19T00:45:38.750 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (8 ms) 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328314503264 notify_id 120259084290 notifier_gid 5006 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (6 ms) 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328314506320 notify_id 120259084291 notifier_gid 5006 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328314506320 notify_id 120259084289 notifier_gid 5006 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328314506320 notify_id 120259084290 notifier_gid 5006 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (8 ms) 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 120259084292 notifier_gid 5006 2024-04-19T00:45:38.751 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (6 ms) 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 120259084293 notifier_gid 5006 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 120259084294 notifier_gid 5006 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-19T00:45:38.752 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: trying... 2024-04-19T00:45:39.566 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 120259084291 notifier_g ec_io: Running main() from gmock_main.cc 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [----------] Global test environment set-up. 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (6029 ms) 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6869 ms) 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12898 ms total) 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [----------] Global test environment tear-down 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12899 ms total) 2024-04-19T00:45:39.567 INFO:tasks.workunit.client.0.smithi161.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-19T00:45:39.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-19T00:45:39.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-19T00:45:39.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: seed 28785 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (184 ms) 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (15 ms) 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-19T00:45:39.584 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1128 ms) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1016 ms) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (10 ms) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (11 ms) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (11 ms) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4 ms) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2379 ms total) 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: 2024-04-19T00:45:39.585 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1147 ms) 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (27 ms) 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1073 ms) 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1010 ms) 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (13 ms) 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (14 ms) 2024-04-19T00:45:39.586 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (17 ms) 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (11 ms) 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3312 ms total) 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13228 ms total) 2024-04-19T00:45:39.587 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-19T00:45:39.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: Running main() from gmock_main.cc 2024-04-19T00:45:39.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [----------] Global test environment set-up. 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (183 ms) 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.LockShared (5 ms) 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1117 ms) 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1012 ms) 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5 ms) 2024-04-19T00:45:39.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.Unlock (8 ms) 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (15 ms) 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [----------] 8 tests from LibRadosLock (2348 ms total) 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1164 ms) 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6 ms) 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-19T00:45:39.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1059 ms) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1010 ms) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (14 ms) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (14 ms) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (17 ms) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (11 ms) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3296 ms total) 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: 2024-04-19T00:45:39.599 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [----------] Global test environment tear-down 2024-04-19T00:45:39.600 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13277 ms total) 2024-04-19T00:45:39.600 INFO:tasks.workunit.client.0.smithi161.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-19T00:45:40.579 INFO:tasks.workunit.client.0.smithi161.stdout: pool: Running main() from gmock_main.cc 2024-04-19T00:45:40.579 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-19T00:45:40.579 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [----------] Global test environment set-up. 2024-04-19T00:45:40.579 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ OK ] NeoRadosPools.PoolList (1969 ms) 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2016 ms) 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2069 ms) 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3813 ms) 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2013 ms) 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2014 ms) 2024-04-19T00:45:40.580 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [----------] 6 tests from NeoRadosPools (13894 ms total) 2024-04-19T00:45:40.581 INFO:tasks.workunit.client.0.smithi161.stdout: pool: 2024-04-19T00:45:40.581 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [----------] Global test environment tear-down 2024-04-19T00:45:40.581 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [==========] 6 tests from 1 test suite ran. (13894 ms total) 2024-04-19T00:45:40.581 INFO:tasks.workunit.client.0.smithi161.stdout: pool: [ PASSED ] 6 tests. 2024-04-19T00:45:45.625 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-04-19T00:45:45.625 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-19T00:45:45.625 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify_test_cb 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (229 ms) 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94135578359760 err -107 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1037 ms) 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94135578370672 err -107 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (9 ms) 2024-04-19T00:45:45.626 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 77309411328 cookie 94135578370672 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (25 ms) 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 77309411328 cookie 94135578379424 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (17 ms) 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 77309411329 cookie 94135578304512 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (8 ms) 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 77309411328 cookie 94135578304512 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 77309411328 cookie 94135578337696 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (20 ms) 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-19T00:45:45.627 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 77309411330 cookie 94135578337696 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 81604378627 cookie 94135578337696 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3006 ms) 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94135578304512 err -107 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_cb from 4497 notify_id 111669149698 cookie 94135578304512 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5019 ms) 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94135578304512 err -107 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1017 ms) 2024-04-19T00:45:45.628 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10387 ms total) 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: watch_notify_test_cb 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1121 ms) 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1121 ms total) 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (19206 ms total) 2024-04-19T00:45:45.629 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-19T00:45:49.608 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: Running main() from gmock_main.cc 2024-04-19T00:45:49.608 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-19T00:45:49.608 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [----------] Global test environment set-up. 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (7026 ms) 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6896 ms) 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (9006 ms) 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [----------] 3 tests from NeoradosECList (22928 ms total) 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [----------] Global test environment tear-down 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (22928 ms total) 2024-04-19T00:45:49.609 INFO:tasks.workunit.client.0.smithi161.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-19T00:45:51.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-04-19T00:45:51.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-19T00:45:51.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-19T00:45:51.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-19T00:45:51.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-19T00:45:51.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-19T00:45:51.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-19T00:45:51.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: 2024-04-19T00:45:51.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-19T00:45:51.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-19T00:45:51.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2183 ms) 2024-04-19T00:45:51.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-19T00:45:51.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3023 ms) 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3065 ms) 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2800 ms) 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3035 ms) 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3003 ms) 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-19T00:45:51.597 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (7988 ms) 2024-04-19T00:45:51.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (25097 ms total) 2024-04-19T00:45:51.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: 2024-04-19T00:45:51.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-19T00:45:51.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (25097 ms total) 2024-04-19T00:45:51.598 INFO:tasks.workunit.client.0.smithi161.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-19T00:45:52.586 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: Running main() from gmock_main.cc 2024-04-19T00:45:52.586 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [----------] Global test environment set-up. 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2974 ms) 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3035 ms) 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3017 ms) 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2829 ms) 2024-04-19T00:45:52.587 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-19T00:45:52.588 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2998 ms) 2024-04-19T00:45:52.588 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-19T00:45:52.588 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3032 ms) 2024-04-19T00:45:52.588 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-19T00:45:52.588 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7981 ms) 2024-04-19T00:45:52.588 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (25866 ms total) 2024-04-19T00:45:52.589 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: 2024-04-19T00:45:52.589 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [----------] Global test environment tear-down 2024-04-19T00:45:52.589 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (25866 ms total) 2024-04-19T00:45:52.589 INFO:tasks.workunit.client.0.smithi161.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-19T00:45:54.227 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [----------] Global test environment set-up. 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5072 ms) 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ OK ] LibRadosService.RegisterLate (64 ms) 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: cluster: 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: id: 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: health: HEALTH_WARN 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 14 pool(s) do not have an application enabled 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.228 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: services: 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,d,i,e (age 23s), out of quorum: h 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: mgr: x(active, since 65s) 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: osd: 6 osds: 6 up (since 60s), 6 in (since 67s) 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: data: 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: pools: 27 pools, 785 pgs 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: objects: 18 objects, 577 KiB 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: pgs: 68.535% pgs unknown 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 16.688% pgs not active 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 538 unknown 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 131 creating+peering 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 116 active+clean 2024-04-19T00:45:54.229 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: io: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: client: 0 B/s rd, 3.5 KiB/s wr, 2 op/s rd, 7 op/s wr 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: cluster: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: id: 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: health: HEALTH_WARN 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 13 pool(s) do not have an application enabled 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: services: 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: mon: 9 daemons, quorum a,f,b,g,c,d,i,e (age 25s), out of quorum: h 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: mgr: x(active, since 67s) 2024-04-19T00:45:54.230 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: osd: 6 osds: 6 up (since 62s), 6 in (since 69s) 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: data: 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: pools: 25 pools, 649 pgs 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: objects: 45 objects, 583 KiB 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: usage: 178 MiB used, 540 GiB / 540 GiB avail 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: pgs: 41.602% pgs unknown 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 10.632% pgs not active 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 310 active+clean 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 270 unknown 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 69 creating+peering 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.231 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: io: 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: client: 2.5 KiB/s rd, 13 KiB/s wr, 1 op/s rd, 17 op/s wr 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2566 ms) 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ OK ] LibRadosService.Status (20050 ms) 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [----------] 4 tests from LibRadosService (27752 ms total) 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [----------] Global test environment tear-down 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27753 ms total) 2024-04-19T00:45:54.232 INFO:tasks.workunit.client.0.smithi161.stdout: api_service: [ PASSED ] 4 tests. 2024-04-19T00:45:54.615 INFO:tasks.workunit.client.0.smithi161.stdout:id 5006 2024-04-19T00:45:54.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: timed out 2024-04-19T00:45:54.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: flushing 2024-04-19T00:45:54.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: flushed 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3011 ms) 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: trying... 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 133143986183 notifier_gid 5006 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: timed out 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: flushing 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: flushed 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3009 ms) 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: List watches 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify2 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 146028888073 notifier_gid 5006 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify2 done 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: watch_check 2024-04-19T00:45:54.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: unwatch2 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: flushing 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: done 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3065 ms) 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: List watches 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify2 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: handle_notify cookie 94328313625008 notify_id 158913789956 notifier_gid 5006 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: notify2 done 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: watch_check 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: unwatch2 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: flushing 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: done 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3021 ms) 2024-04-19T00:45:54.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16116 ms total) 2024-04-19T00:45:54.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: 2024-04-19T00:45:54.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-19T00:45:54.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28189 ms total) 2024-04-19T00:45:54.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5082 ms) 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (46 ms) 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-19T00:45:56.849 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20040 ms) 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: attempt 0 of 20 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5184 ms) 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30352 ms total) 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30352 ms total) 2024-04-19T00:45:56.850 INFO:tasks.workunit.client.0.smithi161.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-19T00:45:59.650 INFO:tasks.workunit.client.0.smithi161.stdout: misc: Running main() from gmock_main.cc 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [----------] Global test environment set-up. 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.Version (1976 ms) 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2019 ms) 2024-04-19T00:45:59.651 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.LongName (3030 ms) 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2844 ms) 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3017 ms) 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2995 ms) 2024-04-19T00:45:59.652 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.Exec (3067 ms) 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2930 ms) 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3011 ms) 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3017 ms) 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-19T00:45:59.653 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1989 ms) 2024-04-19T00:45:59.654 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-19T00:45:59.654 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3069 ms) 2024-04-19T00:45:59.654 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [----------] 12 tests from NeoRadosMisc (32965 ms total) 2024-04-19T00:45:59.654 INFO:tasks.workunit.client.0.smithi161.stdout: misc: 2024-04-19T00:45:59.654 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [----------] Global test environment tear-down 2024-04-19T00:45:59.654 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [==========] 12 tests from 1 test suite ran. (32965 ms total) 2024-04-19T00:45:59.655 INFO:tasks.workunit.client.0.smithi161.stdout: misc: [ PASSED ] 12 tests. 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: Running main() from gmock_main.cc 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [----------] Global test environment set-up. 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2315 ms) 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3018 ms) 2024-04-19T00:46:03.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3033 ms) 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2840 ms) 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3041 ms) 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5017 ms) 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4957 ms) 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5050 ms) 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-19T00:46:03.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7952 ms) 2024-04-19T00:46:03.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [----------] 9 tests from LibRadosPools (37224 ms total) 2024-04-19T00:46:03.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: 2024-04-19T00:46:03.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [----------] Global test environment tear-down 2024-04-19T00:46:03.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37224 ms total) 2024-04-19T00:46:03.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-19T00:46:04.691 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] Global test environment set-up. 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: 2024-04-19T00:45:26.433+0000 7fb4eae51880 -1 failed for service _ceph-mon._tcp 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: 2024-04-19T00:45:26.433+0000 7fb4eae51880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (66 ms) 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-19T00:46:04.692 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2075 ms total) 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: started 0x556b4f102790 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: started 0x556b4e4f64b0 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: started 2 aios 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: waiting 0x556b4f102790 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: waiting 0x556b4e4f64b0 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: done. 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5266 ms) 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5266 ms total) 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: 2024-04-19T00:46:04.693 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.Exec (142 ms) 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (6 ms) 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4704 ms) 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-19T00:46:04.694 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (23517 ms) 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28374 ms total) 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [----------] Global test environment tear-down 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (38272 ms total) 2024-04-19T00:46:04.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-19T00:46:08.602 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: Running main() from gmock_main.cc 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [----------] Global test environment set-up. 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2981 ms) 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3039 ms) 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3033 ms) 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2834 ms) 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-19T00:46:08.603 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2983 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3017 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (2970 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3031 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3002 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2980 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3114 ms) 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-19T00:46:08.604 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2878 ms) 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3076 ms) 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2964 ms) 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41902 ms total) 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [----------] Global test environment tear-down 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41902 ms total) 2024-04-19T00:46:08.605 INFO:tasks.workunit.client.0.smithi161.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: Running main() from gmock_main.cc 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [----------] Global test environment set-up. 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.Limits (3053 ms) 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3048 ms) 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.ReadOp (3025 ms) 2024-04-19T00:46:08.609 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.SparseRead (2852 ms) 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2965 ms) 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3031 ms) 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (2948 ms) 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3048 ms) 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3003 ms) 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-19T00:46:08.610 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.Trunc (2981 ms) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.Remove (3091 ms) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2920 ms) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.RmXattr (3055 ms) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2967 ms) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [----------] 14 tests from NeoRadosIo (41987 ms total) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [----------] Global test environment tear-down 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [==========] 14 tests from 1 test suite ran. (41987 ms total) 2024-04-19T00:46:08.611 INFO:tasks.workunit.client.0.smithi161.stdout: io: [ PASSED ] 14 tests. 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: Running main() from gmock_main.cc 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2035 ms) 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2010 ms) 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-19T00:46:22.604 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2020 ms) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2025 ms) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8090 ms total) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4116 ms) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3919 ms) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (6068 ms) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (14103 ms total) 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-19T00:46:22.605 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3033 ms) 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1942 ms) 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2024 ms) 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2070 ms) 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9069 ms total) 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4074 ms) 2024-04-19T00:46:22.606 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-19T00:46:22.607 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4016 ms) 2024-04-19T00:46:22.607 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8090 ms total) 2024-04-19T00:46:22.607 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: 2024-04-19T00:46:22.607 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-19T00:46:22.607 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (56225 ms total) 2024-04-19T00:46:22.607 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-19T00:46:36.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: seed 28939 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2033 ms) 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2025 ms) 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2020 ms) 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-19T00:46:36.620 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2031 ms) 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3834 ms) 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11943 ms total) 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4043 ms) 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3985 ms) 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5985 ms) 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-19T00:46:36.621 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4054 ms) 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2121 ms) 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:463: Skipped 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi161-28939-7 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17933 ms) 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38121 ms total) 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: 2024-04-19T00:46:36.622 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (37 ms) 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5040 ms) 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5077 ms total) 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2971 ms) 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2007 ms) 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-19T00:46:36.623 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2024 ms) 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_ api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: seed 28831 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-19T00:46:38.683 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (4 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (244 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (10 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (6 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (8 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-19T00:46:38.684 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (1 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (12 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (6 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3495 ms) 2024-04-19T00:46:38.685 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (486 ms) 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: done waiting, doing copies 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: done waiting 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60727 ms) 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (9 ms) 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (7 ms) 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-19T00:46:38.686 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5015 ms) 2024-04-19T00:46:38.687 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-19T00:46:38.687 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-19T00:46:38.687 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-19T00:46:38.687 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-19T00:46:38.687 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-19T00:46:38.687 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-19T00:46:40.179 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-04-19T00:46:40.179 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-04-19T00:46:40.179 INFO:tasks.ceph.mon.h:Restarting daemon 2024-04-19T00:46:40.179 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-04-19T00:46:40.183 INFO:tasks.ceph.mon.h:Started 2024-04-19T00:46:40.183 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-04-19T00:46:40.183 INFO:tasks.ceph.mon.i:Sent signal 19 2024-04-19T00:46:40.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:46:40.274 INFO:tasks.ceph.mon.h.smithi161.stderr:2024-04-19T00:46:40.271+0000 7f994b0e4c80 -1 mon.h@-1(???).log v0 log_external_backlog rewinding external_log_to from 22 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP ( snapshots: Running main() from gmock_main.cc 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [----------] Global test environment set-up. 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4983 ms) 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4863 ms) 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4042 ms) 2024-04-19T00:46:45.519 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5012 ms) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (6984 ms) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25884 ms total) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4988 ms) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5997 ms) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8003 ms) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6028 ms) 2024-04-19T00:46:45.520 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3974 ms) 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi161-29680-11. 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: Waiting for snaps to purge. 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (23937 ms) 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (52927 ms total) 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [----------] Global test environment tear-down 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (78811 ms total) 2024-04-19T00:46:45.521 INFO:tasks.workunit.client.0.smithi161.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-19T00:46:55.184 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-04-19T00:46:55.185 INFO:tasks.ceph.mon.i:Sent signal 18 2024-04-19T00:46:55.185 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:46:55.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:46:55.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient: get_monmap_and_config 2024-04-19T00:46:55.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient: build_initial_monmap 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient: monmap: 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:55.331011+0000 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:55.331011+0000 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:55.330 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:55.331 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:55.331 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:55.331 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:55.331 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:55.331 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 1 Processor -- start 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 1 -- start start 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient: init 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient: _add_conns ranks=[10,4,8,7,9,2,3,6,1,5,0] 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f67b4138a50 0x7f67b4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient(hunting): picked mon.noname-f con 0x7f67b4138a50 addr v2:172.21.15.161:3300/0 2024-04-19T00:46:55.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f67b40a5bb0 0x7f67b4139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 10 monclient(hunting): picked mon.noname-c con 0x7f67b40a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:55.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.326+0000 7f67ba302640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 0x7f67b4137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 10 monclient(hunting): picked mon.noname-e con 0x7f67b4137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f67b40a1020 con 0x7f67b4138a50 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f67b41398b0 con 0x7f67b40a5bb0 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f67b40a0780 con 0x7f67b4137960 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67ba302640 10 monclient(hunting): authenticate will time out at 1085.460814s 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 0x7f67b4137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 0x7f67b4137de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54856/0 (socket says 172.21.15.43:54856) 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 -- 172.21.15.43:0/2709493174 learned_addr learned my addr 172.21.15.43:0/2709493174 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): get_auth_request con 0x7f67b4137960 auth_method 0 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f67b40a5bb0 0x7f67b4139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): get_auth_request con 0x7f67b40a5bb0 auth_method 0 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient(hunting): handle_auth_done global_id 22265 payload 274 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 -- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f67b40a5bb0 msgr2=0x7f67b4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f67b40a5bb0 0x7f67b4139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 -- 172.21.15.43:0/2709493174 >> v2:172.21.15.161:3300/0 conn(0x7f67b4138a50 msgr2=0x7f67b4138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 --2- 172.21.15.43:0/2709493174 >> v2:172.21.15.161:3300/0 conn(0x7f67b4138a50 0x7f67b4138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient: _finish_hunting 0 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 monclient: found mon.noname-e 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:46:55.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.330+0000 7f67b889f640 1 -- 172.21.15.43:0/2709493174 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b40a0d40 con 0x7f67b4137960 2024-04-19T00:46:55.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.334+0000 7f67b37fe640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f67b40a5bb0 0x7f67b4139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:55.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.334+0000 7f67b889f640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 0x7f67b4137de0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009e50 tx=0x7f67ac01c900 comp rx=0 tx=0).ready entity=mon.8 client_cookie=44c97220f7a32db2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 1 -- 172.21.15.43:0/2709493174 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67ac0090d0 con 0x7f67b4137960 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: dump: 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.341 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:46:55.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 5 monclient: authenticate success, global_id 22265 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 1 -- 172.21.15.43:0/2709493174 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac0a5bb0 con 0x7f67b4137960 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 4 set_mon_vals no callback set 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 1 -- 172.21.15.43:0/2709493174 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67ac0a5de0 con 0x7f67b4137960 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67b2ffd640 10 monclient: dump: 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.343 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.344 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 10 monclient: shutdown 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 -- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 msgr2=0x7f67b4137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 0x7f67b4137de0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009e50 tx=0x7f67ac01c900 comp rx=0 tx=0).stop 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 -- 172.21.15.43:0/2709493174 shutdown_connections 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f67b4137960 0x7f67b4137de0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 --2- 172.21.15.43:0/2709493174 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f67b40a5bb0 0x7f67b4139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 --2- 172.21.15.43:0/2709493174 >> v2:172.21.15.161:3300/0 conn(0x7f67b4138a50 0x7f67b4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 -- 172.21.15.43:0/2709493174 >> 172.21.15.43:0/2709493174 conn(0x7f67b40a7cb0 msgr2=0x7f67b40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:55.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.338+0000 7f67ba302640 1 -- 172.21.15.43:0/2709493174 shutdown_connections 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.342+0000 7f67ba302640 1 -- 172.21.15.43:0/2709493174 wait complete. 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.342+0000 7f67ba302640 10 monclient: build_initial_monmap 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.342+0000 7f67ba302640 10 monclient: monmap: 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.342+0000 7f67ba302640 1 librados: starting msgr at 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.342+0000 7f67ba302640 1 librados: starting objecter 2024-04-19T00:46:55.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.342+0000 7f67ba302640 1 Processor -- start 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 -- start start 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 librados: setting wanted keys 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 librados: calling monclient init 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient: init 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient: _add_conns ranks=[1,3,6,2,0,7,4,5,8] 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 0x7f67b41dda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): picked mon.noname-f con 0x7f67b40a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:46:55.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67b4137960 0x7f67b41ddfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): picked mon.noname-g con 0x7f67b4137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f67b4138a50 0x7f67b40725c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): picked mon.noname-d con 0x7f67b4138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67b40a2100 con 0x7f67b40a5bb0 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f67b40a1f80 con 0x7f67b4137960 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f67b4142810 con 0x7f67b4138a50 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67ba302640 10 monclient(hunting): authenticate will time out at 1085.479084s 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 0x7f67b41dda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 0x7f67b41dda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:42614/0 (socket says 172.21.15.43:42614) 2024-04-19T00:46:55.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 1 -- 172.21.15.43:0/2511396236 learned_addr learned my addr 172.21.15.43:0/2511396236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): get_auth_request con 0x7f67b40a5bb0 auth_method 0 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b889f640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f67b4138a50 0x7f67b40725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b889f640 10 monclient(hunting): get_auth_request con 0x7f67b4138a50 auth_method 0 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b889f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b889f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b889f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.346+0000 7f67b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 10 monclient(hunting): handle_auth_done global_id 23221 payload 824 2024-04-19T00:46:55.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 -- 172.21.15.43:0/2511396236 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f67b4138a50 msgr2=0x7f67b40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f67b4138a50 0x7f67b40725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 -- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67b4137960 msgr2=0x7f67b41ddfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67b4137960 0x7f67b41ddfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 monclient: found mon.noname-f 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 -- 172.21.15.43:0/2511396236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b4134860 con 0x7f67b40a5bb0 2024-04-19T00:46:55.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b3fff640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 0x7f67b41dda90 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f67a800b3f0 tx=0x7f67a800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b0ff9640 10 client.?.objecter ms_handle_connect 0x7f67b40a5bb0 2024-04-19T00:46:55.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.350+0000 7f67b0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67a8015020 con 0x7f67b40a5bb0 2024-04-19T00:46:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: dump: 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.359 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: _finish_auth 0 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 5 monclient: authenticate success, global_id 23221 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 monclient: _renew_subs 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b4133970 con 0x7f67b40a5bb0 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a8011aa0 con 0x7f67b40a5bb0 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f67a8011c50 con 0x7f67b40a5bb0 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b0ff9640 10 monclient: dump: 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.361 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 client.23221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 monclient: _renew_subs 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b409fc90 con 0x7f67b40a5bb0 2024-04-19T00:46:55.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 1 librados: init done 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: Running main() from gmock_main.cc 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [----------] Global test environment set-up. 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2379 ms) 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3012 ms) 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3067 ms) 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-19T00:46:55.363 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2825 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3020 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2988 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2961 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3017 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3028 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3027 ms) 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-19T00:46:55.364 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2980 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2963 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3077 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2945 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.Flush (3021 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2994 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3014 ms) 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-19T00:46:55.365 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3016 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2973 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3013 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3019 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2961 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3013 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3075 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (7845 ms) 2024-04-19T00:46:55.366 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-19T00:46:55.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3034 ms) 2024-04-19T00:46:55.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [----------] 26 tests from LibRadosAio (82267 ms total) 2024-04-19T00:46:55.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: 2024-04-19T00:46:55.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-19T00:46:55.367 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67b27fc640 4 set_mon_vals no callback set 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.354+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6794005090 con 0x7f67b40a5bb0 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f67a8018020 con 0x7f67b40a5bb0 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f67a0038e10 0x7f67a003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8123+0+0 (secure 0 0 0) 0x7f67a804a0b0 con 0x7f67b40a5bb0 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 10 client.23221.objecter ms_dispatch 0x7f67b4000ba0 osd_map(100..100 src has 1..100) 2024-04-19T00:46:55.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 3 client.23221.objecter handle_osd_map got epochs [100,100] > 0 2024-04-19T00:46:55.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 3 client.23221.objecter handle_osd_map decoding full epoch 100 2024-04-19T00:46:55.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.362+0000 7f67b0ff9640 20 client.23221.objecter dump_active .. 0 homeless 2024-04-19T00:46:55.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.366+0000 7f67b37fe640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f67a0038e10 0x7f67a003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.366+0000 7f67b37fe640 10 monclient: get_auth_request con 0x7f67a0038e10 auth_method 0 2024-04-19T00:46:55.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.366+0000 7f67b37fe640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f67a0038e10 0x7f67a003b2d0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f67b4133890 tx=0x7f67a4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.366+0000 7f67b0ff9640 10 client.23221.objecter ms_handle_connect 0x7f67a0038e10 2024-04-19T00:46:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.386+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f67a801f020 con 0x7f67b40a5bb0 2024-04-19T00:46:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.386+0000 7f67b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.386+0000 7f67b0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:55.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67ba302640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:46:55.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67ba302640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:46:55.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67ba302640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:46:55.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6794005650 con 0x7f67b40a5bb0 2024-04-19T00:46:55.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67b0ff9640 1 -- 172.21.15.43:0/2511396236 <== mon.1 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f67b4134860 con 0x7f67b40a5bb0 2024-04-19T00:46:55.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:46:55.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67b0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:55.619 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.614+0000 7f67ba302640 10 librados: watch_flush enter 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 10 librados: watch_flush exit 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 20 client.23221.objecter shutdown clearing up homeless session... 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 10 client.23221.objecter successfully canceled tick 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f67a0038e10 msgr2=0x7f67a003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f67a0038e10 0x7f67a003b2d0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f67b4133890 tx=0x7f67a4009190 comp rx=0 tx=0).stop 2024-04-19T00:46:55.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 10 monclient: shutdown 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 msgr2=0x7f67b41dda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 0x7f67b41dda90 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f67a800b3f0 tx=0x7f67a800b8b0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 shutdown_connections 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f67a0038e10 0x7f67a003b2d0 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f67b4138a50 0x7f67b40725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f67b4137960 0x7f67b41ddfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 --2- 172.21.15.43:0/2511396236 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67b40a5bb0 0x7f67b41dda90 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 >> 172.21.15.43:0/2511396236 conn(0x7f67b40a7cb0 msgr2=0x7f67b40a4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 shutdown_connections 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 -- 172.21.15.43:0/2511396236 wait complete. 2024-04-19T00:46:55.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.618+0000 7f67ba302640 1 librados: shutdown 2024-04-19T00:46:55.638 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:46:55.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:46:55.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient: get_monmap_and_config 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient: build_initial_monmap 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient: monmap: 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:55.766913+0000 2024-04-19T00:46:55.766 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:55.766913+0000 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:55.767 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:55.768 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 Processor -- start 2024-04-19T00:46:55.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 -- start start 2024-04-19T00:46:55.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient: init 2024-04-19T00:46:55.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:55.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient: _add_conns ranks=[3,8,9,10,4,6,5,7,0,1,2] 2024-04-19T00:46:55.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34cc138a50 0x7f34cc138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): picked mon.noname-g con 0x7f34cc138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:55.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34cc0a5bb0 0x7f34cc139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): picked mon.noname-e con 0x7f34cc0a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f34cc137960 0x7f34cc137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): picked mon.noname-a con 0x7f34cc137960 addr v2:172.21.15.43:3300/0 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f34cc0a1020 con 0x7f34cc137960 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc1398b0 con 0x7f34cc138a50 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f34cc0a0780 con 0x7f34cc0a5bb0 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d2430640 10 monclient(hunting): authenticate will time out at 1085.894011s 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f34cc137960 0x7f34cc137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34cc0a5bb0 0x7f34cc139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f34cc137960 0x7f34cc137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55362/0 (socket says 172.21.15.43:55362) 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 1 -- 172.21.15.43:0/296085881 learned_addr learned my addr 172.21.15.43:0/296085881 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): get_auth_request con 0x7f34cc0a5bb0 auth_method 0 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): get_auth_request con 0x7f34cc137960 auth_method 0 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34d09cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cbfff640 1 --2- 172.21.15.43:0/296085881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34cc138a50 0x7f34cc138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cbfff640 10 monclient(hunting): get_auth_request con 0x7f34cc138a50 auth_method 0 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient(hunting): handle_auth_done global_id 22274 payload 274 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 -- 172.21.15.43:0/296085881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34cc138a50 msgr2=0x7f34cc138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 --2- 172.21.15.43:0/296085881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34cc138a50 0x7f34cc138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 -- 172.21.15.43:0/296085881 >> v2:172.21.15.43:3300/0 conn(0x7f34cc137960 msgr2=0x7f34cc137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 --2- 172.21.15.43:0/296085881 >> v2:172.21.15.43:3300/0 conn(0x7f34cc137960 0x7f34cc137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:46:55.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 monclient: found mon.noname-e 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.762+0000 7f34cb7fe640 1 -- 172.21.15.43:0/296085881 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc0a0d40 con 0x7f34cc0a5bb0 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cb7fe640 1 --2- 172.21.15.43:0/296085881 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34cc0a5bb0 0x7f34cc139370 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f34c00097c0 tx=0x7f34c001c330 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b3d0cafc523ca820 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 1 -- 172.21.15.43:0/296085881 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34c00b0070 con 0x7f34cc0a5bb0 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: dump: 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.794 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: _finish_auth 0 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: _check_auth_tickets 2024-04-19T00:46:55.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 1 -- 172.21.15.43:0/296085881 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c00aa810 con 0x7f34cc0a5bb0 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 5 monclient: authenticate success, global_id 22274 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 1 -- 172.21.15.43:0/296085881 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34c00a9300 con 0x7f34cc0a5bb0 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 4 set_mon_vals no callback set 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34caffd640 10 monclient: dump: 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.796 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: shutdown 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:55.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- 172.21.15.43:0/296085881 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34cc0a5bb0 msgr2=0x7f34cc139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- 172.21.15.43:0/296085881 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34cc0a5bb0 0x7f34cc139370 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f34c00097c0 tx=0x7f34c001c330 comp rx=0 tx=0).stop 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- 172.21.15.43:0/296085881 shutdown_connections 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- 172.21.15.43:0/296085881 >> v2:172.21.15.43:3300/0 conn(0x7f34cc137960 0x7f34cc137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- 172.21.15.43:0/296085881 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34cc0a5bb0 0x7f34cc139370 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- 172.21.15.43:0/296085881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34cc138a50 0x7f34cc138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- 172.21.15.43:0/296085881 >> 172.21.15.43:0/296085881 conn(0x7f34cc0a7cb0 msgr2=0x7f34cc0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- 172.21.15.43:0/296085881 shutdown_connections 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- 172.21.15.43:0/296085881 wait complete. 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: build_initial_monmap 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: monmap: 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:55.798 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 librados: starting msgr at 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 librados: starting objecter 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 Processor -- start 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- start start 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 librados: setting wanted keys 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 librados: calling monclient init 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: init 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient: _add_conns ranks=[0,5,2,8,3,1,6,7,4] 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0a5bb0 0x7f34cc1d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): picked mon.noname-a con 0x7f34cc0a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f34cc137960 0x7f34cc1d1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): picked mon.noname-h con 0x7f34cc137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc138a50 0x7f34cc13c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): picked mon.noname-b con 0x7f34cc138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:55.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc0a2100 con 0x7f34cc0a5bb0 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f34cc0a1f80 con 0x7f34cc138a50 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f34cc142810 con 0x7f34cc137960 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d2430640 10 monclient(hunting): authenticate will time out at 1085.899130s 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc138a50 0x7f34cc13c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc138a50 0x7f34cc13c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37246/0 (socket says 172.21.15.43:37246) 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 -- 172.21.15.43:0/4293237766 learned_addr learned my addr 172.21.15.43:0/4293237766 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): get_auth_request con 0x7f34cc138a50 auth_method 0 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0a5bb0 0x7f34cc1d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): get_auth_request con 0x7f34cc0a5bb0 auth_method 0 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cb7fe640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f34cc137960 0x7f34cc1d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cb7fe640 10 monclient(hunting): get_auth_request con 0x7f34cc137960 auth_method 0 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient(hunting): handle_auth_done global_id 22592 payload 824 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f34cc137960 msgr2=0x7f34cc1d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f34cc137960 0x7f34cc1d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0a5bb0 msgr2=0x7f34cc1d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0a5bb0 0x7f34cc1d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient: _finish_hunting 0 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 monclient: found mon.noname-b 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34cc1221a0 con 0x7f34cc138a50 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34cbfff640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc0a5bb0 0x7f34cc1d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:55.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34d09cd640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc138a50 0x7f34cc13c950 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009d90 tx=0x7f34c400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34c8ff9640 10 client.?.objecter ms_handle_connect 0x7f34cc138a50 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.766+0000 7f34c8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34c40039a0 con 0x7f34cc138a50 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: dump: 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.804 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: _finish_auth 0 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34c4003b80 con 0x7f34cc138a50 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34c4003d30 con 0x7f34cc138a50 2024-04-19T00:46:55.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: dump: 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.806 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34ca7fc640 4 set_mon_vals no callback set 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 5 monclient: authenticate success, global_id 22592 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 monclient: _renew_subs 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34cc09fc90 con 0x7f34cc138a50 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 client.22592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 monclient: _renew_subs 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34cc122480 con 0x7f34cc138a50 2024-04-19T00:46:55.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 1 librados: init done 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 0x7f34cc13ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc06d140 con 0x7f34cc1d16c0 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 0x7f34cc13ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient: get_auth_request con 0x7f34cc1d16c0 auth_method 0 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f34c4012050 con 0x7f34cc138a50 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34b8038e40 0x7f34b803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 10 monclient(hunting): handle_auth_done global_id 29502 payload 824 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(100..100 src has 1..100) ==== 8123+0+0 (secure 0 0 0) 0x7f34c404a060 con 0x7f34cc138a50 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 client.22592.objecter ms_dispatch 0x7f34cc000ba0 osd_map(100..100 src has 1..100) 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 3 client.22592.objecter handle_osd_map got epochs [100,100] > 0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 3 client.22592.objecter handle_osd_map decoding full epoch 100 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34cc09ee60 con 0x7f34cc1d16c0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 20 client.22592.objecter dump_active .. 0 homeless 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 client.22592.objecter ms_handle_connect 0x7f34cc1d16c0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 client.22592.objecter resend_mon_ops 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cbfff640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 0x7f34cc13ce90 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f34bc010b90 tx=0x7f34bc00e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cb7fe640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34b8038e40 0x7f34b803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cb7fe640 10 monclient: get_auth_request con 0x7f34b8038e40 auth_method 0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34bc018070 con 0x7f34cc1d16c0 2024-04-19T00:46:55.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: dump: 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.810 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f34bc010e90 con 0x7f34cc1d16c0 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 msgr2=0x7f34cc13ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 0x7f34cc13ce90 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f34bc010b90 tx=0x7f34bc00e5a0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 msgr2=0x7f34cc13ce90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc1d16c0 0x7f34cc13ce90 secure :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f34bc010b90 tx=0x7f34bc00e5a0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34c8ff9640 10 client.22592.objecter ms_handle_connect 0x7f34b8038e40 2024-04-19T00:46:55.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.770+0000 7f34cb7fe640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34b8038e40 0x7f34b803b300 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f34c00ad000 tx=0x7f34c00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 0x7f34cc09c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34cc13d7e0 con 0x7f34cc13d3d0 2024-04-19T00:46:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 0x7f34cc09c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient: get_auth_request con 0x7f34cc13d3d0 auth_method 0 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 10 monclient(hunting): handle_auth_done global_id 29511 payload 824 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 1 -- 172.21.15.43:0/4293237766 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f34cc076190 con 0x7f34cc13d3d0 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34cbfff640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 0x7f34cc09c280 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f34bc01a000 tx=0x7f34bc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 client.22592.objecter ms_handle_connect 0x7f34cc13d3d0 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 client.22592.objecter resend_mon_ops 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34bc014b70 con 0x7f34cc13d3d0 2024-04-19T00:46:55.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 monclient: dump: 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.815 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7f34bc014700 con 0x7f34cc13d3d0 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 msgr2=0x7f34cc09c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 0x7f34cc09c280 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f34bc01a000 tx=0x7f34bc0046d0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 msgr2=0x7f34cc09c280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34c8ff9640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34cc13d3d0 0x7f34cc09c280 secure :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7f34bc01a000 tx=0x7f34bc0046d0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.816 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 159274, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:55.817 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:55.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:55.819 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:55.820 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:55.821 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.822 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:55.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:55.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:55.826 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:55.827 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:55.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:55.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:55.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.838 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:55.839 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 10 librados: watch_flush enter 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 10 librados: watch_flush exit 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 20 client.22592.objecter shutdown clearing up homeless session... 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 10 client.22592.objecter successfully canceled tick 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34b8038e40 msgr2=0x7f34b803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34b8038e40 0x7f34b803b300 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7f34c00ad000 tx=0x7f34c00a6000 comp rx=0 tx=0).stop 2024-04-19T00:46:55.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 10 monclient: shutdown 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc138a50 msgr2=0x7f34cc13c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 --2- 172.21.15.43:0/4293237766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34cc138a50 0x7f34cc13c950 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f34c4009d90 tx=0x7f34c400c490 comp rx=0 tx=0).stop 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d09cd640 1 -- 172.21.15.43:0/4293237766 reap_dead start 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d09cd640 1 -- 172.21.15.43:0/4293237766 reap_dead start 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 shutdown_connections 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 >> 172.21.15.43:0/4293237766 conn(0x7f34cc0a7cb0 msgr2=0x7f34cc06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 shutdown_connections 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 -- 172.21.15.43:0/4293237766 wait complete. 2024-04-19T00:46:55.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.778+0000 7f34d2430640 1 librados: shutdown 2024-04-19T00:46:55.842 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:55.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:46:55.843 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient: get_monmap_and_config 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient: build_initial_monmap 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient: monmap: 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:55.964584+0000 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:55.964584+0000 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:55.963 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:55.964 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:55.965 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:55.965 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:55.965 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:55.965 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 1 Processor -- start 2024-04-19T00:46:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 1 -- start start 2024-04-19T00:46:55.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient: init 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient: _add_conns ranks=[5,1,0,8,2,9,3,7,10,6,4] 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40d8138cd0 0x7f40d81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient(hunting): picked mon.noname-h con 0x7f40d8138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f40d80a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40d81395f0 addr v1:172.21.15.43:6789/0 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d81366d0 con 0x7f40d81395f0 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d80a3dc0 con 0x7f40d80a5b80 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f40d80a4410 con 0x7f40d8138cd0 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40dd6c4640 10 monclient(hunting): authenticate will time out at 1086.091654s 2024-04-19T00:46:55.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40d77fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f40d81395f0 0x7f40d8135fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50456/0 (socket says 172.21.15.43:50456) 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.958+0000 7f40d77fe640 1 -- 172.21.15.43:0/3368260233 learned_addr learned my addr 172.21.15.43:0/3368260233 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/3368260233 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40d8138cd0 0x7f40d81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): get_auth_request con 0x7f40d8138cd0 auth_method 0 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 347229172 0 0) 0x7f40d81366d0 con 0x7f40d81395f0 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): my global_id is 29520 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40c0003530 con 0x7f40d81395f0 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443699109 0 0) 0x7f40d80a3dc0 con 0x7f40d80a5b80 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient(hunting): my global_id is 23230 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40d81366d0 con 0x7f40d80a5b80 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_done global_id 44168 payload 274 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3716931431 0 0) 0x7f40c0003530 con 0x7f40d81395f0 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 1 -- 172.21.15.43:0/3368260233 >> v1:172.21.15.161:6789/0 conn(0x7f40d80a5b80 legacy=0x7f40d81358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 1 -- 172.21.15.43:0/3368260233 >> v1:172.21.15.43:6789/0 conn(0x7f40d81395f0 legacy=0x7f40d8135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 1 monclient: found mon.noname-h 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 1 -- 172.21.15.43:0/3368260233 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d80a40a0 con 0x7f40d8138cd0 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/3368260233 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40d8138cd0 0x7f40d81390b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f40c80097f0 tx=0x7f40c801c260 comp rx=0 tx=0).ready entity=mon.5 client_cookie=d5d935af7745bf13 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c80b0070 con 0x7f40d8138cd0 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: dump: 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.980 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: _finish_auth 0 2024-04-19T00:46:55.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 5 monclient: authenticate success, global_id 44168 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c80aa3a0 con 0x7f40d8138cd0 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 4 set_mon_vals no callback set 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 1 -- 172.21.15.43:0/3368260233 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40d0001cf0 con 0x7f40d8138cd0 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40d5ffb640 10 monclient: dump: 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.982 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 10 monclient: shutdown 2024-04-19T00:46:55.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 1 -- 172.21.15.43:0/3368260233 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40d8138cd0 msgr2=0x7f40d81390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/3368260233 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40d8138cd0 0x7f40d81390b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f40c80097f0 tx=0x7f40c801c260 comp rx=0 tx=0).stop 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 1 -- 172.21.15.43:0/3368260233 shutdown_connections 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/3368260233 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f40d8138cd0 0x7f40d81390b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.962+0000 7f40dd6c4640 1 -- 172.21.15.43:0/3368260233 >> 172.21.15.43:0/3368260233 conn(0x7f40d80ab2f0 msgr2=0x7f40d80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- 172.21.15.43:0/3368260233 shutdown_connections 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- 172.21.15.43:0/3368260233 wait complete. 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: build_initial_monmap 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: monmap: 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:55.984 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 librados: starting msgr at 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 librados: starting objecter 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 Processor -- start 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- start start 2024-04-19T00:46:55.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 librados: setting wanted keys 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 librados: calling monclient init 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: init 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: _add_conns ranks=[1,8,0,5,3,2,6,4,7] 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d80a5b80 0x7f40d809f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): picked mon.noname-f con 0x7f40d80a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40d8138cd0 0x7f40d809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): picked mon.noname-e con 0x7f40d8138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 0x7f40d809bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): picked mon.noname-a con 0x7f40d81395f0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f40d80a2010 con 0x7f40d81395f0 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f40d80a1e90 con 0x7f40d80a5b80 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f40d8144ac0 con 0x7f40d8138cd0 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): authenticate will time out at 1086.097498s 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 0x7f40d809bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40d8138cd0 0x7f40d809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 0x7f40d809bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55414/0 (socket says 172.21.15.43:55414) 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 -- 172.21.15.43:0/2799127756 learned_addr learned my addr 172.21.15.43:0/2799127756 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): get_auth_request con 0x7f40d81395f0 auth_method 0 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): get_auth_request con 0x7f40d8138cd0 auth_method 0 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d80a5b80 0x7f40d809f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): get_auth_request con 0x7f40d80a5b80 auth_method 0 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient(hunting): handle_auth_done global_id 29529 payload 824 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40d8138cd0 msgr2=0x7f40d809ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f40d8138cd0 0x7f40d809ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d80a5b80 msgr2=0x7f40d809f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d80a5b80 0x7f40d809f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 monclient: found mon.noname-a 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d809e820 con 0x7f40d81395f0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d77fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 0x7f40d809bfb0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f40d000cc20 tx=0x7f40d000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 client.?.objecter ms_handle_connect 0x7f40d81395f0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40d000ce30 con 0x7f40d81395f0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:46:55.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: dump: 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.990 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: _finish_auth 0 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40d000acf0 con 0x7f40d81395f0 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 5 monclient: authenticate success, global_id 29529 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: _renew_subs 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40d809ede0 con 0x7f40d81395f0 2024-04-19T00:46:55.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40d0012ce0 con 0x7f40d81395f0 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 monclient: dump: 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.992 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d57fa640 4 set_mon_vals no callback set 2024-04-19T00:46:55.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 client.29529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: _renew_subs 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d81361e0 con 0x7f40d81395f0 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 librados: init done 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 0x7f40d809c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f40d806d140 con 0x7f40d809d2b0 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 0x7f40d809c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient: get_auth_request con 0x7f40d809d2b0 auth_method 0 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:55.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f40d0012400 con 0x7f40d81395f0 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40c4038e10 0x7f40c403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_done global_id 22601 payload 824 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40d8068240 con 0x7f40d809d2b0 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8123+0+0 (secure 0 0 0) 0x7f40d004a2a0 con 0x7f40d81395f0 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 client.29529.objecter ms_dispatch 0x7f40d8000ba0 osd_map(100..100 src has 1..100) 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 3 client.29529.objecter handle_osd_map got epochs [100,100] > 0 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 3 client.29529.objecter handle_osd_map decoding full epoch 100 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 20 client.29529.objecter dump_active .. 0 homeless 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 client.29529.objecter ms_handle_connect 0x7f40d809d2b0 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40bf7fe640 10 client.29529.objecter resend_mon_ops 2024-04-19T00:46:55.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.966+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 0x7f40d809c4f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f40d80a3520 tx=0x7f40c8019060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40d67fc640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40c4038e10 0x7f40c403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40d67fc640 10 monclient: get_auth_request con 0x7f40c4038e10 auth_method 0 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c80b9070 con 0x7f40d809d2b0 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 10 monclient: dump: 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:55.996 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:55.997 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40c8003620 con 0x7f40d809d2b0 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 msgr2=0x7f40d809c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 0x7f40d809c4f0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f40d80a3520 tx=0x7f40c8019060 comp rx=0 tx=0).stop 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 msgr2=0x7f40d809c4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d809d2b0 0x7f40d809c4f0 secure :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f40d80a3520 tx=0x7f40c8019060 comp rx=0 tx=0).stop 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40d67fc640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40c4038e10 0x7f40c403b2d0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f40d81365f0 tx=0x7f40cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:55.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.970+0000 7f40bf7fe640 10 client.29529.objecter ms_handle_connect 0x7f40c4038e10 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40dd6c4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 0x7f40d81d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40dd6c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f40d80693b0 con 0x7f40d81d09c0 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 0x7f40d81d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient: get_auth_request con 0x7f40d81d09c0 auth_method 0 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 10 monclient(hunting): handle_auth_done global_id 22610 payload 824 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 1 -- 172.21.15.43:0/2799127756 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40d8076190 con 0x7f40d81d09c0 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40d6ffd640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 0x7f40d81d0dd0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f40c801c2a0 tx=0x7f40c80a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40bf7fe640 10 client.29529.objecter ms_handle_connect 0x7f40d81d09c0 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40bf7fe640 10 client.29529.objecter resend_mon_ops 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f40c80b9040 con 0x7f40d81d09c0 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:56.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.974+0000 7f40bf7fe640 10 monclient: dump: 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.002 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f40c80b5990 con 0x7f40d81d09c0 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 msgr2=0x7f40d81d0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 0x7f40d81d0dd0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f40c801c2a0 tx=0x7f40c80a8000 comp rx=0 tx=0).stop 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 msgr2=0x7f40d81d0dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40bf7fe640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f40d81d09c0 0x7f40d81d0dd0 secure :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f40c801c2a0 tx=0x7f40c80a8000 comp rx=0 tx=0).stop 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:56.003 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 159474, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:56.004 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:56.005 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.006 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.008 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:56.009 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.010 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.011 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.013 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.017 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.018 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.021 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.028 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.029 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:56.030 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 10 librados: watch_flush enter 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 10 librados: watch_flush exit 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 20 client.29529.objecter shutdown clearing up homeless session... 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 10 client.29529.objecter successfully canceled tick 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40c4038e10 msgr2=0x7f40c403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f40c4038e10 0x7f40c403b2d0 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f40d81365f0 tx=0x7f40cc009040 comp rx=0 tx=0).stop 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 10 monclient: shutdown 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 msgr2=0x7f40d809bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 0x7f40d809bfb0 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f40d000cc20 tx=0x7f40d000a430 comp rx=0 tx=0).stop 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40d77fe640 1 -- 172.21.15.43:0/2799127756 reap_dead start 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 shutdown_connections 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 --2- 172.21.15.43:0/2799127756 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f40d81395f0 0x7f40d809bfb0 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 >> 172.21.15.43:0/2799127756 conn(0x7f40d80ab2f0 msgr2=0x7f40d8130c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 shutdown_connections 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 -- 172.21.15.43:0/2799127756 wait complete. 2024-04-19T00:46:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:55.978+0000 7f40dd6c4640 1 librados: shutdown 2024-04-19T00:46:56.033 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:56.033 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:46:56.033 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:46:56.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.146+0000 7f00636c7640 10 monclient: get_monmap_and_config 2024-04-19T00:46:56.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.146+0000 7f00636c7640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.146+0000 7f00636c7640 10 monclient: monmap: 2024-04-19T00:46:56.151 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:56.153073+0000 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:56.153073+0000 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.152 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:56.153 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.146+0000 7f00636c7640 1 Processor -- start 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.146+0000 7f00636c7640 1 -- start start 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.146+0000 7f00636c7640 10 monclient: init 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient: _add_conns ranks=[3,2,9,5,8,10,7,4,1,6,0] 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c138a50 0x7f005c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): picked mon.noname-g con 0x7f005c138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 0x7f005c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): picked mon.noname-b con 0x7f005c0a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:56.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f005c137960 0x7f005c137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): picked mon.noname-a con 0x7f005c137960 addr v2:172.21.15.43:3300/0 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f005c0a1020 con 0x7f005c137960 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f005c1398b0 con 0x7f005c0a5bb0 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f005c0a0780 con 0x7f005c138a50 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f00636c7640 10 monclient(hunting): authenticate will time out at 1086.280675s 2024-04-19T00:46:56.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f005c137960 0x7f005c137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 0x7f005c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f005c137960 0x7f005c137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55422/0 (socket says 172.21.15.43:55422) 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 0x7f005c139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37274/0 (socket says 172.21.15.43:37274) 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 -- 172.21.15.43:0/3135340238 learned_addr learned my addr 172.21.15.43:0/3135340238 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): get_auth_request con 0x7f005c137960 auth_method 0 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): get_auth_request con 0x7f005c0a5bb0 auth_method 0 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061463640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c138a50 0x7f005c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061463640 10 monclient(hunting): get_auth_request con 0x7f005c138a50 auth_method 0 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061463640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061463640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061463640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient(hunting): handle_auth_done global_id 29538 payload 274 2024-04-19T00:46:56.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 -- 172.21.15.43:0/3135340238 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c138a50 msgr2=0x7f005c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c138a50 0x7f005c138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 -- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 msgr2=0x7f005c139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 0x7f005c139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 monclient: found mon.noname-a 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 -- 172.21.15.43:0/3135340238 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c0a0d40 con 0x7f005c137960 2024-04-19T00:46:56.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0060c62640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 0x7f005c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:56.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061463640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c138a50 0x7f005c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.150+0000 7f0061c64640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c137960 0x7f005c137dc0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f0054009870 tx=0x7f005401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cfd9a4a877d7fe server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 1 -- 172.21.15.43:0/3135340238 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00540b2070 con 0x7f005c137960 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: dump: 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.172 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.173 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 1 -- 172.21.15.43:0/3135340238 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00540ab7b0 con 0x7f005c137960 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 1 -- 172.21.15.43:0/3135340238 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00540abe50 con 0x7f005c137960 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 5 monclient: authenticate success, global_id 29538 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 4 set_mon_vals no callback set 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0053fff640 10 monclient: dump: 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.175 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: shutdown 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c137960 msgr2=0x7f005c137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c137960 0x7f005c137dc0 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7f0054009870 tx=0x7f005401c3f0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- 172.21.15.43:0/3135340238 shutdown_connections 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c137960 0x7f005c137dc0 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c0a5bb0 0x7f005c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- 172.21.15.43:0/3135340238 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c138a50 0x7f005c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- 172.21.15.43:0/3135340238 >> 172.21.15.43:0/3135340238 conn(0x7f005c0a7cb0 msgr2=0x7f005c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- 172.21.15.43:0/3135340238 shutdown_connections 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- 172.21.15.43:0/3135340238 wait complete. 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: monmap: 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:56.177 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.178 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 librados: starting msgr at 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 librados: starting objecter 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 Processor -- start 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- start start 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 librados: setting wanted keys 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 librados: calling monclient init 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: init 2024-04-19T00:46:56.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: _add_conns ranks=[0,3,2,6,7,5,4,1,8] 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c0a5bb0 0x7f005c1368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): picked mon.noname-a con 0x7f005c0a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c137960 0x7f005c136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): picked mon.noname-g con 0x7f005c137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c138a50 0x7f005c13c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): picked mon.noname-b con 0x7f005c138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:56.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f005c0a2100 con 0x7f005c0a5bb0 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f005c0a1f80 con 0x7f005c138a50 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f005c142810 con 0x7f005c137960 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient(hunting): authenticate will time out at 1086.286972s 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c138a50 0x7f005c13c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c0a5bb0 0x7f005c1368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c138a50 0x7f005c13c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37276/0 (socket says 172.21.15.43:37276) 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 -- 172.21.15.43:0/1610332976 learned_addr learned my addr 172.21.15.43:0/1610332976 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): get_auth_request con 0x7f005c0a5bb0 auth_method 0 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): get_auth_request con 0x7f005c138a50 auth_method 0 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0060c62640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c137960 0x7f005c136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient(hunting): handle_auth_done global_id 22628 payload 824 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c137960 msgr2=0x7f005c136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c137960 0x7f005c136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c0a5bb0 msgr2=0x7f005c1368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c0a5bb0 0x7f005c1368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 monclient: found mon.noname-b 2024-04-19T00:46:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c09ee60 con 0x7f005c138a50 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0060c62640 10 monclient: get_auth_request con 0x7f005c137960 auth_method 0 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0060c62640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f005c137960 0x7f005c136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061463640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f005c0a5bb0 0x7f005c1368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0061c64640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c138a50 0x7f005c13c860 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f00540b0040 tx=0x7f0054019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 client.?.objecter ms_handle_connect 0x7f005c138a50 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00540b2070 con 0x7f005c138a50 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 monclient: dump: 2024-04-19T00:46:56.186 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.187 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 5 monclient: authenticate success, global_id 22628 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00540abdd0 con 0x7f005c138a50 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0054003a90 con 0x7f005c138a50 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00636c7640 10 monclient: _renew_subs 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.154+0000 7f00537fe640 4 set_mon_vals no callback set 2024-04-19T00:46:56.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f005c1219e0 con 0x7f005c138a50 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 10 client.22628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: dump: 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.189 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 10 monclient: _renew_subs 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f005c1376a0 con 0x7f005c138a50 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 1 librados: init done 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 0x7f005c13cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f005c06d140 con 0x7f005c121bb0 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 0x7f005c13cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient: get_auth_request con 0x7f005c121bb0 auth_method 0 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f00540b6030 con 0x7f005c138a50 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0040038e00 0x7f004003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 10 monclient(hunting): handle_auth_done global_id 22459 payload 824 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f005c1d17e0 con 0x7f005c121bb0 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(100..100 src has 1..100) ==== 8123+0+0 (secure 0 0 0) 0x7f00540e37a0 con 0x7f005c138a50 2024-04-19T00:46:56.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 client.22628.objecter ms_dispatch 0x7f005c000ba0 osd_map(100..100 src has 1..100) 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 3 client.22628.objecter handle_osd_map got epochs [100,100] > 0 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 3 client.22628.objecter handle_osd_map decoding full epoch 100 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 20 client.22628.objecter dump_active .. 0 homeless 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0061463640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 0x7f005c13cda0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f0058004950 tx=0x7f00580077c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 client.22628.objecter ms_handle_connect 0x7f005c121bb0 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 client.22628.objecter resend_mon_ops 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0060c62640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0040038e00 0x7f004003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0058014a60 con 0x7f005c121bb0 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: dump: 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.192 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0060c62640 10 monclient: get_auth_request con 0x7f0040038e00 auth_method 0 2024-04-19T00:46:56.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f005801a070 con 0x7f005c121bb0 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 msgr2=0x7f005c13cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 0x7f005c13cda0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f0058004950 tx=0x7f00580077c0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 msgr2=0x7f005c13cda0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c121bb0 0x7f005c13cda0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0060c62640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0040038e00 0x7f004003b2c0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f005c13df40 tx=0x7f004c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.158+0000 7f0051ffb640 10 client.22628.objecter ms_handle_connect 0x7f0040038e00 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 0x7f005c13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f005c076190 con 0x7f005c13d2e0 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 0x7f005c13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient: get_auth_request con 0x7f005c13d2e0 auth_method 0 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 10 monclient(hunting): handle_auth_done global_id 22468 payload 824 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 1 -- 172.21.15.43:0/1610332976 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f005c13dc00 con 0x7f005c13d2e0 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061463640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 0x7f005c13d6c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f00580049e0 tx=0x7f0058004a50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 client.22628.objecter ms_handle_connect 0x7f005c13d2e0 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 client.22628.objecter resend_mon_ops 2024-04-19T00:46:56.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0058014e60 con 0x7f005c13d2e0 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 monclient: dump: 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.197 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 159686, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:56.198 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.199 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.200 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:56.201 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.202 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.204 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.206 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.208 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.210 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.211 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.213 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:56.214 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.216 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.219 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:56.220 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f0058015690 con 0x7f005c13d2e0 2024-04-19T00:46:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 msgr2=0x7f005c13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 0x7f005c13d6c0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f00580049e0 tx=0x7f0058004a50 comp rx=0 tx=0).stop 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 msgr2=0x7f005c13d6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0051ffb640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f005c13d2e0 0x7f005c13d6c0 secure :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f00580049e0 tx=0x7f0058004a50 comp rx=0 tx=0).stop 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 10 librados: watch_flush enter 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 10 librados: watch_flush exit 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 20 client.22628.objecter shutdown clearing up homeless session... 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 10 client.22628.objecter successfully canceled tick 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0040038e00 msgr2=0x7f004003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0040038e00 0x7f004003b2c0 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7f005c13df40 tx=0x7f004c008040 comp rx=0 tx=0).stop 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 10 monclient: shutdown 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c138a50 msgr2=0x7f005c13c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 --2- 172.21.15.43:0/1610332976 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f005c138a50 0x7f005c13c860 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f00540b0040 tx=0x7f0054019190 comp rx=0 tx=0).stop 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061c64640 1 -- 172.21.15.43:0/1610332976 reap_dead start 2024-04-19T00:46:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f0061c64640 1 -- 172.21.15.43:0/1610332976 reap_dead start 2024-04-19T00:46:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 shutdown_connections 2024-04-19T00:46:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 >> 172.21.15.43:0/1610332976 conn(0x7f005c0a7cb0 msgr2=0x7f005c06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 shutdown_connections 2024-04-19T00:46:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 -- 172.21.15.43:0/1610332976 wait complete. 2024-04-19T00:46:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.166+0000 7f00636c7640 1 librados: shutdown 2024-04-19T00:46:56.224 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:56.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:46:56.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:46:56.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient: get_monmap_and_config 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient: monmap: 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:56.364123+0000 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:56.364123+0000 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.363 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:56.364 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:56.364 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 Processor -- start 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 -- start start 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient: init 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient: _add_conns ranks=[5,4,1,2,9,7,10,0,8,6,3] 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6004138a30 0x7f6004138e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): picked mon.noname-h con 0x7f6004138a30 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:56.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 0x7f6004135680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): picked mon.noname-c con 0x7f6004139350 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6004135bc0 addr v1:172.21.15.161:6789/0 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60040a3a30 con 0x7f6004135bc0 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f60040a3ff0 con 0x7f6004139350 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f60040a3d10 con 0x7f6004138a30 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.358+0000 7f6008e84640 10 monclient(hunting): authenticate will time out at 1086.491772s 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 0x7f6004135680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 0x7f6004135680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42878/0 (socket says 172.21.15.43:42878) 2024-04-19T00:46:56.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 -- 172.21.15.43:0/3884474526 learned_addr learned my addr 172.21.15.43:0/3884474526 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): get_auth_request con 0x7f6004139350 auth_method 0 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60037fe640 1 --2- 172.21.15.43:0/3884474526 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6004138a30 0x7f6004138e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60037fe640 10 monclient(hunting): get_auth_request con 0x7f6004138a30 auth_method 0 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient(hunting): handle_auth_done global_id 22477 payload 274 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 -- 172.21.15.43:0/3884474526 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6004138a30 msgr2=0x7f6004138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3884474526 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6004138a30 0x7f6004138e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 -- 172.21.15.43:0/3884474526 >> v1:172.21.15.161:6789/0 conn(0x7f6004135bc0 legacy=0x7f600413ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 monclient: found mon.noname-c 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:56.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 -- 172.21.15.43:0/3884474526 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60040a4ab0 con 0x7f6004139350 2024-04-19T00:46:56.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3884474526 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 0x7f6004135680 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009e80 tx=0x7f5ffc01c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f704b44e4fa419b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60027fc640 1 -- 172.21.15.43:0/3884474526 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ffc0090d0 con 0x7f6004139350 2024-04-19T00:46:56.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.362+0000 7f60027fc640 10 monclient: dump: 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.371 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 1 -- 172.21.15.43:0/3884474526 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ffc0a5b60 con 0x7f6004139350 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 5 monclient: authenticate success, global_id 22477 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 4 set_mon_vals no callback set 2024-04-19T00:46:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 1 -- 172.21.15.43:0/3884474526 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ffc0a5d50 con 0x7f6004139350 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f60027fc640 10 monclient: dump: 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.374 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.374 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.374 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 10 monclient: shutdown 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 -- 172.21.15.43:0/3884474526 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 msgr2=0x7f6004135680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 --2- 172.21.15.43:0/3884474526 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 0x7f6004135680 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc009e80 tx=0x7f5ffc01c910 comp rx=0 tx=0).stop 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 -- 172.21.15.43:0/3884474526 shutdown_connections 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 --2- 172.21.15.43:0/3884474526 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6004139350 0x7f6004135680 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 --2- 172.21.15.43:0/3884474526 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6004138a30 0x7f6004138e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 -- 172.21.15.43:0/3884474526 >> 172.21.15.43:0/3884474526 conn(0x7f60040ab0c0 msgr2=0x7f60040ab4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 -- 172.21.15.43:0/3884474526 shutdown_connections 2024-04-19T00:46:56.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 -- 172.21.15.43:0/3884474526 wait complete. 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 10 monclient: monmap: 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:56.381 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 librados: starting msgr at 2024-04-19T00:46:56.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 librados: starting objecter 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 Processor -- start 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 -- start start 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 librados: setting wanted keys 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 1 librados: calling monclient init 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.366+0000 7f6008e84640 10 monclient: init 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient: _add_conns ranks=[0,8,6,1,3,5,2,4,7] 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 0x7f6004134e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): picked mon.noname-a con 0x7f6004135bc0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6004138a30 0x7f6004135360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): picked mon.noname-e con 0x7f6004138a30 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f6004139350 0x7f600409fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): picked mon.noname-d con 0x7f6004139350 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:56.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f60040a27e0 con 0x7f6004135bc0 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f60040a2660 con 0x7f6004139350 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f60040a1c40 con 0x7f6004138a30 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): authenticate will time out at 1086.500249s 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 0x7f6004134e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f6004139350 0x7f600409fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 0x7f6004134e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55442/0 (socket says 172.21.15.43:55442) 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 -- 172.21.15.43:0/3670833629 learned_addr learned my addr 172.21.15.43:0/3670833629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): get_auth_request con 0x7f6004135bc0 auth_method 0 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): get_auth_request con 0x7f6004139350 auth_method 0 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient(hunting): handle_auth_done global_id 29556 payload 824 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6004138a30 msgr2=0x7f6004135360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6004138a30 0x7f6004135360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6004138a30 0x7f6004135360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f6004139350 msgr2=0x7f600409fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f6004139350 0x7f600409fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 monclient: found mon.noname-a 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60041310d0 con 0x7f6004135bc0 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6003fff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f6004139350 0x7f600409fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f60037fe640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 0x7f6004134e20 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400d800 tx=0x7f5ff400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f6004135bc0 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ff400bb30 con 0x7f6004135bc0 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: dump: 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.388 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff4019070 con 0x7f6004135bc0 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 5 monclient: authenticate success, global_id 29556 2024-04-19T00:46:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient: _renew_subs 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6004130df0 con 0x7f6004135bc0 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ff400bcd0 con 0x7f6004135bc0 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 client.29556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f5febfff640 10 monclient: dump: 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6001ffb640 4 set_mon_vals no callback set 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient: _renew_subs 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60041d00d0 con 0x7f6004135bc0 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 librados: init done 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 0x7f60040a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f600406d140 con 0x7f60040a0370 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 0x7f60040a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6002ffd640 10 monclient: get_auth_request con 0x7f60040a0370 auth_method 0 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6002ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6002ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.370+0000 7f6002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5ff400b6c0 con 0x7f6004135bc0 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6002ffd640 10 monclient(hunting): handle_auth_done global_id 22668 payload 824 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6002ffd640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f600406a4c0 con 0x7f60040a0370 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ff0038e40 0x7f5ff003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 7725+0+0 (secure 0 0 0) 0x7f5ff400f040 con 0x7f6004135bc0 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 client.29556.objecter ms_dispatch 0x7f6004000ba0 osd_map(101..101 src has 1..101) 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 3 client.29556.objecter handle_osd_map got epochs [101,101] > 0 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 3 client.29556.objecter handle_osd_map decoding full epoch 101 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 0x7f60040a0750 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f60040a72d0 tx=0x7f5ffc0196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 20 client.29556.objecter dump_active .. 0 homeless 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 client.29556.objecter ms_handle_connect 0x7f60040a0370 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 client.29556.objecter resend_mon_ops 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ffc0bd070 con 0x7f60040a0370 2024-04-19T00:46:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6003fff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ff0038e40 0x7f5ff003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 monclient: dump: 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6003fff640 10 monclient: get_auth_request con 0x7f5ff0038e40 auth_method 0 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ffc0a56f0 con 0x7f60040a0370 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 msgr2=0x7f60040a0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 0x7f60040a0750 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f60040a72d0 tx=0x7f5ffc0196b0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 msgr2=0x7f60040a0750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0370 0x7f60040a0750 secure :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f60040a72d0 tx=0x7f5ffc0196b0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f5febfff640 10 client.29556.objecter ms_handle_connect 0x7f5ff0038e40 2024-04-19T00:46:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.374+0000 7f6003fff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ff0038e40 0x7f5ff003b300 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7f6004130410 tx=0x7f5ff8007dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6008e84640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6008e84640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 0x7f600409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6008e84640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f6004068240 con 0x7f60040a0c90 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 0x7f600409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient: get_auth_request con 0x7f60040a0c90 auth_method 0 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 10 monclient(hunting): handle_auth_done global_id 22677 payload 824 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 1 -- 172.21.15.43:0/3670833629 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f600402d680 con 0x7f60040a0c90 2024-04-19T00:46:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f6002ffd640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 0x7f600409ba70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc01c740 tx=0x7f5ffc0a7000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f5febfff640 10 client.29556.objecter ms_handle_connect 0x7f60040a0c90 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.378+0000 7f5febfff640 10 client.29556.objecter resend_mon_ops 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5ffc009b00 con 0x7f60040a0c90 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 10 monclient: dump: 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f5ffc009b00 con 0x7f60040a0c90 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 msgr2=0x7f600409ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 0x7f600409ba70 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc01c740 tx=0x7f5ffc0a7000 comp rx=0 tx=0).stop 2024-04-19T00:46:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 msgr2=0x7f600409ba70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f5febfff640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f60040a0c90 0x7f600409ba70 secure :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc01c740 tx=0x7f5ffc0a7000 comp rx=0 tx=0).stop 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 159898, 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:56.403 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.404 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.405 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:56.425 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 10 librados: watch_flush enter 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 10 librados: watch_flush exit 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 20 client.29556.objecter shutdown clearing up homeless session... 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 10 client.29556.objecter successfully canceled tick 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ff0038e40 msgr2=0x7f5ff003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ff0038e40 0x7f5ff003b300 secure :-1 s=READY pgs=3329 cs=0 l=1 rev1=1 crypto rx=0x7f6004130410 tx=0x7f5ff8007dc0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 10 monclient: shutdown 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 msgr2=0x7f6004134e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 0x7f6004134e20 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f5ff400d800 tx=0x7f5ff400dcc0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6003fff640 1 -- 172.21.15.43:0/3670833629 reap_dead start 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 shutdown_connections 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 --2- 172.21.15.43:0/3670833629 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6004135bc0 0x7f6004134e20 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 >> 172.21.15.43:0/3670833629 conn(0x7f60040ab0c0 msgr2=0x7f60040a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 shutdown_connections 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 -- 172.21.15.43:0/3670833629 wait complete. 2024-04-19T00:46:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.382+0000 7f6008e84640 1 librados: shutdown 2024-04-19T00:46:56.428 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:56.428 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:46:56.428 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.538+0000 7fac287d6640 10 monclient: get_monmap_and_config 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.538+0000 7fac287d6640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient: monmap: 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:56.546444+0000 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:56.546444+0000 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.545 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:56.546 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 Processor -- start 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 -- start start 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient: init 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient: _add_conns ranks=[2,7,0,1,3,8,10,5,6,4,9] 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 0x7fac20136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): picked mon.noname-b con 0x7fac20135d20 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac200a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): picked mon.noname-i con 0x7fac20135860 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac200a3e80 addr v1:172.21.15.43:6789/0 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac200a2400 con 0x7fac200a3e80 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fac200a1780 con 0x7fac20135d20 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fac200a1130 con 0x7fac20135860 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac287d6640 10 monclient(hunting): authenticate will time out at 1086.673753s 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 0x7fac20136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 0x7fac20136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37278/0 (socket says 172.21.15.43:37278) 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26d73640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fac200a3e80 0x7fac2013cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50462/0 (socket says 172.21.15.43:50462) 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 -- 172.21.15.43:0/1374892078 learned_addr learned my addr 172.21.15.43:0/1374892078 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): get_auth_request con 0x7fac20135d20 auth_method 0 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 1 --2- 172.21.15.43:0/1374892078 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac200a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): get_auth_request con 0x7fac20135860 auth_method 0 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25d71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient(hunting): handle_auth_done global_id 22646 payload 274 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25570640 1 -- 172.21.15.43:0/1374892078 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611892760 0 0) 0x7fac200a2400 con 0x7fac200a3e80 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 -- 172.21.15.43:0/1374892078 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 msgr2=0x7fac200a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 --2- 172.21.15.43:0/1374892078 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac200a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 -- 172.21.15.43:0/1374892078 >> v1:172.21.15.43:6789/0 conn(0x7fac200a3e80 legacy=0x7fac2013cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 monclient: found mon.noname-b 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac26572640 1 -- 172.21.15.43:0/1374892078 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac200a1410 con 0x7fac20135d20 2024-04-19T00:46:56.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.542+0000 7fac25570640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:46:56.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26572640 1 --2- 172.21.15.43:0/1374892078 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 0x7fac20136100 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fac18009e80 tx=0x7fac1801d400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad3ff719cff27a6b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 1 -- 172.21.15.43:0/1374892078 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac180090d0 con 0x7fac20135d20 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: dump: 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.559 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 5 monclient: authenticate success, global_id 22646 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 1 -- 172.21.15.43:0/1374892078 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac180a5bb0 con 0x7fac20135d20 2024-04-19T00:46:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 4 set_mon_vals no callback set 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 1 -- 172.21.15.43:0/1374892078 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac180a5d60 con 0x7fac20135d20 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25570640 10 monclient: dump: 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: shutdown 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- 172.21.15.43:0/1374892078 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 msgr2=0x7fac20136100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 --2- 172.21.15.43:0/1374892078 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 0x7fac20136100 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fac18009e80 tx=0x7fac1801d400 comp rx=0 tx=0).stop 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- 172.21.15.43:0/1374892078 shutdown_connections 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 --2- 172.21.15.43:0/1374892078 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac200a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 --2- 172.21.15.43:0/1374892078 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fac20135d20 0x7fac20136100 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- 172.21.15.43:0/1374892078 >> 172.21.15.43:0/1374892078 conn(0x7fac200a85c0 msgr2=0x7fac200a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- 172.21.15.43:0/1374892078 shutdown_connections 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- 172.21.15.43:0/1374892078 wait complete. 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: monmap: 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 librados: starting msgr at 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 librados: starting objecter 2024-04-19T00:46:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 Processor -- start 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- start start 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 librados: setting wanted keys 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 librados: calling monclient init 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: init 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient: _add_conns ranks=[3,7,4,5,6,0,2,1,8] 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac200a3e80 0x7fac20134470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): picked mon.noname-g con 0x7fac200a3e80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac201349b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): picked mon.noname-i con 0x7fac20135860 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 0x7fac2011e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): picked mon.noname-c con 0x7fac20135d20 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fac20073250 con 0x7fac200a3e80 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fac200733d0 con 0x7fac20135d20 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fac20140ad0 con 0x7fac20135860 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac287d6640 10 monclient(hunting): authenticate will time out at 1086.679123s 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 0x7fac2011e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 0x7fac2011e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42894/0 (socket says 172.21.15.43:42894) 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 -- 172.21.15.43:0/2200173309 learned_addr learned my addr 172.21.15.43:0/2200173309 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): get_auth_request con 0x7fac20135d20 auth_method 0 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25d71640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac201349b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26572640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac200a3e80 0x7fac20134470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25d71640 10 monclient(hunting): get_auth_request con 0x7fac20135860 auth_method 0 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25d71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25d71640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac25d71640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26572640 10 monclient(hunting): get_auth_request con 0x7fac200a3e80 auth_method 0 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26572640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26572640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient(hunting): handle_auth_done global_id 22504 payload 824 2024-04-19T00:46:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 msgr2=0x7fac201349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac20135860 0x7fac201349b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac200a3e80 msgr2=0x7fac20134470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac200a3e80 0x7fac20134470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 monclient: found mon.noname-c 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2011fb40 con 0x7fac20135d20 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac26d73640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 0x7fac2011e580 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c410 tx=0x7fac1c00c8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac16ffd640 10 client.?.objecter ms_handle_connect 0x7fac20135d20 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.546+0000 7fac16ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac1c0047f0 con 0x7fac20135d20 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:56.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: dump: 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.569 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1c0049f0 con 0x7fac20135d20 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac1c004ba0 con 0x7fac20135d20 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 5 monclient: authenticate success, global_id 22504 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 monclient: _renew_subs 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac20121e30 con 0x7fac20135d20 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac24d6f640 4 set_mon_vals no callback set 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: dump: 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.571 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 client.22504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 monclient: _renew_subs 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fac20121b50 con 0x7fac20135d20 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 1 librados: init done 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 0x7fac2011eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fac200a55d0 con 0x7fac201c5e20 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fac1c011070 con 0x7fac20135d20 2024-04-19T00:46:56.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 0x7fac2011eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient: get_auth_request con 0x7fac201c5e20 auth_method 0 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac04038e40 0x7fac0403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(101..101 src has 1..101) ==== 7725+0+0 (secure 0 0 0) 0x7fac1c049530 con 0x7fac20135d20 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 10 monclient(hunting): handle_auth_done global_id 22292 payload 824 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 client.22504.objecter ms_dispatch 0x7fac20000ba0 osd_map(101..101 src has 1..101) 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 3 client.22504.objecter handle_osd_map got epochs [101,101] > 0 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 3 client.22504.objecter handle_osd_map decoding full epoch 101 2024-04-19T00:46:56.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac201203f0 con 0x7fac201c5e20 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 20 client.22504.objecter dump_active .. 0 homeless 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac26572640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 0x7fac2011eac0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fac2009f4a0 tx=0x7fac18019740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 client.22504.objecter ms_handle_connect 0x7fac201c5e20 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 client.22504.objecter resend_mon_ops 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac25d71640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac04038e40 0x7fac0403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac25d71640 10 monclient: get_auth_request con 0x7fac04038e40 auth_method 0 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac18009510 con 0x7fac201c5e20 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: dump: 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.574 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac180039a0 con 0x7fac201c5e20 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 msgr2=0x7fac2011eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 0x7fac2011eac0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fac2009f4a0 tx=0x7fac18019740 comp rx=0 tx=0).stop 2024-04-19T00:46:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 msgr2=0x7fac2011eac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac201c5e20 0x7fac2011eac0 secure :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fac2009f4a0 tx=0x7fac18019740 comp rx=0 tx=0).stop 2024-04-19T00:46:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac16ffd640 10 client.22504.objecter ms_handle_connect 0x7fac04038e40 2024-04-19T00:46:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.550+0000 7fac25d71640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac04038e40 0x7fac0403b300 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7fac2009e570 tx=0x7fac10008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 0x7fac20070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fac2006d140 con 0x7fac2011f000 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 0x7fac20070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient: get_auth_request con 0x7fac2011f000 auth_method 0 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 10 monclient(hunting): handle_auth_done global_id 22301 payload 824 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 1 -- 172.21.15.43:0/2200173309 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac2011f410 con 0x7fac2011f000 2024-04-19T00:46:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26572640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 0x7fac20070bc0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fac1801d440 tx=0x7fac180a7070 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 client.22504.objecter ms_handle_connect 0x7fac2011f000 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 client.22504.objecter resend_mon_ops 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac180c1bc0 con 0x7fac2011f000 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 monclient: dump: 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fac180c1750 con 0x7fac2011f000 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 msgr2=0x7fac20070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 0x7fac20070bc0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fac1801d440 tx=0x7fac180a7070 comp rx=0 tx=0).stop 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 msgr2=0x7fac20070bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac16ffd640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac2011f000 0x7fac20070bc0 secure :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fac1801d440 tx=0x7fac180a7070 comp rx=0 tx=0).stop 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 160070, 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:56.581 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.582 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.583 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:56.584 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.585 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:56.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.589 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:56.590 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:56.604 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 10 librados: watch_flush enter 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 10 librados: watch_flush exit 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 20 client.22504.objecter shutdown clearing up homeless session... 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 10 client.22504.objecter successfully canceled tick 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac04038e40 msgr2=0x7fac0403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac04038e40 0x7fac0403b300 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7fac2009e570 tx=0x7fac10008040 comp rx=0 tx=0).stop 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 10 monclient: shutdown 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 msgr2=0x7fac2011e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac26d73640 1 -- 172.21.15.43:0/2200173309 reap_dead start 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 0x7fac2011e580 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fac1c00c410 tx=0x7fac1c00c8d0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 shutdown_connections 2024-04-19T00:46:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 --2- 172.21.15.43:0/2200173309 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac20135d20 0x7fac2011e580 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 >> 172.21.15.43:0/2200173309 conn(0x7fac200a85c0 msgr2=0x7fac20070020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 shutdown_connections 2024-04-19T00:46:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 -- 172.21.15.43:0/2200173309 wait complete. 2024-04-19T00:46:56.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.558+0000 7fac287d6640 1 librados: shutdown 2024-04-19T00:46:56.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:56.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:46:56.607 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:46:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.734+0000 7f8d6d6be640 10 monclient: get_monmap_and_config 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.734+0000 7f8d6d6be640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.734+0000 7f8d6d6be640 10 monclient: monmap: 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:56.742370+0000 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:56.742370+0000 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.742 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:56.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 Processor -- start 2024-04-19T00:46:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 -- start start 2024-04-19T00:46:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient: init 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient: _add_conns ranks=[1,3,8,2,0,9,10,5,6,7,4] 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8d68138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d6813e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): picked mon.noname-g con 0x7f8d680a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 0x7f8d681407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): picked mon.noname-e con 0x7f8d681397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d680a3dc0 con 0x7f8d68138cd0 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8d680a4410 con 0x7f8d680a5b80 2024-04-19T00:46:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8d680a40a0 con 0x7f8d681397c0 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d6d6be640 10 monclient(hunting): authenticate will time out at 1086.869478s 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 0x7f8d681407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 0x7f8d681407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54910/0 (socket says 172.21.15.43:54910) 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 -- 172.21.15.43:0/2060438307 learned_addr learned my addr 172.21.15.43:0/2060438307 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): get_auth_request con 0x7f8d681397c0 auth_method 0 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d667fc640 1 --2- 172.21.15.43:0/2060438307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d6813e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d667fc640 10 monclient(hunting): get_auth_request con 0x7f8d680a5b80 auth_method 0 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d667fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient(hunting): handle_auth_done global_id 22310 payload 274 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 -- 172.21.15.43:0/2060438307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 msgr2=0x7f8d6813e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 --2- 172.21.15.43:0/2060438307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d6813e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 -- 172.21.15.43:0/2060438307 >> v1:172.21.15.161:6789/0 conn(0x7f8d68138cd0 legacy=0x7f8d681390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 monclient: found mon.noname-e 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.738+0000 7f8d677fe640 1 -- 172.21.15.43:0/2060438307 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d680a2ae0 con 0x7f8d681397c0 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 1 --2- 172.21.15.43:0/2060438307 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 0x7f8d681407c0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009e80 tx=0x7f8d6001c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=44b1a31041cb9faf server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 1 -- 172.21.15.43:0/2060438307 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d600090d0 con 0x7f8d681397c0 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: dump: 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 1 -- 172.21.15.43:0/2060438307 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d600a5b10 con 0x7f8d681397c0 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 1 -- 172.21.15.43:0/2060438307 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d600a5d00 con 0x7f8d681397c0 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 5 monclient: authenticate success, global_id 22310 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 4 set_mon_vals no callback set 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d65ffb640 10 monclient: dump: 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: shutdown 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- 172.21.15.43:0/2060438307 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 msgr2=0x7f8d681407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/2060438307 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 0x7f8d681407c0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009e80 tx=0x7f8d6001c910 comp rx=0 tx=0).stop 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- 172.21.15.43:0/2060438307 shutdown_connections 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/2060438307 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8d681397c0 0x7f8d681407c0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/2060438307 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d6813e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- 172.21.15.43:0/2060438307 >> 172.21.15.43:0/2060438307 conn(0x7f8d680ab2f0 msgr2=0x7f8d680ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- 172.21.15.43:0/2060438307 shutdown_connections 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- 172.21.15.43:0/2060438307 wait complete. 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: monmap: 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 librados: starting msgr at 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 librados: starting objecter 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 Processor -- start 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- start start 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 librados: setting wanted keys 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 librados: calling monclient init 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: init 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient: _add_conns ranks=[3,2,0,7,8,4,6,5,1] 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d680a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): picked mon.noname-g con 0x7f8d680a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8d68138cd0 0x7f8d681358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): picked mon.noname-b con 0x7f8d68138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d681397c0 0x7f8d68135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): picked mon.noname-a con 0x7f8d681397c0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8d68143730 con 0x7f8d681397c0 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8d680a20a0 con 0x7f8d68138cd0 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8d681438b0 con 0x7f8d680a5b80 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d6d6be640 10 monclient(hunting): authenticate will time out at 1086.875605s 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8d68138cd0 0x7f8d681358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d681397c0 0x7f8d68135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8d68138cd0 0x7f8d681358b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37292/0 (socket says 172.21.15.43:37292) 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 1 -- 172.21.15.43:0/3904463292 learned_addr learned my addr 172.21.15.43:0/3904463292 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): get_auth_request con 0x7f8d68138cd0 auth_method 0 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 10 monclient(hunting): get_auth_request con 0x7f8d681397c0 auth_method 0 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d66ffd640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d680a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.742+0000 7f8d677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): get_auth_request con 0x7f8d680a5b80 auth_method 0 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 10 monclient(hunting): handle_auth_done global_id 22655 payload 824 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 msgr2=0x7f8d680a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8d680a5b80 0x7f8d680a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d681397c0 msgr2=0x7f8d68135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d681397c0 0x7f8d68135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 monclient: found mon.noname-b 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d680a3590 con 0x7f8d68138cd0 2024-04-19T00:46:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d677fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8d681397c0 0x7f8d68135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d667fc640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8d68138cd0 0x7f8d681358b0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00d800 tx=0x7f8d5c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d68138cd0 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d5c0044a0 con 0x7f8d68138cd0 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: dump: 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.772 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.773 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5c00b6c0 con 0x7f8d68138cd0 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d5c00bce0 con 0x7f8d68138cd0 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: dump: 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.775 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d64ff9640 4 set_mon_vals no callback set 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 5 monclient: authenticate success, global_id 22655 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 monclient: _renew_subs 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d680a3040 con 0x7f8d68138cd0 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 client.22655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:56.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 monclient: _renew_subs 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d6809ef40 con 0x7f8d68138cd0 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 1 librados: init done 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 0x7f8d6809d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6806d140 con 0x7f8d68136330 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8d5c00b870 con 0x7f8d68138cd0 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 0x7f8d6809d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient: get_auth_request con 0x7f8d68136330 auth_method 0 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8d54038e40 0x7f8d5403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 8603+0+0 (secure 0 0 0) 0x7f8d5c00f040 con 0x7f8d68138cd0 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 client.22655.objecter ms_dispatch 0x7f8d68000ba0 osd_map(102..102 src has 1..102) 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 3 client.22655.objecter handle_osd_map got epochs [102,102] > 0 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 3 client.22655.objecter handle_osd_map decoding full epoch 102 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 20 client.22655.objecter dump_active .. 0 homeless 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_done global_id 23257 payload 824 2024-04-19T00:46:56.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d677fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8d54038e40 0x7f8d5403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d68136710 con 0x7f8d68136330 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d677fe640 10 monclient: get_auth_request con 0x7f8d54038e40 auth_method 0 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 client.22655.objecter ms_handle_connect 0x7f8d68136330 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 client.22655.objecter resend_mon_ops 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d66ffd640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 0x7f8d6809d050 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8d680a72e0 tx=0x7f8d5800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d58019070 con 0x7f8d68136330 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: dump: 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.779 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.780 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 client.22655.objecter ms_handle_connect 0x7f8d54038e40 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d677fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8d54038e40 0x7f8d5403b300 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f8d600ae000 tx=0x7f8d600a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d58005110 con 0x7f8d68136330 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 msgr2=0x7f8d6809d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 0x7f8d6809d050 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8d680a72e0 tx=0x7f8d5800fc00 comp rx=0 tx=0).stop 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 msgr2=0x7f8d6809d050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.746+0000 7f8d4f7fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d68136330 0x7f8d6809d050 secure :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8d680a72e0 tx=0x7f8d5800fc00 comp rx=0 tx=0).stop 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d6d6be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 0x7f8d6802c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d6d6be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8d68068240 con 0x7f8d6802c450 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 0x7f8d6802c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient: get_auth_request con 0x7f8d6802c450 auth_method 0 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 10 monclient(hunting): handle_auth_done global_id 23266 payload 824 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 1 -- 172.21.15.43:0/3904463292 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d680693b0 con 0x7f8d6802c450 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d4f7fe640 10 client.22655.objecter ms_handle_connect 0x7f8d6802c450 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d4f7fe640 10 client.22655.objecter resend_mon_ops 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d66ffd640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 0x7f8d6802c860 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8d680a72e0 tx=0x7f8d58000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8d58019030 con 0x7f8d6802c450 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d4f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.754+0000 7f8d4f7fe640 10 monclient: dump: 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.785 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 160228, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.792 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.794 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:56.795 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.796 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.800 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.802 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.803 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.805 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:56.806 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.807 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.809 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.811 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.812 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:56.813 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f8d5801a040 con 0x7f8d6802c450 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 msgr2=0x7f8d6802c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 0x7f8d6802c860 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8d680a72e0 tx=0x7f8d58000ee0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 msgr2=0x7f8d6802c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d4f7fe640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8d6802c450 0x7f8d6802c860 secure :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f8d680a72e0 tx=0x7f8d58000ee0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 10 librados: watch_flush enter 2024-04-19T00:46:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 10 librados: watch_flush exit 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 20 client.22655.objecter shutdown clearing up homeless session... 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 10 client.22655.objecter successfully canceled tick 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8d54038e40 msgr2=0x7f8d5403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8d54038e40 0x7f8d5403b300 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f8d600ae000 tx=0x7f8d600a7070 comp rx=0 tx=0).stop 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 10 monclient: shutdown 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8d68138cd0 msgr2=0x7f8d681358b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 --2- 172.21.15.43:0/3904463292 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8d68138cd0 0x7f8d681358b0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c00d800 tx=0x7f8d5c00dcc0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d677fe640 1 -- 172.21.15.43:0/3904463292 reap_dead start 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d677fe640 1 -- 172.21.15.43:0/3904463292 reap_dead start 2024-04-19T00:46:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 shutdown_connections 2024-04-19T00:46:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 >> 172.21.15.43:0/3904463292 conn(0x7f8d680ab2f0 msgr2=0x7f8d681373d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 shutdown_connections 2024-04-19T00:46:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 -- 172.21.15.43:0/3904463292 wait complete. 2024-04-19T00:46:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.758+0000 7f8d6d6be640 1 librados: shutdown 2024-04-19T00:46:56.818 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:56.818 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:46:56.818 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient: get_monmap_and_config 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient: monmap: 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:56.942626+0000 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:56.942626+0000 2024-04-19T00:46:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:56.943 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 Processor -- start 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 -- start start 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient: init 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient: _add_conns ranks=[6,7,0,5,4,10,9,1,8,3,2] 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1e30138cd0 0x7f1e301390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): picked mon.noname-d con 0x7f1e30138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1e300a5b80 0x7f1e301395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): picked mon.noname-i con 0x7f1e300a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e301358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e300a3dc0 con 0x7f1e301358b0 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1e300a4410 con 0x7f1e30138cd0 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1e30139b30 con 0x7f1e300a5b80 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e36e8b640 10 monclient(hunting): authenticate will time out at 1087.069818s 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e35428640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1e301358b0 0x7f1e30135d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50464/0 (socket says 172.21.15.43:50464) 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1e30138cd0 0x7f1e301390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e35428640 1 -- 172.21.15.43:0/1523308835 learned_addr learned my addr 172.21.15.43:0/1523308835 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): get_auth_request con 0x7f1e30138cd0 auth_method 0 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1e300a5b80 0x7f1e301395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request con 0x7f1e300a5b80 auth_method 0 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient(hunting): handle_auth_done global_id 22704 payload 274 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2f7fe640 1 -- 172.21.15.43:0/1523308835 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1505519292 0 0) 0x7f1e300a3dc0 con 0x7f1e301358b0 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 1 -- 172.21.15.43:0/1523308835 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1e300a5b80 msgr2=0x7f1e301395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1e300a5b80 0x7f1e301395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 1 -- 172.21.15.43:0/1523308835 >> v1:172.21.15.43:6789/0 conn(0x7f1e301358b0 legacy=0x7f1e30135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 1 monclient: found mon.noname-d 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e34c27640 1 -- 172.21.15.43:0/1523308835 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e300a40a0 con 0x7f1e30138cd0 2024-04-19T00:46:56.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:46:56.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.938+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1e300a5b80 0x7f1e301395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e34c27640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1e30138cd0 0x7f1e301390b0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f1e200097f0 tx=0x7f1e2001c340 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ac5596e71cf891bc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 1 -- 172.21.15.43:0/1523308835 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e200b0070 con 0x7f1e30138cd0 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: dump: 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.957 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 1 -- 172.21.15.43:0/1523308835 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e200aa810 con 0x7f1e30138cd0 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 1 -- 172.21.15.43:0/1523308835 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e200a9300 con 0x7f1e30138cd0 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e2f7fe640 10 monclient: dump: 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.959 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 5 monclient: authenticate success, global_id 22704 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 4 set_mon_vals no callback set 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 10 monclient: shutdown 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1523308835 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1e30138cd0 msgr2=0x7f1e301390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1e30138cd0 0x7f1e301390b0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f1e200097f0 tx=0x7f1e2001c340 comp rx=0 tx=0).stop 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1523308835 shutdown_connections 2024-04-19T00:46:56.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1e300a5b80 0x7f1e301395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1523308835 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1e30138cd0 0x7f1e301390b0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1523308835 >> 172.21.15.43:0/1523308835 conn(0x7f1e300ab2f0 msgr2=0x7f1e300ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1523308835 shutdown_connections 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1523308835 wait complete. 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 10 monclient: build_initial_monmap 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.942+0000 7f1e36e8b640 10 monclient: monmap: 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:56.961 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 librados: starting msgr at 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 librados: starting objecter 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 Processor -- start 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- start start 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 librados: setting wanted keys 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 librados: calling monclient init 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: init 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: _add_conns ranks=[2,3,5,6,4,1,0,7,8] 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 0x7f1e3012aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): picked mon.noname-b con 0x7f1e300a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e301358b0 0x7f1e3012b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): picked mon.noname-g con 0x7f1e301358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:56.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1e30138cd0 0x7f1e301322f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): picked mon.noname-h con 0x7f1e30138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1e30143ab0 con 0x7f1e300a5b80 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1e300a20a0 con 0x7f1e301358b0 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f1e30143c30 con 0x7f1e30138cd0 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): authenticate will time out at 1087.076785s 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 0x7f1e3012aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 0x7f1e3012aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37302/0 (socket says 172.21.15.43:37302) 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 -- 172.21.15.43:0/1730656531 learned_addr learned my addr 172.21.15.43:0/1730656531 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): get_auth_request con 0x7f1e300a5b80 auth_method 0 2024-04-19T00:46:56.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1e30138cd0 0x7f1e301322f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 10 monclient(hunting): get_auth_request con 0x7f1e30138cd0 auth_method 0 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e301358b0 0x7f1e3012b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient(hunting): handle_auth_done global_id 22664 payload 824 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1e30138cd0 msgr2=0x7f1e301322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1e30138cd0 0x7f1e301322f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e301358b0 msgr2=0x7f1e3012b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e301358b0 0x7f1e3012b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient: _finish_hunting 0 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 monclient: found mon.noname-b 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e300a3590 con 0x7f1e300a5b80 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient: get_auth_request con 0x7f1e301358b0 auth_method 0 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e301358b0 0x7f1e3012b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e34c27640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 0x7f1e3012aac0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1e2001cbe0 tx=0x7f1e20019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 client.?.objecter ms_handle_connect 0x7f1e300a5b80 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e200b0070 con 0x7f1e300a5b80 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: dump: 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e200ba070 con 0x7f1e300a5b80 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e200044b0 con 0x7f1e300a5b80 2024-04-19T00:46:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 5 monclient: authenticate success, global_id 22664 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: _renew_subs 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e300a3040 con 0x7f1e300a5b80 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2effd640 4 set_mon_vals no callback set 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 monclient: dump: 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.969 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 client.22664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: _renew_subs 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e3009bdf0 con 0x7f1e300a5b80 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 librados: init done 2024-04-19T00:46:56.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1e20004040 con 0x7f1e300a5b80 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 0x7f1e30132c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1e3006d140 con 0x7f1e30132830 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1e10038e10 0x7f1e1003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 8603+0+0 (secure 0 0 0) 0x7f1e200e3ef0 con 0x7f1e300a5b80 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 client.22664.objecter ms_dispatch 0x7f1e30000ba0 osd_map(102..102 src has 1..102) 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 3 client.22664.objecter handle_osd_map got epochs [102,102] > 0 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 3 client.22664.objecter handle_osd_map decoding full epoch 102 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 20 client.22664.objecter dump_active .. 0 homeless 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 0x7f1e30132c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient: get_auth_request con 0x7f1e30132830 auth_method 0 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1e10038e10 0x7f1e1003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 10 monclient: get_auth_request con 0x7f1e10038e10 auth_method 0 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_done global_id 18417 payload 824 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2ffff640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e300693b0 con 0x7f1e30132830 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e35428640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1e10038e10 0x7f1e1003b2d0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f1e30138ca0 tx=0x7f1e28007ad0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.946+0000 7f1e2d7fa640 10 client.22664.objecter ms_handle_connect 0x7f1e10038e10 2024-04-19T00:46:56.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 client.22664.objecter ms_handle_connect 0x7f1e30132830 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 client.22664.objecter resend_mon_ops 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 0x7f1e30132c10 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1e3012d850 tx=0x7f1e1800e5d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e18015880 con 0x7f1e30132830 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 monclient: dump: 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.973 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1e1801a070 con 0x7f1e30132830 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 msgr2=0x7f1e30132c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 0x7f1e30132c10 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1e3012d850 tx=0x7f1e1800e5d0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 msgr2=0x7f1e30132c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.950+0000 7f1e2d7fa640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30132830 0x7f1e30132c10 secure :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f1e3012d850 tx=0x7f1e1800e5d0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e36e8b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:56.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 0x7f1e30137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:56.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e36e8b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:56.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1e30068240 con 0x7f1e30133150 2024-04-19T00:46:56.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 0x7f1e30137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:56.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient: get_auth_request con 0x7f1e30133150 auth_method 0 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.954+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2ffff640 10 monclient(hunting): handle_auth_done global_id 18426 payload 824 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2ffff640 1 -- 172.21.15.43:0/1730656531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1e301344b0 con 0x7f1e30133150 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 client.22664.objecter ms_handle_connect 0x7f1e30133150 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 client.22664.objecter resend_mon_ops 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2ffff640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 0x7f1e30137850 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1e18007fd0 tx=0x7f1e18004a50 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1e1801f0f0 con 0x7f1e30133150 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 monclient: dump: 2024-04-19T00:46:56.977 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:56.978 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f1e1801f0f0 con 0x7f1e30133150 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 msgr2=0x7f1e30137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 0x7f1e30137850 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f1e18007fd0 tx=0x7f1e18004a50 comp rx=0 tx=0).stop 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 msgr2=0x7f1e30137850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e2d7fa640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1e30133150 0x7f1e30137850 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:56.979 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 160440, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:56.980 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.981 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.982 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:56.983 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:56.984 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.986 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:56.989 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:56.991 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.992 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.993 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.994 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.996 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:56.998 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:56.999 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.000 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:57.001 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 10 librados: watch_flush enter 2024-04-19T00:46:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 10 librados: watch_flush exit 2024-04-19T00:46:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 20 client.22664.objecter shutdown clearing up homeless session... 2024-04-19T00:46:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 10 client.22664.objecter successfully canceled tick 2024-04-19T00:46:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1e10038e10 msgr2=0x7f1e1003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1e10038e10 0x7f1e1003b2d0 secure :-1 s=READY pgs=3334 cs=0 l=1 rev1=1 crypto rx=0x7f1e30138ca0 tx=0x7f1e28007ad0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 10 monclient: shutdown 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 msgr2=0x7f1e3012aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 0x7f1e3012aac0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f1e2001cbe0 tx=0x7f1e20019140 comp rx=0 tx=0).stop 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e35428640 1 -- 172.21.15.43:0/1730656531 reap_dead start 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 shutdown_connections 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 --2- 172.21.15.43:0/1730656531 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1e300a5b80 0x7f1e3012aac0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 >> 172.21.15.43:0/1730656531 conn(0x7f1e300ab2f0 msgr2=0x7f1e300a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 shutdown_connections 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 -- 172.21.15.43:0/1730656531 wait complete. 2024-04-19T00:46:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:56.958+0000 7f1e36e8b640 1 librados: shutdown 2024-04-19T00:46:57.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:57.005 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:46:57.005 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:46:57.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.126+0000 7fa4f481c640 10 monclient: get_monmap_and_config 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.126+0000 7fa4f481c640 10 monclient: build_initial_monmap 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.126+0000 7fa4f481c640 10 monclient: monmap: 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:57.134227+0000 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:57.134227+0000 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:57.133 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:57.134 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 Processor -- start 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 -- start start 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient: init 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient: _add_conns ranks=[9,2,0,10,4,3,1,5,7,6,8] 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa4f00a13b0 0x7fa4f0136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f00a13b0 addr v2:172.21.15.43:3300/0 2024-04-19T00:46:57.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f00a1790 0x7fa4f00a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f00a1790 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4f00a6800 addr v1:172.21.15.43:6789/0 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f0137310 con 0x7fa4f00a6800 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fa4f00a52a0 con 0x7fa4f00a13b0 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f00a4be0 con 0x7fa4f00a1790 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4f481c640 10 monclient(hunting): authenticate will time out at 1087.261535s 2024-04-19T00:46:57.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa4f00a13b0 0x7fa4f0136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4eed9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa4f00a6800 0x7fa4f00a6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50478/0 (socket says 172.21.15.43:50478) 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f00a1790 0x7fa4f00a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4eed9d640 1 -- 172.21.15.43:0/1440251089 learned_addr learned my addr 172.21.15.43:0/1440251089 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): get_auth_request con 0x7fa4f00a13b0 auth_method 0 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request con 0x7fa4f00a1790 auth_method 0 2024-04-19T00:46:57.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_done global_id 22673 payload 274 2024-04-19T00:46:57.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 1 -- 172.21.15.43:0/1440251089 >> v2:172.21.15.43:3300/0 conn(0x7fa4f00a13b0 msgr2=0x7fa4f0136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 1 --2- 172.21.15.43:0/1440251089 >> v2:172.21.15.43:3300/0 conn(0x7fa4f00a13b0 0x7fa4f0136dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 1 -- 172.21.15.43:0/1440251089 >> v1:172.21.15.43:6789/0 conn(0x7fa4f00a6800 legacy=0x7fa4f00a6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient: _finish_hunting 0 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 1 monclient: found mon.noname-b 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ed59a640 1 -- 172.21.15.43:0/1440251089 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819665451 0 0) 0x7fa4f0137310 con 0x7fa4f00a6800 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4edd9b640 1 -- 172.21.15.43:0/1440251089 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f0070380 con 0x7fa4f00a1790 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ed59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.130+0000 7fa4ee59c640 1 --2- 172.21.15.43:0/1440251089 >> v2:172.21.15.43:3300/0 conn(0x7fa4f00a13b0 0x7fa4f0136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 --2- 172.21.15.43:0/1440251089 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f00a1790 0x7fa4f00a1bf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80097f0 tx=0x7fa4d801c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bfb171e536b11bc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 1 -- 172.21.15.43:0/1440251089 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4d80b0070 con 0x7fa4f00a1790 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: dump: 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.144 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: _finish_auth 0 2024-04-19T00:46:57.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: _check_auth_tickets 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 1 -- 172.21.15.43:0/1440251089 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d80aa810 con 0x7fa4f00a1790 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 1 -- 172.21.15.43:0/1440251089 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4d80aae30 con 0x7fa4f00a1790 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 5 monclient: authenticate success, global_id 22673 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 4 set_mon_vals no callback set 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ed59a640 10 monclient: dump: 2024-04-19T00:46:57.146 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.147 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: shutdown 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- 172.21.15.43:0/1440251089 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f00a1790 msgr2=0x7fa4f00a1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 --2- 172.21.15.43:0/1440251089 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f00a1790 0x7fa4f00a1bf0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80097f0 tx=0x7fa4d801c330 comp rx=0 tx=0).stop 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- 172.21.15.43:0/1440251089 shutdown_connections 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 --2- 172.21.15.43:0/1440251089 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa4f00a1790 0x7fa4f00a1bf0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 --2- 172.21.15.43:0/1440251089 >> v2:172.21.15.43:3300/0 conn(0x7fa4f00a13b0 0x7fa4f0136dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- 172.21.15.43:0/1440251089 >> 172.21.15.43:0/1440251089 conn(0x7fa4f00a7ef0 msgr2=0x7fa4f00a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- 172.21.15.43:0/1440251089 shutdown_connections 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- 172.21.15.43:0/1440251089 wait complete. 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: build_initial_monmap 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: monmap: 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:57.149 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 librados: starting msgr at 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 librados: starting objecter 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 Processor -- start 2024-04-19T00:46:57.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- start start 2024-04-19T00:46:57.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 librados: setting wanted keys 2024-04-19T00:46:57.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 librados: calling monclient init 2024-04-19T00:46:57.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: init 2024-04-19T00:46:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient: _add_conns ranks=[8,4,6,1,0,3,5,2,7] 2024-04-19T00:46:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4f00a13b0 0x7fa4f01d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): picked mon.noname-e con 0x7fa4f00a13b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4f00a1790 0x7fa4f01d0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f00a1790 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa4f00a6800 0x7fa4f01d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): picked mon.noname-d con 0x7fa4f00a6800 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa4f01415f0 con 0x7fa4f00a1790 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa4f0141470 con 0x7fa4f00a6800 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa4f0141770 con 0x7fa4f00a13b0 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4f481c640 10 monclient(hunting): authenticate will time out at 1087.266944s 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4f00a13b0 0x7fa4f01d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4f00a1790 0x7fa4f01d0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa4f00a6800 0x7fa4f01d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4f00a13b0 0x7fa4f01d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54912/0 (socket says 172.21.15.43:54912) 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 1 -- 172.21.15.43:0/3890193784 learned_addr learned my addr 172.21.15.43:0/3890193784 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): get_auth_request con 0x7fa4f00a13b0 auth_method 0 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request con 0x7fa4f00a1790 auth_method 0 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): get_auth_request con 0x7fa4f00a6800 auth_method 0 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient(hunting): handle_auth_done global_id 22513 payload 824 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4f00a13b0 msgr2=0x7fa4f01d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4f00a13b0 0x7fa4f01d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa4f00a6800 msgr2=0x7fa4f01d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa4f00a6800 0x7fa4f01d14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient: _finish_hunting 0 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 monclient: found mon.noname-c 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f0072720 con 0x7fa4f00a1790 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4ee59c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4f00a13b0 0x7fa4f01d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4eed9d640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa4f00a6800 0x7fa4f01d14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:57.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4edd9b640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4f00a1790 0x7fa4f01d0f60 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80ab000 tx=0x7fa4d80190d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4e6ffd640 10 client.?.objecter ms_handle_connect 0x7fa4f00a1790 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4e6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4d80b0040 con 0x7fa4f00a1790 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.134+0000 7fa4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: dump: 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:46:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4d80b9070 con 0x7fa4f00a1790 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 5 monclient: authenticate success, global_id 22513 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 monclient: _renew_subs 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f00714a0 con 0x7fa4f00a1790 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4d80044b0 con 0x7fa4f00a1790 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e7fff640 4 set_mon_vals no callback set 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: dump: 2024-04-19T00:46:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 client.22513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 monclient: _renew_subs 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f0071a60 con 0x7fa4f00a1790 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 1 librados: init done 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 0x7fa4f0125fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa4f006d140 con 0x7fa4f0130f60 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa4d80aac80 con 0x7fa4f00a1790 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 0x7fa4f0125fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient: get_auth_request con 0x7fa4f0130f60 auth_method 0 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa4d0038e10 0x7fa4d003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(102..102 src has 1..102) ==== 8603+0+0 (secure 0 0 0) 0x7fa4d80e38d0 con 0x7fa4f00a1790 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 client.22513.objecter ms_dispatch 0x7fa4f0000ba0 osd_map(102..102 src has 1..102) 2024-04-19T00:46:57.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 3 client.22513.objecter handle_osd_map got epochs [102,102] > 0 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 3 client.22513.objecter handle_osd_map decoding full epoch 102 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 20 client.22513.objecter dump_active .. 0 homeless 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4eed9d640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa4d0038e10 0x7fa4d003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4eed9d640 10 monclient: get_auth_request con 0x7fa4d0038e10 auth_method 0 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_done global_id 44213 payload 824 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4f006fae0 con 0x7fa4f0130f60 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 client.22513.objecter ms_handle_connect 0x7fa4f0130f60 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 client.22513.objecter resend_mon_ops 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4ee59c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 0x7fa4f0125fc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0004980 tx=0x7fa4e000e8c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 client.22513.objecter ms_handle_connect 0x7fa4d0038e10 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4eed9d640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa4d0038e10 0x7fa4d003b2d0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8007bc0 tx=0x7fa4e8007b10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4e0019070 con 0x7fa4f0130f60 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:46:57.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: dump: 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.173 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa4e0015720 con 0x7fa4f0130f60 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 msgr2=0x7fa4f0125fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 0x7fa4f0125fc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0004980 tx=0x7fa4e000e8c0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 msgr2=0x7fa4f0125fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.138+0000 7fa4e6ffd640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0130f60 0x7fa4f0125fc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4f481c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4f481c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 0x7fa4f0126910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4f481c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa4f0126e50 con 0x7fa4f0126500 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 0x7fa4f0126910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient: get_auth_request con 0x7fa4f0126500 auth_method 0 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 10 monclient(hunting): handle_auth_done global_id 44222 payload 824 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 1 -- 172.21.15.43:0/3890193784 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4f00a4f60 con 0x7fa4f0126500 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 client.22513.objecter ms_handle_connect 0x7fa4f0126500 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 client.22513.objecter resend_mon_ops 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4ee59c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 0x7fa4f0126910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000f90 tx=0x7fa4e0004ac0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4e001ee20 con 0x7fa4f0126500 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 monclient: dump: 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.177 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fa4e001ee20 con 0x7fa4f0126500 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 msgr2=0x7fa4f0126910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 0x7fa4f0126910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000f90 tx=0x7fa4e0004ac0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 msgr2=0x7fa4f0126910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.146+0000 7fa4e6ffd640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa4f0126500 0x7fa4f0126910 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000f90 tx=0x7fa4e0004ac0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.178 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16879, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:57.179 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:57.180 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:57.181 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:57.182 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:57.183 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:57.184 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.185 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:57.186 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:57.187 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.188 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:57.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.193 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:57.194 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.195 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:57.196 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.198 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.200 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.202 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.203 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:57.204 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:57.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 10 librados: watch_flush enter 2024-04-19T00:46:57.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 10 librados: watch_flush exit 2024-04-19T00:46:57.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 20 client.22513.objecter shutdown clearing up homeless session... 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 10 client.22513.objecter successfully canceled tick 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa4d0038e10 msgr2=0x7fa4d003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa4d0038e10 0x7fa4d003b2d0 secure :-1 s=READY pgs=3335 cs=0 l=1 rev1=1 crypto rx=0x7fa4e8007bc0 tx=0x7fa4e8007b10 comp rx=0 tx=0).stop 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 10 monclient: shutdown 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4f00a1790 msgr2=0x7fa4f01d0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 --2- 172.21.15.43:0/3890193784 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4f00a1790 0x7fa4f01d0f60 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fa4d80ab000 tx=0x7fa4d80190d0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4eed9d640 1 -- 172.21.15.43:0/3890193784 reap_dead start 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4eed9d640 1 -- 172.21.15.43:0/3890193784 reap_dead start 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 shutdown_connections 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 >> 172.21.15.43:0/3890193784 conn(0x7fa4f00a7ef0 msgr2=0x7fa4f00a2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 shutdown_connections 2024-04-19T00:46:57.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 -- 172.21.15.43:0/3890193784 wait complete. 2024-04-19T00:46:57.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.150+0000 7fa4f481c640 1 librados: shutdown 2024-04-19T00:46:57.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:57.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:46:57.208 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient: get_monmap_and_config 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient: build_initial_monmap 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient: monmap: 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:57.334665+0000 2024-04-19T00:46:57.333 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:57.334665+0000 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:57.334 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 Processor -- start 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 -- start start 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient: init 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient: _add_conns ranks=[2,7,4,10,9,5,3,8,1,0,6] 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa89c0a4b10 0x7fa89c1391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): picked mon.noname-b con 0x7fa89c0a4b10 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c0a4ef0 0x7fa89c0a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): picked mon.noname-i con 0x7fa89c0a4ef0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:57.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 0x7fa89c135d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): picked mon.noname-c con 0x7fa89c1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa89c06e750 con 0x7fa89c0a4b10 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa89c06f250 con 0x7fa89c1358b0 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa89c06f530 con 0x7fa89c0a4ef0 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a3271640 10 monclient(hunting): authenticate will time out at 1087.461846s 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 0x7fa89c135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 0x7fa89c135d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42926/0 (socket says 172.21.15.43:42926) 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 -- 172.21.15.43:0/1348181583 learned_addr learned my addr 172.21.15.43:0/1348181583 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): get_auth_request con 0x7fa89c1358b0 auth_method 0 2024-04-19T00:46:57.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa89c0a4b10 0x7fa89c1391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): get_auth_request con 0x7fa89c0a4b10 auth_method 0 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a080c640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c0a4ef0 0x7fa89c0a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a080c640 10 monclient(hunting): get_auth_request con 0x7fa89c0a4ef0 auth_method 0 2024-04-19T00:46:57.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a080c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a080c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a100d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient(hunting): handle_auth_done global_id 22522 payload 274 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 -- 172.21.15.43:0/1348181583 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c0a4ef0 msgr2=0x7fa89c0a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c0a4ef0 0x7fa89c0a5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 -- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa89c0a4b10 msgr2=0x7fa89c1391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa89c0a4b10 0x7fa89c1391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient: _finish_hunting 0 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 monclient: found mon.noname-c 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:57.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a180e640 1 -- 172.21.15.43:0/1348181583 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89c0a58b0 con 0x7fa89c1358b0 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.330+0000 7fa8a080c640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c0a4ef0 0x7fa89c0a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a180e640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 0x7fa89c135d10 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fa8980098a0 tx=0x7fa89801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=18799be74deee22d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 1 -- 172.21.15.43:0/1348181583 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8980b0070 con 0x7fa89c1358b0 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: dump: 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.343 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: _finish_auth 0 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: _check_auth_tickets 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 5 monclient: authenticate success, global_id 22522 2024-04-19T00:46:57.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 1 -- 172.21.15.43:0/1348181583 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8980aa810 con 0x7fa89c1358b0 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 1 -- 172.21.15.43:0/1348181583 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8980a9300 con 0x7fa89c1358b0 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 4 set_mon_vals no callback set 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa893fff640 10 monclient: dump: 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.345 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: shutdown 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 msgr2=0x7fa89c135d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 0x7fa89c135d10 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fa8980098a0 tx=0x7fa89801c440 comp rx=0 tx=0).stop 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- 172.21.15.43:0/1348181583 shutdown_connections 2024-04-19T00:46:57.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c1358b0 0x7fa89c135d10 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c0a4ef0 0x7fa89c0a5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- 172.21.15.43:0/1348181583 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa89c0a4b10 0x7fa89c1391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- 172.21.15.43:0/1348181583 >> 172.21.15.43:0/1348181583 conn(0x7fa89c0a9900 msgr2=0x7fa89c0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- 172.21.15.43:0/1348181583 shutdown_connections 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- 172.21.15.43:0/1348181583 wait complete. 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: build_initial_monmap 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: monmap: 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:57.347 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 librados: starting msgr at 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 librados: starting objecter 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 Processor -- start 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- start start 2024-04-19T00:46:57.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 librados: setting wanted keys 2024-04-19T00:46:57.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 librados: calling monclient init 2024-04-19T00:46:57.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: init 2024-04-19T00:46:57.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:57.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient: _add_conns ranks=[4,0,5,3,8,6,2,1,7] 2024-04-19T00:46:57.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 0x7fa89c137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): picked mon.noname-c con 0x7fa89c0a4b10 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa89c0a4ef0 0x7fa89c0a2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): picked mon.noname-a con 0x7fa89c0a4ef0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa89c1358b0 0x7fa89c0a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): picked mon.noname-h con 0x7fa89c1358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa89c1429a0 con 0x7fa89c0a4ef0 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa89c142b20 con 0x7fa89c0a4b10 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa89c142ca0 con 0x7fa89c1358b0 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a3271640 10 monclient(hunting): authenticate will time out at 1087.467212s 2024-04-19T00:46:57.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 0x7fa89c137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 0x7fa89c137e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42928/0 (socket says 172.21.15.43:42928) 2024-04-19T00:46:57.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 -- 172.21.15.43:0/2385613132 learned_addr learned my addr 172.21.15.43:0/2385613132 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:57.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): get_auth_request con 0x7fa89c0a4b10 auth_method 0 2024-04-19T00:46:57.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa89c0a4ef0 0x7fa89c0a2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): get_auth_request con 0x7fa89c0a4ef0 auth_method 0 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a180e640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa89c1358b0 0x7fa89c0a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient(hunting): handle_auth_done global_id 22531 payload 824 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa89c1358b0 msgr2=0x7fa89c0a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa89c1358b0 0x7fa89c0a2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa89c0a4ef0 msgr2=0x7fa89c0a2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa89c0a4ef0 0x7fa89c0a2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient: _finish_hunting 0 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 monclient: found mon.noname-c 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa89c06e920 con 0x7fa89c0a4b10 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a180e640 10 monclient: get_auth_request con 0x7fa89c1358b0 auth_method 0 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a180e640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa89c1358b0 0x7fa89c0a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa8a100d640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 0x7fa89c137e20 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fa894009d10 tx=0x7fa89400c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa891ffb640 10 client.?.objecter ms_handle_connect 0x7fa89c0a4b10 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.334+0000 7fa891ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa894018020 con 0x7fa89c0a4b10 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:57.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: dump: 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.360 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: _finish_auth 0 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 5 monclient: authenticate success, global_id 22531 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa89400cdb0 con 0x7fa89c0a4b10 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 monclient: _renew_subs 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa89c0a0bf0 con 0x7fa89c0a4b10 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa894014d60 con 0x7fa89c0a4b10 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8937fe640 4 set_mon_vals no callback set 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 client.22531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: dump: 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.362 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 monclient: _renew_subs 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa89c138990 con 0x7fa89c0a4b10 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 1 librados: init done 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 0x7fa89c1d1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa89c134b60 con 0x7fa89c1d0c30 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa89401b020 con 0x7fa89c0a4b10 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 0x7fa89c1d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient: get_auth_request con 0x7fa89c1d0c30 auth_method 0 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa880038e40 0x7fa88003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(102..102 src has 1..102) ==== 8603+0+0 (secure 0 0 0) 0x7fa89404a5b0 con 0x7fa89c0a4b10 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 client.22531.objecter ms_dispatch 0x7fa89c000ba0 osd_map(102..102 src has 1..102) 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 3 client.22531.objecter handle_osd_map got epochs [102,102] > 0 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 3 client.22531.objecter handle_osd_map decoding full epoch 102 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 20 client.22531.objecter dump_active .. 0 homeless 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a180e640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa880038e40 0x7fa88003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a180e640 10 monclient: get_auth_request con 0x7fa880038e40 auth_method 0 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 10 monclient(hunting): handle_auth_done global_id 18853 payload 824 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa89c069630 con 0x7fa89c1d0c30 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 client.22531.objecter ms_handle_connect 0x7fa89c1d0c30 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a080c640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 0x7fa89c1d1010 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0a01a0 tx=0x7fa88c010c80 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 client.22531.objecter resend_mon_ops 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 client.22531.objecter ms_handle_connect 0x7fa880038e40 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa8a180e640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa880038e40 0x7fa88003b300 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7fa8980ad000 tx=0x7fa8980a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa88c018070 con 0x7fa89c1d0c30 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: dump: 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.365 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.366 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa88c010e70 con 0x7fa89c1d0c30 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 msgr2=0x7fa89c1d1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 0x7fa89c1d1010 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0a01a0 tx=0x7fa88c010c80 comp rx=0 tx=0).stop 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 msgr2=0x7fa89c1d1010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.338+0000 7fa891ffb640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d0c30 0x7fa89c1d1010 secure :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fa89c0a01a0 tx=0x7fa88c010c80 comp rx=0 tx=0).stop 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a3271640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a3271640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 0x7fa89c12fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a3271640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa89c06e4d0 con 0x7fa89c1d1550 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 0x7fa89c12fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient: get_auth_request con 0x7fa89c1d1550 auth_method 0 2024-04-19T00:46:57.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 10 monclient(hunting): handle_auth_done global_id 18862 payload 824 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 1 -- 172.21.15.43:0/2385613132 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa89c06ec10 con 0x7fa89c1d1550 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa8a080c640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 0x7fa89c12fe10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000c10 tx=0x7fa88c0046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa891ffb640 10 client.22531.objecter ms_handle_connect 0x7fa89c1d1550 2024-04-19T00:46:57.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa891ffb640 10 client.22531.objecter resend_mon_ops 2024-04-19T00:46:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa88c01dd90 con 0x7fa89c1d1550 2024-04-19T00:46:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa891ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa891ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:46:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.350+0000 7fa891ffb640 10 monclient: dump: 2024-04-19T00:46:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.369 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.370 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa88c014430 con 0x7fa89c1d1550 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 msgr2=0x7fa89c12fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 0x7fa89c12fe10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000c10 tx=0x7fa88c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 msgr2=0x7fa89c12fe10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa891ffb640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa89c1d1550 0x7fa89c12fe10 secure :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fa88c000c10 tx=0x7fa88c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:57.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 160812, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:57.372 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:57.373 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:57.374 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:57.375 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:57.376 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.378 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:57.379 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.380 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:57.381 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:57.383 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:57.385 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:57.387 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:57.389 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.391 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:57.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:57.393 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:57.393 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:57.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 10 librados: watch_flush enter 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 10 librados: watch_flush exit 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 20 client.22531.objecter shutdown clearing up homeless session... 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 10 client.22531.objecter successfully canceled tick 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa880038e40 msgr2=0x7fa88003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa880038e40 0x7fa88003b300 secure :-1 s=READY pgs=3336 cs=0 l=1 rev1=1 crypto rx=0x7fa8980ad000 tx=0x7fa8980a6000 comp rx=0 tx=0).stop 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 10 monclient: shutdown 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 msgr2=0x7fa89c137e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 0x7fa89c137e20 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fa894009d10 tx=0x7fa89400c490 comp rx=0 tx=0).stop 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a180e640 1 -- 172.21.15.43:0/2385613132 reap_dead start 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 shutdown_connections 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 --2- 172.21.15.43:0/2385613132 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa89c0a4b10 0x7fa89c137e20 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 >> 172.21.15.43:0/2385613132 conn(0x7fa89c0a9900 msgr2=0x7fa89c071820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 shutdown_connections 2024-04-19T00:46:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 -- 172.21.15.43:0/2385613132 wait complete. 2024-04-19T00:46:57.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.354+0000 7fa8a3271640 1 librados: shutdown 2024-04-19T00:46:57.396 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:57.396 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T00:46:57.396 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T00:46:57.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.506+0000 7f51005e7640 10 monclient: get_monmap_and_config 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.506+0000 7f51005e7640 10 monclient: build_initial_monmap 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.506+0000 7f51005e7640 10 monclient: monmap: 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:57.513304+0000 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:57.513304+0000 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:57.512 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:57.513 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.506+0000 7f51005e7640 1 Processor -- start 2024-04-19T00:46:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.506+0000 7f51005e7640 1 -- start start 2024-04-19T00:46:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.506+0000 7f51005e7640 10 monclient: init 2024-04-19T00:46:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient: _add_conns ranks=[6,7,3,10,0,2,9,8,1,5,4] 2024-04-19T00:46:57.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 0x7f50f8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): picked mon.noname-d con 0x7f50f8138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f50f80a5bb0 0x7f50f8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): picked mon.noname-i con 0x7f50f80a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f8137960 0x7f50f8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): picked mon.noname-g con 0x7f50f8137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f50f80a1020 con 0x7f50f8137960 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f50f81398b0 con 0x7f50f8138a50 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f50f80a0780 con 0x7f50f80a5bb0 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f51005e7640 10 monclient(hunting): authenticate will time out at 1087.640403s 2024-04-19T00:46:57.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 0x7f50f8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 0x7f50f8138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:48106/0 (socket says 172.21.15.43:48106) 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 -- 172.21.15.43:0/1356402492 learned_addr learned my addr 172.21.15.43:0/1356402492 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): get_auth_request con 0x7f50f8138a50 auth_method 0 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50feb84640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f8137960 0x7f50f8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fdb82640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f50f80a5bb0 0x7f50f8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50feb84640 10 monclient(hunting): get_auth_request con 0x7f50f8137960 auth_method 0 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50feb84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50feb84640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50feb84640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient(hunting): handle_auth_done global_id 22722 payload 274 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 -- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f50f80a5bb0 msgr2=0x7f50f8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f50f80a5bb0 0x7f50f8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 -- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f8137960 msgr2=0x7f50f8137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f8137960 0x7f50f8137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient: _finish_hunting 0 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 monclient: found mon.noname-d 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fe383640 1 -- 172.21.15.43:0/1356402492 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f80a0d40 con 0x7f50f8138a50 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fdb82640 10 monclient: get_auth_request con 0x7f50f80a5bb0 auth_method 0 2024-04-19T00:46:57.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.510+0000 7f50fdb82640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f50f80a5bb0 0x7f50f8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fe383640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 0x7f50f8138e30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f50e80098a0 tx=0x7f50e801c410 comp rx=0 tx=0).ready entity=mon.6 client_cookie=749f03b614def47a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 1 -- 172.21.15.43:0/1356402492 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50e80b0070 con 0x7f50f8138a50 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: dump: 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.525 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.526 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: _finish_auth 0 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: _check_auth_tickets 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 5 monclient: authenticate success, global_id 22722 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 1 -- 172.21.15.43:0/1356402492 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50e80044a0 con 0x7f50f8138a50 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 1 -- 172.21.15.43:0/1356402492 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50e8004030 con 0x7f50f8138a50 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 4 set_mon_vals no callback set 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fd381640 10 monclient: dump: 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.528 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: shutdown 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- 172.21.15.43:0/1356402492 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 msgr2=0x7f50f8138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 0x7f50f8138e30 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f50e80098a0 tx=0x7f50e801c410 comp rx=0 tx=0).stop 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- 172.21.15.43:0/1356402492 shutdown_connections 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f8137960 0x7f50f8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f50f80a5bb0 0x7f50f8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- 172.21.15.43:0/1356402492 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f50f8138a50 0x7f50f8138e30 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- 172.21.15.43:0/1356402492 >> 172.21.15.43:0/1356402492 conn(0x7f50f80a7cb0 msgr2=0x7f50f80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- 172.21.15.43:0/1356402492 shutdown_connections 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- 172.21.15.43:0/1356402492 wait complete. 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: build_initial_monmap 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: monmap: 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:57.530 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 librados: starting msgr at 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 librados: starting objecter 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 Processor -- start 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- start start 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 librados: setting wanted keys 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 librados: calling monclient init 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: init 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient: _add_conns ranks=[3,4,1,8,0,7,5,6,2] 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f80a5bb0 0x7f50f81d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): picked mon.noname-g con 0x7f50f80a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 0x7f50f8136710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): picked mon.noname-c con 0x7f50f8137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:57.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50f8138a50 0x7f50f813c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): picked mon.noname-f con 0x7f50f8138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f50f80a2100 con 0x7f50f8138a50 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f50f80a1f80 con 0x7f50f80a5bb0 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f50f8142810 con 0x7f50f8137960 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f51005e7640 10 monclient(hunting): authenticate will time out at 1087.647403s 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 0x7f50f8136710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 0x7f50f8136710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42944/0 (socket says 172.21.15.43:42944) 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 -- 172.21.15.43:0/463382671 learned_addr learned my addr 172.21.15.43:0/463382671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): get_auth_request con 0x7f50f8137960 auth_method 0 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fe383640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f80a5bb0 0x7f50f81d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50feb84640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50f8138a50 0x7f50f813c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fe383640 10 monclient(hunting): get_auth_request con 0x7f50f80a5bb0 auth_method 0 2024-04-19T00:46:57.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fe383640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fe383640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fe383640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50feb84640 10 monclient(hunting): get_auth_request con 0x7f50f8138a50 auth_method 0 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50feb84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50feb84640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50feb84640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient(hunting): handle_auth_done global_id 22540 payload 824 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 -- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f80a5bb0 msgr2=0x7f50f81d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f80a5bb0 0x7f50f81d07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 -- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50f8138a50 msgr2=0x7f50f813c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50f8138a50 0x7f50f813c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient: _finish_hunting 0 2024-04-19T00:46:57.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 monclient: found mon.noname-c 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.514+0000 7f50fdb82640 1 -- 172.21.15.43:0/463382671 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f809fc90 con 0x7f50f8137960 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50fdb82640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 0x7f50f8136710 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f50f400c820 tx=0x7f50f400cce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 client.?.objecter ms_handle_connect 0x7f50f8137960 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f4015070 con 0x7f50f8137960 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: dump: 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.536 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.537 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: _finish_auth 0 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: _check_auth_tickets 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 5 monclient: authenticate success, global_id 22540 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f4007ca0 con 0x7f50f8137960 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 monclient: _renew_subs 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50f809ee60 con 0x7f50f8137960 2024-04-19T00:46:57.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50f4007e50 con 0x7f50f8137960 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 client.22540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 monclient: dump: 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:57.539 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50fcb80640 4 set_mon_vals no callback set 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 monclient: _renew_subs 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f50f81221b0 con 0x7f50f8137960 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 1 librados: init done 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50f80008e0 con 0x7f50f8137960 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f50f4018070 con 0x7f50f8137960 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f50dc038fe0 0x7f50dc03b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(102..102 src has 1..102) ==== 8603+0+0 (secure 0 0 0) 0x7f50f4049a10 con 0x7f50f8137960 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 client.22540.objecter ms_dispatch 0x7f50f8000ba0 osd_map(102..102 src has 1..102) 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 3 client.22540.objecter handle_osd_map got epochs [102,102] > 0 2024-04-19T00:46:57.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 3 client.22540.objecter handle_osd_map decoding full epoch 102 2024-04-19T00:46:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 20 client.22540.objecter dump_active .. 0 homeless 2024-04-19T00:46:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50fe383640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f50dc038fe0 0x7f50dc03b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50fe383640 10 monclient: get_auth_request con 0x7f50dc038fe0 auth_method 0 2024-04-19T00:46:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50eeffd640 10 client.22540.objecter ms_handle_connect 0x7f50dc038fe0 2024-04-19T00:46:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.518+0000 7f50fe383640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f50dc038fe0 0x7f50dc03b4a0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f50e80ad000 tx=0x7f50e80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:57.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.522+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f50f40474d0 con 0x7f50f8137960 2024-04-19T00:46:57.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.522+0000 7f50eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:57.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.522+0000 7f50eeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:57.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.786+0000 7f51005e7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T00:46:57.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.786+0000 7f51005e7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T00:46:57.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.786+0000 7f51005e7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:46:57.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.786+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f50f8068240 con 0x7f50f8137960 2024-04-19T00:46:57.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.790+0000 7f50eeffd640 1 -- 172.21.15.43:0/463382671 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f50f4011910 con 0x7f50f8137960 2024-04-19T00:46:57.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.790+0000 7f50eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T00:46:57.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.790+0000 7f50eeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:57.798 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=3276270298,config=2586702516,health=2376989687,kv=4196696685,logm=1996563883}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=10,kv=11,logm=66} crc {auth=4107329962,config=2586702516,health=2156108355,kv=3951634984,logm=1334838157}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=4107329962,config=2586702516,health=2376989687,kv=4196696685,logm=4160635956}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=10,config=2,health=11,kv=10,logm=67} crc {auth=158811932,config=2586702516,health=310147046,kv=801871553,logm=1674547552}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.799 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=10,config=1,health=10,kv=11,logm=68} crc {auth=3639041360,config=1228700967,health=215110181,kv=2173196854,logm=886709241}) 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=2276727083,config=2586702516,health=2376989687,kv=4196696685,logm=3641198130}) 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1345923955,config=2586702516,health=2376989687,kv=3166246699,logm=625023799}) 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=1150319592,config=2586702516,health=2376989687,kv=4196696685,logm=157664661}) 2024-04-19T00:46:57.800 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=8,config=2,health=12,kv=11,logm=67} crc {auth=3984750240,config=2586702516,health=2376989687,kv=4171600807,logm=713449924}) 2024-04-19T00:46:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 10 librados: watch_flush enter 2024-04-19T00:46:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 10 librados: watch_flush exit 2024-04-19T00:46:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 20 client.22540.objecter shutdown clearing up homeless session... 2024-04-19T00:46:57.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 10 client.22540.objecter successfully canceled tick 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f50dc038fe0 msgr2=0x7f50dc03b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f50dc038fe0 0x7f50dc03b4a0 secure :-1 s=READY pgs=3337 cs=0 l=1 rev1=1 crypto rx=0x7f50e80ad000 tx=0x7f50e80a6000 comp rx=0 tx=0).stop 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 10 monclient: shutdown 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 msgr2=0x7f50f8136710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 0x7f50f8136710 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f50f400c820 tx=0x7f50f400cce0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 shutdown_connections 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f50dc038fe0 0x7f50dc03b4a0 unknown :-1 s=CLOSED pgs=3337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f50f8138a50 0x7f50f813c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f50f8137960 0x7f50f8136710 secure :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f50f400c820 tx=0x7f50f400cce0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 --2- 172.21.15.43:0/463382671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50f80a5bb0 0x7f50f81d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 >> 172.21.15.43:0/463382671 conn(0x7f50f80a7cb0 msgr2=0x7f50f806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:57.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 shutdown_connections 2024-04-19T00:46:57.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 -- 172.21.15.43:0/463382671 wait complete. 2024-04-19T00:46:57.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:57.794+0000 7f51005e7640 1 librados: shutdown 2024-04-19T00:46:57.804 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.804 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.804 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1281000812}) 2024-04-19T00:46:57.804 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3204565913}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=613734871}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3278925058}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2852046609}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.805 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3232290001}) 2024-04-19T00:46:57.806 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.806 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.806 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=800283621}) 2024-04-19T00:46:57.806 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.806 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679700730}) 2024-04-19T00:46:57.806 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.798+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3235260666}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1226865060}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543506627}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=347517914}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2633881033}) 2024-04-19T00:46:57.808 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3069618923}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1203433620}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1317939992}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1423776096}) 2024-04-19T00:46:57.809 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.802+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3843659076}) 2024-04-19T00:46:57.810 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.810 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.810 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1988813926}) 2024-04-19T00:46:57.810 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.810 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279022272}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=813587741}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3736130204}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=76864165}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=876798497}) 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.811 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.812 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2024683845}) 2024-04-19T00:46:57.812 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.812 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123190780}) 2024-04-19T00:46:57.812 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.806+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2083876726}) 2024-04-19T00:46:57.815 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2956602137}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=798968514}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2091322376}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=156961598}) 2024-04-19T00:46:57.816 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=197112994}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1836456829}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=237060062}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983099558}) 2024-04-19T00:46:57.817 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.810+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=368958255}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596700526}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1211134111}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1062710570}) 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.819 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4009075353}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1729050344}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2064213290}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=179426684}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115779719}) 2024-04-19T00:46:57.820 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2630781646}) 2024-04-19T00:46:57.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T00:46:57.824 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597699853}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=400845365}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2892854479}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=909451317}) 2024-04-19T00:46:57.825 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2652276666}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3271104546}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2422597182}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1895341364}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.814+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4204787994}) 2024-04-19T00:46:57.826 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62420402}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=834240901}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3052115866}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=770180770}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.827 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4126678281}) 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2935784005}) 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2187115012}) 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651086352}) 2024-04-19T00:46:57.828 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.818+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2429287994}) 2024-04-19T00:46:57.829 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.829 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.829 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314355630}) 2024-04-19T00:46:57.829 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.829 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.829 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293031825}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3375417727}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=327745650}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2454462352}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2284467465}) 2024-04-19T00:46:57.830 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.831 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.831 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2284786647}) 2024-04-19T00:46:57.831 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.831 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678033796}) 2024-04-19T00:46:57.831 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.822+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4280198548}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667842720}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2015237515}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2878152171}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4259043667}) 2024-04-19T00:46:57.832 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=459829975}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2158590715}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3184349132}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182148885}) 2024-04-19T00:46:57.833 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.826+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3426858225}) 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2658876268}) 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3711795455}) 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.839 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=789167020}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4152279942}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3659036764}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3317982026}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=918526463}) 2024-04-19T00:46:57.840 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3988361718}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4030915131}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3433812025}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737405408}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2511900510}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.841 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=380982683}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2563370517}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4290637568}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3737916864}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804875234}) 2024-04-19T00:46:57.842 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.830+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3414672167}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155001636}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466608768}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2950995975}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3799076803}) 2024-04-19T00:46:57.843 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2358780522}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1237047449}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=277057398}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861739577}) 2024-04-19T00:46:57.844 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3961779334}) 2024-04-19T00:46:57.851 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.851 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.851 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=337790695}) 2024-04-19T00:46:57.851 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.851 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3039282759}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=395197437}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3055714022}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4004787369}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.834+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2987708088}) 2024-04-19T00:46:57.852 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2276214365}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4242375798}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1262285105}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707792807}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481367723}) 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.853 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1251380183}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=864810162}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2334595470}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=791624886}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2607974131}) 2024-04-19T00:46:57.854 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426104334}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.838+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3993240758}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1905228885}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925955474}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1701871961}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.855 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1797487580}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=23905310}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3912446394}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1336568126}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179719997}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.842+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3145720304}) 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.856 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3104522431}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3067005319}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2740534348}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3005711589}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=803832016}) 2024-04-19T00:46:57.857 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=993514999}) 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=387012675}) 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265687297}) 2024-04-19T00:46:57.858 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.846+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2273995151}) 2024-04-19T00:46:57.861 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=810270985,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3340302065,mon_config_key=1197721214,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1556734149,osd_snap=775344158}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=1742362799,mdsmap=2840952586,mgr=3781018680,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1652125471,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1556734149,osd_snap=1738499245}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=1374028047,mdsmap=2840952586,mgr=3781018680,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3340302065,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1556734149,osd_snap=1369780678}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=47,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=6} crc {logm=1443990890,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=345987241,mon_config_key=4212284759,monmap=3140514010,osd_metadata=1120045825,osd_pg_creating=1556734149,osd_snap=1738499245}) 2024-04-19T00:46:57.862 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=48,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=3377086349,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3340302065,mon_config_key=3659917387,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1556734149,osd_snap=775344158}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=3819712778,mdsmap=2840952586,mgr=4250140951,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3340302065,mon_config_key=2744799076,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=48,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=264020417,mdsmap=2840952586,mgr=2803061145,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2518249186,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=4} crc {logm=1498058565,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1132293858,mon_config_key=3051443354,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1556734149,osd_snap=4184678892}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.850+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=7} crc {logm=3893219535,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3340302065,mon_config_key=2662526518,monmap=4080741109,osd_metadata=3305883572,osd_pg_creating=1556734149,osd_snap=2868658283}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=815288094,osdmap=776109432}) 2024-04-19T00:46:57.863 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=71729161,osdmap=2974459177}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=71729161,osdmap=1429644369}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=71729161,osdmap=1806444503}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=71729161,osdmap=875626050}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=71729161,osdmap=3714841311}) 2024-04-19T00:46:57.864 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.865 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.865 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=3957320197,osdmap=2938764602}) 2024-04-19T00:46:57.865 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.865 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=30056001,osdmap=3596552091}) 2024-04-19T00:46:57.865 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.854+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=13,osdmap=87} crc {osd_snap=1166113925,osdmap=1469061663}) 2024-04-19T00:46:57.873 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.873 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.873 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=3418022026}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=75} crc {osdmap=540632697}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=77} crc {osdmap=1562081099}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=78} crc {osdmap=138327192}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=77} crc {osdmap=1172576228}) 2024-04-19T00:46:57.874 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=76} crc {osdmap=247891525}) 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=76} crc {osdmap=2129904434}) 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=77} crc {osdmap=920481253}) 2024-04-19T00:46:57.875 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:46:57.870+0000 7f204c69d640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=76} crc {osdmap=1014772750}) 2024-04-19T00:46:58.823 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:46:58.823 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:46:58.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient: get_monmap_and_config 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient: build_initial_monmap 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient: monmap: 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:58.956216+0000 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:58.956216+0000 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:58.955 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:58.956 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:58.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 Processor -- start 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 -- start start 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient: init 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient: _add_conns ranks=[6,4,8,3,9,1,5,10,2,7,0] 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcec8138a50 0x7fcec8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): picked mon.noname-d con 0x7fcec8138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): picked mon.noname-c con 0x7fcec80a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcec8137960 0x7fcec8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): picked mon.noname-e con 0x7fcec8137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fcec80a1020 con 0x7fcec80a5bb0 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:58.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fcec81398b0 con 0x7fcec8138a50 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcec80a0780 con 0x7fcec8137960 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcecd060640 10 monclient(hunting): authenticate will time out at 1089.083631s 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec6d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcec8138a50 0x7fcec8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42958/0 (socket says 172.21.15.43:42958) 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 1 -- 172.21.15.43:0/337347346 learned_addr learned my addr 172.21.15.43:0/337347346 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): get_auth_request con 0x7fcec80a5bb0 auth_method 0 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec759e640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcec8137960 0x7fcec8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec6d9d640 10 monclient(hunting): get_auth_request con 0x7fcec8138a50 auth_method 0 2024-04-19T00:46:58.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec759e640 10 monclient(hunting): get_auth_request con 0x7fcec8137960 auth_method 0 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.950+0000 7fcec659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:58.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 10 monclient(hunting): handle_auth_done global_id 22558 payload 274 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 -- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcec8137960 msgr2=0x7fcec8137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcec8137960 0x7fcec8137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 -- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcec8138a50 msgr2=0x7fcec8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcec8138a50 0x7fcec8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 10 monclient: _finish_hunting 0 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 monclient: found mon.noname-c 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 -- 172.21.15.43:0/337347346 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec80a0d40 con 0x7fcec80a5bb0 2024-04-19T00:46:58.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec659c640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec8139370 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fceb00098a0 tx=0x7fceb001c410 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3ce062f1eb7f31b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:58.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 1 -- 172.21.15.43:0/337347346 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fceb00b0070 con 0x7fcec80a5bb0 2024-04-19T00:46:58.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:58.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:58.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: dump: 2024-04-19T00:46:58.968 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:58.969 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: _finish_auth 0 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 5 monclient: authenticate success, global_id 22558 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 1 -- 172.21.15.43:0/337347346 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb00044a0 con 0x7fcec80a5bb0 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 4 set_mon_vals no callback set 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 1 -- 172.21.15.43:0/337347346 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fceb0003a90 con 0x7fcec80a5bb0 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcec5d9b640 10 monclient: dump: 2024-04-19T00:46:58.970 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:58.971 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 10 monclient: shutdown 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 -- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 msgr2=0x7fcec8139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec8139370 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fceb00098a0 tx=0x7fceb001c410 comp rx=0 tx=0).stop 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 -- 172.21.15.43:0/337347346 shutdown_connections 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcec8137960 0x7fcec8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec8139370 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 --2- 172.21.15.43:0/337347346 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcec8138a50 0x7fcec8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 -- 172.21.15.43:0/337347346 >> 172.21.15.43:0/337347346 conn(0x7fcec80a7cb0 msgr2=0x7fcec80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 -- 172.21.15.43:0/337347346 shutdown_connections 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 1 -- 172.21.15.43:0/337347346 wait complete. 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 10 monclient: build_initial_monmap 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.954+0000 7fcecd060640 10 monclient: monmap: 2024-04-19T00:46:58.972 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:58.973 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 librados: starting msgr at 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 librados: starting objecter 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 Processor -- start 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- start start 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 librados: setting wanted keys 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 librados: calling monclient init 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: init 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _add_conns ranks=[4,1,2,0,6,8,5,7,3] 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec81d0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): picked mon.noname-c con 0x7fcec80a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec8137960 0x7fcec8136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): picked mon.noname-f con 0x7fcec8137960 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcec8138a50 0x7fcec813c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): picked mon.noname-b con 0x7fcec8138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcec80a2100 con 0x7fcec8137960 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fcec80a1f80 con 0x7fcec8138a50 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fcec8142810 con 0x7fcec80a5bb0 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient(hunting): authenticate will time out at 1089.088945s 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec81d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec81d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42968/0 (socket says 172.21.15.43:42968) 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcec8138a50 0x7fcec813c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 1 -- 172.21.15.43:0/2717686630 learned_addr learned my addr 172.21.15.43:0/2717686630 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): get_auth_request con 0x7fcec80a5bb0 auth_method 0 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:58.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): get_auth_request con 0x7fcec8138a50 auth_method 0 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient(hunting): handle_auth_done global_id 22691 payload 824 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec659c640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec8137960 0x7fcec8136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 -- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 msgr2=0x7fcec81d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:58.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec81d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 -- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec8137960 msgr2=0x7fcec8136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec8137960 0x7fcec8136950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient: _finish_hunting 0 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 monclient: found mon.noname-b 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 -- 172.21.15.43:0/2717686630 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec81d1830 con 0x7fcec8138a50 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec81d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec759e640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcec8138a50 0x7fcec813c770 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fceb800e7d0 tx=0x7fceb800ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 client.?.objecter ms_handle_connect 0x7fcec8138a50 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fceb8017070 con 0x7fcec8138a50 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:58.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: dump: 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: _finish_auth 0 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb800cd50 con 0x7fcec8138a50 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fceb80138e0 con 0x7fcec8138a50 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 5 monclient: authenticate success, global_id 22691 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec559a640 4 set_mon_vals no callback set 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 monclient: dump: 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _renew_subs 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec809fc90 con 0x7fcec8138a50 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 client.22691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _renew_subs 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec813e3b0 con 0x7fcec8138a50 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 librados: init done 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fceb801a070 con 0x7fcec8138a50 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcea8038e10 0x7fcea803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec813ec90 con 0x7fcec8138a50 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 8218+0+0 (secure 0 0 0) 0x7fceb800f040 con 0x7fcec8138a50 2024-04-19T00:46:58.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 10 client.22691.objecter ms_dispatch 0x7fcec8000ba0 osd_map(104..104 src has 1..104) 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 3 client.22691.objecter handle_osd_map got epochs [104,104] > 0 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 3 client.22691.objecter handle_osd_map decoding full epoch 104 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcebf7fe640 20 client.22691.objecter dump_active .. 0 homeless 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcea8038e10 0x7fcea803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.958+0000 7fcec6d9d640 10 monclient: get_auth_request con 0x7fcea8038e10 auth_method 0 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.962+0000 7fcec6d9d640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcea8038e10 0x7fcea803b2d0 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7fcec00094c0 tx=0x7fcec000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:58.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.962+0000 7fcebf7fe640 10 client.22691.objecter ms_handle_connect 0x7fcea8038e10 2024-04-19T00:46:58.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.966+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fceb8021070 con 0x7fcec8138a50 2024-04-19T00:46:58.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.966+0000 7fcebf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:58.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:58.966+0000 7fcebf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.194+0000 7fcecd060640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.194+0000 7fcecd060640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.194+0000 7fcecd060640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.194+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcec8076190 con 0x7fcec8138a50 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcebf7fe640 1 -- 172.21.15.43:0/2717686630 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fceb80479d0 con 0x7fcec8138a50 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcebf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:46:59.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcebf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:59.203 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":24,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:46:59.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 10 librados: watch_flush enter 2024-04-19T00:46:59.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 10 librados: watch_flush exit 2024-04-19T00:46:59.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 20 client.22691.objecter shutdown clearing up homeless session... 2024-04-19T00:46:59.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 10 client.22691.objecter successfully canceled tick 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcea8038e10 msgr2=0x7fcea803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcea8038e10 0x7fcea803b2d0 secure :-1 s=READY pgs=3340 cs=0 l=1 rev1=1 crypto rx=0x7fcec00094c0 tx=0x7fcec000b040 comp rx=0 tx=0).stop 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 10 monclient: shutdown 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcec8138a50 msgr2=0x7fcec813c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcec8138a50 0x7fcec813c770 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7fceb800e7d0 tx=0x7fceb800ec90 comp rx=0 tx=0).stop 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 shutdown_connections 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcea8038e10 0x7fcea803b2d0 unknown :-1 s=CLOSED pgs=3340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fcec8138a50 0x7fcec813c770 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec8137960 0x7fcec8136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 --2- 172.21.15.43:0/2717686630 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcec80a5bb0 0x7fcec81d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 >> 172.21.15.43:0/2717686630 conn(0x7fcec80a7cb0 msgr2=0x7fcec806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 shutdown_connections 2024-04-19T00:46:59.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 -- 172.21.15.43:0/2717686630 wait complete. 2024-04-19T00:46:59.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.198+0000 7fcecd060640 1 librados: shutdown 2024-04-19T00:46:59.220 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:46:59.220 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:46:59.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:46:59.220 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient: get_monmap_and_config 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient: monmap: 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:59.347397+0000 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:59.347397+0000 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:59.347 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 Processor -- start 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 -- start start 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient: init 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient: _add_conns ranks=[6,7,10,3,2,1,0,5,4,9,8] 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 0x7f62100a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): picked mon.noname-d con 0x7f62100a2a80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6210136ed0 0x7f62100a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): picked mon.noname-i con 0x7f6210136ed0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f62101372f0 0x7f62100a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): picked mon.noname-f con 0x7f62101372f0 addr v2:172.21.15.161:3300/0 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6210072fd0 con 0x7f62101372f0 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f62100732b0 con 0x7f62100a2a80 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6210073590 con 0x7f6210136ed0 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6217770640 10 monclient(hunting): authenticate will time out at 1089.474813s 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 0x7f62100a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 0x7f62100a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:48138/0 (socket says 172.21.15.43:48138) 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 -- 172.21.15.43:0/1825655018 learned_addr learned my addr 172.21.15.43:0/1825655018 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): get_auth_request con 0x7f62100a2a80 auth_method 0 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6214d0b640 1 --2- 172.21.15.43:0/1825655018 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6210136ed0 0x7f62100a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6214d0b640 10 monclient(hunting): get_auth_request con 0x7f6210136ed0 auth_method 0 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6214d0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6214d0b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6214d0b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient(hunting): handle_auth_done global_id 22758 payload 274 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6215d0d640 1 --2- 172.21.15.43:0/1825655018 >> v2:172.21.15.161:3300/0 conn(0x7f62101372f0 0x7f62100a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 -- 172.21.15.43:0/1825655018 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6210136ed0 msgr2=0x7f62100a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 --2- 172.21.15.43:0/1825655018 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6210136ed0 0x7f62100a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 -- 172.21.15.43:0/1825655018 >> v2:172.21.15.161:3300/0 conn(0x7f62101372f0 msgr2=0x7f62100a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 --2- 172.21.15.43:0/1825655018 >> v2:172.21.15.161:3300/0 conn(0x7f62101372f0 0x7f62100a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 monclient: found mon.noname-d 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f621550c640 1 -- 172.21.15.43:0/1825655018 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621006c5b0 con 0x7f62100a2a80 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6215d0d640 10 monclient: get_auth_request con 0x7f62101372f0 auth_method 0 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.342+0000 7f6215d0d640 1 --2- 172.21.15.43:0/1825655018 >> v2:172.21.15.161:3300/0 conn(0x7f62101372f0 0x7f62100a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f621550c640 1 --2- 172.21.15.43:0/1825655018 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 0x7f62100a2e60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f620c009870 tx=0x7f620c01c4b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7a7a6a0e0ba997ec server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 1 -- 172.21.15.43:0/1825655018 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f620c0b0070 con 0x7f62100a2a80 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: dump: 2024-04-19T00:46:59.354 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.355 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 5 monclient: authenticate success, global_id 22758 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 1 -- 172.21.15.43:0/1825655018 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f620c0aa810 con 0x7f62100a2a80 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 1 -- 172.21.15.43:0/1825655018 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f620c0a9300 con 0x7f62100a2a80 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6207fff640 10 monclient: dump: 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.356 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 4 set_mon_vals no callback set 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: shutdown 2024-04-19T00:46:59.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- 172.21.15.43:0/1825655018 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 msgr2=0x7f62100a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- 172.21.15.43:0/1825655018 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 0x7f62100a2e60 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f620c009870 tx=0x7f620c01c4b0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- 172.21.15.43:0/1825655018 shutdown_connections 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- 172.21.15.43:0/1825655018 >> v2:172.21.15.161:3300/0 conn(0x7f62101372f0 0x7f62100a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- 172.21.15.43:0/1825655018 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6210136ed0 0x7f62100a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- 172.21.15.43:0/1825655018 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f62100a2a80 0x7f62100a2e60 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- 172.21.15.43:0/1825655018 >> 172.21.15.43:0/1825655018 conn(0x7f62100a6410 msgr2=0x7f621006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- 172.21.15.43:0/1825655018 shutdown_connections 2024-04-19T00:46:59.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- 172.21.15.43:0/1825655018 wait complete. 2024-04-19T00:46:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: monmap: 2024-04-19T00:46:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 librados: starting msgr at 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 librados: starting objecter 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 Processor -- start 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- start start 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 librados: setting wanted keys 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 librados: calling monclient init 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: init 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient: _add_conns ranks=[1,2,8,3,4,0,7,5,6] 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62100a2a80 0x7f621011fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): picked mon.noname-f con 0x7f62100a2a80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6210136ed0 0x7f6210135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): picked mon.noname-b con 0x7f6210136ed0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f62101372f0 0x7f621013b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): picked mon.noname-e con 0x7f62101372f0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6210140620 con 0x7f62100a2a80 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f62100a5930 con 0x7f6210136ed0 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f62101407a0 con 0x7f62101372f0 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.346+0000 7f6217770640 10 monclient(hunting): authenticate will time out at 1089.479980s 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6210136ed0 0x7f6210135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6210136ed0 0x7f6210135880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37326/0 (socket says 172.21.15.43:37326) 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 1 -- 172.21.15.43:0/3921419439 learned_addr learned my addr 172.21.15.43:0/3921419439 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): get_auth_request con 0x7f6210136ed0 auth_method 0 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f62101372f0 0x7f621013b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): get_auth_request con 0x7f62101372f0 auth_method 0 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62100a2a80 0x7f621011fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): get_auth_request con 0x7f62100a2a80 auth_method 0 2024-04-19T00:46:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient(hunting): handle_auth_done global_id 22337 payload 824 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6210136ed0 msgr2=0x7f6210135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6210136ed0 0x7f6210135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62100a2a80 msgr2=0x7f621011fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62100a2a80 0x7f621011fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 monclient: found mon.noname-e 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f621013d8e0 con 0x7f62101372f0 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f62100a2a80 0x7f621011fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:46:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6210136ed0 0x7f6210135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6215d0d640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f62101372f0 0x7f621013b810 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f620800cc60 tx=0x7f620800a650 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 client.?.objecter ms_handle_connect 0x7f62101372f0 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f620800f8f0 con 0x7f62101372f0 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: dump: 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f620800fad0 con 0x7f62101372f0 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f620800fc80 con 0x7f62101372f0 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f62077fe640 4 set_mon_vals no callback set 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: dump: 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 5 monclient: authenticate success, global_id 22337 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 monclient: _renew_subs 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f621009ff00 con 0x7f62101372f0 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 client.22337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 monclient: _renew_subs 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:46:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f621013d440 con 0x7f62101372f0 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 1 librados: init done 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 0x7f621013bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f621006d140 con 0x7f6210120ea0 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f620800f480 con 0x7f62101372f0 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 0x7f621013bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient: get_auth_request con 0x7f6210120ea0 auth_method 0 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f61f4038e10 0x7f61f403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(104..104 src has 1..104) ==== 8218+0+0 (secure 0 0 0) 0x7f620804a770 con 0x7f62101372f0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 client.22337.objecter ms_dispatch 0x7f6210000ba0 osd_map(104..104 src has 1..104) 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 3 client.22337.objecter handle_osd_map got epochs [104,104] > 0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 3 client.22337.objecter handle_osd_map decoding full epoch 104 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 10 monclient(hunting): handle_auth_done global_id 29637 payload 824 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 20 client.22337.objecter dump_active .. 0 homeless 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6210121280 con 0x7f6210120ea0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 client.22337.objecter ms_handle_connect 0x7f6210120ea0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 client.22337.objecter resend_mon_ops 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f621550c640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 0x7f621013bd50 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f620c0ab000 tx=0x7f620c01acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f61f4038e10 0x7f61f403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 10 monclient: get_auth_request con 0x7f61f4038e10 auth_method 0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f620c0b9020 con 0x7f6210120ea0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: dump: 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f620c003950 con 0x7f6210120ea0 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 msgr2=0x7f621013bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 0x7f621013bd50 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f620c0ab000 tx=0x7f620c01acf0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 msgr2=0x7f621013bd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6210120ea0 0x7f621013bd50 secure :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f620c0ab000 tx=0x7f620c01acf0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6205ffb640 10 client.22337.objecter ms_handle_connect 0x7f61f4038e10 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.350+0000 7f6214d0b640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f61f4038e10 0x7f61f403b2d0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f62100a4de0 tx=0x7f6200008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 0x7f6210137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f62101cf530 con 0x7f621013c290 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 0x7f6210137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient: get_auth_request con 0x7f621013c290 auth_method 0 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 10 monclient(hunting): handle_auth_done global_id 29646 payload 824 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 1 -- 172.21.15.43:0/3921419439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f621013c6a0 con 0x7f621013c290 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f621550c640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 0x7f6210137860 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f620c000c10 tx=0x7f620c019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 client.22337.objecter ms_handle_connect 0x7f621013c290 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 client.22337.objecter resend_mon_ops 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f620c0b9040 con 0x7f621013c290 2024-04-19T00:46:59.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 monclient: dump: 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.380 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7f620c0be620 con 0x7f621013c290 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 msgr2=0x7f6210137860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 0x7f6210137860 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f620c000c10 tx=0x7f620c019250 comp rx=0 tx=0).stop 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 msgr2=0x7f6210137860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6205ffb640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f621013c290 0x7f6210137860 secure :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f620c000c10 tx=0x7f620c019250 comp rx=0 tx=0).stop 2024-04-19T00:46:59.381 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 162854, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.401 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:59.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:59.403 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 10 librados: watch_flush enter 2024-04-19T00:46:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 10 librados: watch_flush exit 2024-04-19T00:46:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 20 client.22337.objecter shutdown clearing up homeless session... 2024-04-19T00:46:59.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 10 client.22337.objecter successfully canceled tick 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f61f4038e10 msgr2=0x7f61f403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f61f4038e10 0x7f61f403b2d0 secure :-1 s=READY pgs=3341 cs=0 l=1 rev1=1 crypto rx=0x7f62100a4de0 tx=0x7f6200008040 comp rx=0 tx=0).stop 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 10 monclient: shutdown 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f62101372f0 msgr2=0x7f621013b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6215d0d640 1 -- 172.21.15.43:0/3921419439 reap_dead start 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f62101372f0 0x7f621013b810 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f620800cc60 tx=0x7f620800a650 comp rx=0 tx=0).stop 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 shutdown_connections 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 --2- 172.21.15.43:0/3921419439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f62101372f0 0x7f621013b810 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 >> 172.21.15.43:0/3921419439 conn(0x7f62100a6410 msgr2=0x7f62100a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 shutdown_connections 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 -- 172.21.15.43:0/3921419439 wait complete. 2024-04-19T00:46:59.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.358+0000 7f6217770640 1 librados: shutdown 2024-04-19T00:46:59.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:59.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:46:59.408 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:46:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient: get_monmap_and_config 2024-04-19T00:46:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient: monmap: 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:59.527005+0000 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:59.527005+0000 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.526 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:59.527 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 Processor -- start 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 -- start start 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient: init 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient: _add_conns ranks=[1,4,5,3,8,6,10,9,0,2,7] 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3150138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 0x7f315013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): picked mon.noname-c con 0x7f3150139540 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f31500a5bb0 0x7f3150140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): picked mon.noname-h con 0x7f31500a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31500a0d40 con 0x7f3150138a50 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f31500a1020 con 0x7f3150139540 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f31500a0780 con 0x7f31500a5bb0 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f315676d640 10 monclient(hunting): authenticate will time out at 1089.654206s 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 0x7f315013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 0x7f315013dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42970/0 (socket says 172.21.15.43:42970) 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 -- 172.21.15.43:0/3324457619 learned_addr learned my addr 172.21.15.43:0/3324457619 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): get_auth_request con 0x7f3150139540 auth_method 0 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f3154d0a640 1 --2- 172.21.15.43:0/3324457619 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f31500a5bb0 0x7f3150140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f3154d0a640 10 monclient(hunting): get_auth_request con 0x7f31500a5bb0 auth_method 0 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f3154d0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f3154d0a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f3154d0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient(hunting): handle_auth_done global_id 22576 payload 274 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 -- 172.21.15.43:0/3324457619 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f31500a5bb0 msgr2=0x7f3150140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 --2- 172.21.15.43:0/3324457619 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f31500a5bb0 0x7f3150140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 -- 172.21.15.43:0/3324457619 >> v1:172.21.15.161:6789/0 conn(0x7f3150138a50 legacy=0x7f3150138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 monclient: found mon.noname-c 2024-04-19T00:46:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:59.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.522+0000 7f314f7fe640 1 -- 172.21.15.43:0/3324457619 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31500a0a60 con 0x7f3150139540 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314f7fe640 1 --2- 172.21.15.43:0/3324457619 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 0x7f315013dfe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f31380098a0 tx=0x7f313801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f63dd828538103ed server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 1 -- 172.21.15.43:0/3324457619 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31380b0070 con 0x7f3150139540 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: dump: 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 1 -- 172.21.15.43:0/3324457619 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31380aa810 con 0x7f3150139540 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 5 monclient: authenticate success, global_id 22576 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 1 -- 172.21.15.43:0/3324457619 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31380a9300 con 0x7f3150139540 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 4 set_mon_vals no callback set 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314effd640 10 monclient: dump: 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.542 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: shutdown 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- 172.21.15.43:0/3324457619 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 msgr2=0x7f315013dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 --2- 172.21.15.43:0/3324457619 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 0x7f315013dfe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f31380098a0 tx=0x7f313801c440 comp rx=0 tx=0).stop 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- 172.21.15.43:0/3324457619 shutdown_connections 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 --2- 172.21.15.43:0/3324457619 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f31500a5bb0 0x7f3150140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 --2- 172.21.15.43:0/3324457619 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3150139540 0x7f315013dfe0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- 172.21.15.43:0/3324457619 >> 172.21.15.43:0/3324457619 conn(0x7f31500a7cb0 msgr2=0x7f31500a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- 172.21.15.43:0/3324457619 shutdown_connections 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- 172.21.15.43:0/3324457619 wait complete. 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: monmap: 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.544 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 librados: starting msgr at 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 librados: starting objecter 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 Processor -- start 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- start start 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 librados: setting wanted keys 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 librados: calling monclient init 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: init 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient: _add_conns ranks=[2,1,7,4,0,3,8,6,5] 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 0x7f31501d0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): picked mon.noname-b con 0x7f31500a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3150138a50 0x7f31501378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): picked mon.noname-f con 0x7f3150138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:46:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3150139540 0x7f315013c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): picked mon.noname-i con 0x7f3150139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3150143620 con 0x7f3150138a50 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f31501434a0 con 0x7f31500a5bb0 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f31501437a0 con 0x7f3150139540 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f315676d640 10 monclient(hunting): authenticate will time out at 1089.659422s 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 0x7f31501d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 0x7f31501d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37340/0 (socket says 172.21.15.43:37340) 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 -- 172.21.15.43:0/2134916968 learned_addr learned my addr 172.21.15.43:0/2134916968 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): get_auth_request con 0x7f31500a5bb0 auth_method 0 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f3154d0a640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3150139540 0x7f315013c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314f7fe640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3150138a50 0x7f31501378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314f7fe640 10 monclient(hunting): get_auth_request con 0x7f3150138a50 auth_method 0 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f3154d0a640 10 monclient(hunting): get_auth_request con 0x7f3150139540 auth_method 0 2024-04-19T00:46:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f3154d0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f3154d0a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f3154d0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient(hunting): handle_auth_done global_id 22709 payload 824 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3150139540 msgr2=0x7f315013c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3150139540 0x7f315013c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3150138a50 msgr2=0x7f31501378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3150138a50 0x7f31501378f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 monclient: found mon.noname-b 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.526+0000 7f314ffff640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31501d1630 con 0x7f31500a5bb0 2024-04-19T00:46:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314ffff640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 0x7f31501d0720 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f314000c7f0 tx=0x7f314000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 client.?.objecter ms_handle_connect 0x7f31500a5bb0 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3140016020 con 0x7f31500a5bb0 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: dump: 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314000acc0 con 0x7f31500a5bb0 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 5 monclient: authenticate success, global_id 22709 2024-04-19T00:46:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 monclient: _renew_subs 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315009ee60 con 0x7f31500a5bb0 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3140012890 con 0x7f31500a5bb0 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314e7fc640 4 set_mon_vals no callback set 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 client.22709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: dump: 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 monclient: _renew_subs 2024-04-19T00:46:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3150121fa0 con 0x7f31500a5bb0 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 1 librados: init done 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 0x7f315013d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f315006d140 con 0x7f315013cc80 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3140019020 con 0x7f31500a5bb0 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 0x7f315013d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient: get_auth_request con 0x7f315013cc80 auth_method 0 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3130038e10 0x7f313003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 10 monclient(hunting): handle_auth_done global_id 22718 payload 824 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f315013d5a0 con 0x7f315013cc80 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 8218+0+0 (secure 0 0 0) 0x7f3140049cb0 con 0x7f31500a5bb0 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 client.22709.objecter ms_dispatch 0x7f3150000ba0 osd_map(104..104 src has 1..104) 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 3 client.22709.objecter handle_osd_map got epochs [104,104] > 0 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 3 client.22709.objecter handle_osd_map decoding full epoch 104 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 20 client.22709.objecter dump_active .. 0 homeless 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 client.22709.objecter ms_handle_connect 0x7f315013cc80 2024-04-19T00:46:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 client.22709.objecter resend_mon_ops 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314f7fe640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 0x7f315013d060 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f31500a72d0 tx=0x7f31380191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f3154d0a640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3130038e10 0x7f313003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31380ba030 con 0x7f315013cc80 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: dump: 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f3154d0a640 10 monclient: get_auth_request con 0x7f3130038e10 auth_method 0 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31380036e0 con 0x7f315013cc80 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 msgr2=0x7f315013d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 0x7f315013d060 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f31500a72d0 tx=0x7f31380191e0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 msgr2=0x7f315013d060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f315013cc80 0x7f315013d060 secure :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f31500a72d0 tx=0x7f31380191e0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f314cff9640 10 client.22709.objecter ms_handle_connect 0x7f3130038e10 2024-04-19T00:46:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.530+0000 7f3154d0a640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3130038e10 0x7f313003b2d0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f314800b950 tx=0x7f3148008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 0x7f31500a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f315009fc90 con 0x7f31500a3650 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 0x7f31500a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient: get_auth_request con 0x7f31500a3650 auth_method 0 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 10 monclient(hunting): handle_auth_done global_id 22727 payload 824 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 1 -- 172.21.15.43:0/2134916968 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3150076190 con 0x7f31500a3650 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314f7fe640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 0x7f31500a3a60 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3138000c10 tx=0x7f31380a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 client.22709.objecter ms_handle_connect 0x7f31500a3650 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 client.22709.objecter resend_mon_ops 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f31380ba040 con 0x7f31500a3650 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:59.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 monclient: dump: 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.561 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f31380c1ce0 con 0x7f31500a3650 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 msgr2=0x7f31500a3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 0x7f31500a3a60 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3138000c10 tx=0x7f31380a8000 comp rx=0 tx=0).stop 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 msgr2=0x7f31500a3a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f314cff9640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a3650 0x7f31500a3a60 secure :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f3138000c10 tx=0x7f31380a8000 comp rx=0 tx=0).stop 2024-04-19T00:46:59.562 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163038, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:59.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.583 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.584 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:59.585 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:59.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 10 librados: watch_flush enter 2024-04-19T00:46:59.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 10 librados: watch_flush exit 2024-04-19T00:46:59.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 20 client.22709.objecter shutdown clearing up homeless session... 2024-04-19T00:46:59.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 10 client.22709.objecter successfully canceled tick 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3130038e10 msgr2=0x7f313003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3130038e10 0x7f313003b2d0 secure :-1 s=READY pgs=3342 cs=0 l=1 rev1=1 crypto rx=0x7f314800b950 tx=0x7f3148008040 comp rx=0 tx=0).stop 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 10 monclient: shutdown 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 msgr2=0x7f31501d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 0x7f31501d0720 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f314000c7f0 tx=0x7f314000ccb0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f3154d0a640 1 -- 172.21.15.43:0/2134916968 reap_dead start 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 shutdown_connections 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 --2- 172.21.15.43:0/2134916968 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f31500a5bb0 0x7f31501d0720 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 >> 172.21.15.43:0/2134916968 conn(0x7f31500a7cb0 msgr2=0x7f31500a9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 shutdown_connections 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 -- 172.21.15.43:0/2134916968 wait complete. 2024-04-19T00:46:59.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.538+0000 7f315676d640 1 librados: shutdown 2024-04-19T00:46:59.589 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:59.589 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:46:59.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient: get_monmap_and_config 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient: monmap: 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:59.724172+0000 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:59.724172+0000 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 Processor -- start 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 -- start start 2024-04-19T00:46:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient: init 2024-04-19T00:46:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient: _add_conns ranks=[7,4,5,9,0,6,10,1,2,3,8] 2024-04-19T00:46:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610138a50 0x7f6610138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): picked mon.noname-i con 0x7f6610138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 0x7f6610139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): picked mon.noname-c con 0x7f66100a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6610137960 0x7f6610137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): picked mon.noname-h con 0x7f6610137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f66100a1020 con 0x7f66100a5bb0 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f66101398b0 con 0x7f6610137960 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f66100a0780 con 0x7f6610138a50 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6617958640 10 monclient(hunting): authenticate will time out at 1089.850929s 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 0x7f6610139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 0x7f6610139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42982/0 (socket says 172.21.15.43:42982) 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 -- 172.21.15.43:0/1615093014 learned_addr learned my addr 172.21.15.43:0/1615093014 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): get_auth_request con 0x7f66100a5bb0 auth_method 0 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610138a50 0x7f6610138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6610137960 0x7f6610137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): get_auth_request con 0x7f6610138a50 auth_method 0 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): get_auth_request con 0x7f6610137960 auth_method 0 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6615ef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient(hunting): handle_auth_done global_id 22585 payload 274 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 -- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610138a50 msgr2=0x7f6610138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610138a50 0x7f6610138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 -- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6610137960 msgr2=0x7f6610137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6610137960 0x7f6610137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 monclient: found mon.noname-c 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.718+0000 7f6614ef3640 1 -- 172.21.15.43:0/1615093014 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66100a0d40 con 0x7f66100a5bb0 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6614ef3640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 0x7f6610139370 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f66000098a0 tx=0x7f660001c410 comp rx=0 tx=0).ready entity=mon.4 client_cookie=5bf9086e854e6061 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 1 -- 172.21.15.43:0/1615093014 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f66000b0070 con 0x7f66100a5bb0 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: dump: 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 1 -- 172.21.15.43:0/1615093014 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66000044a0 con 0x7f66100a5bb0 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 5 monclient: authenticate success, global_id 22585 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 1 -- 172.21.15.43:0/1615093014 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6600003a90 con 0x7f66100a5bb0 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6607fff640 10 monclient: dump: 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.739 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 4 set_mon_vals no callback set 2024-04-19T00:46:59.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 10 monclient: shutdown 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 -- 172.21.15.43:0/1615093014 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 msgr2=0x7f6610139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 0x7f6610139370 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f66000098a0 tx=0x7f660001c410 comp rx=0 tx=0).stop 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 -- 172.21.15.43:0/1615093014 shutdown_connections 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6610137960 0x7f6610137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66100a5bb0 0x7f6610139370 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 --2- 172.21.15.43:0/1615093014 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610138a50 0x7f6610138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 -- 172.21.15.43:0/1615093014 >> 172.21.15.43:0/1615093014 conn(0x7f66100a7cb0 msgr2=0x7f66100a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 -- 172.21.15.43:0/1615093014 shutdown_connections 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 -- 172.21.15.43:0/1615093014 wait complete. 2024-04-19T00:46:59.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 10 monclient: monmap: 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.742 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 librados: starting msgr at 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 librados: starting objecter 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 Processor -- start 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 -- start start 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 librados: setting wanted keys 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 1 librados: calling monclient init 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.722+0000 7f6617958640 10 monclient: init 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient: _add_conns ranks=[3,7,2,6,4,0,5,8,1] 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f66100a5bb0 0x7f66101cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): picked mon.noname-g con 0x7f66100a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610137960 0x7f6610136e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): picked mon.noname-i con 0x7f6610137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6610138a50 0x7f66101d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): picked mon.noname-b con 0x7f6610138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f66100a2100 con 0x7f6610138a50 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f66100a1f80 con 0x7f66100a5bb0 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6610142810 con 0x7f6610137960 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): authenticate will time out at 1089.856191s 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6610138a50 0x7f66101d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6610138a50 0x7f66101d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37380/0 (socket says 172.21.15.43:37380) 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 -- 172.21.15.43:0/1995398985 learned_addr learned my addr 172.21.15.43:0/1995398985 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): get_auth_request con 0x7f6610138a50 auth_method 0 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610137960 0x7f6610136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): get_auth_request con 0x7f6610137960 auth_method 0 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f66100a5bb0 0x7f66101cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): get_auth_request con 0x7f66100a5bb0 auth_method 0 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient(hunting): handle_auth_done global_id 22736 payload 824 2024-04-19T00:46:59.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610137960 msgr2=0x7f6610136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6610137960 0x7f6610136e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f66100a5bb0 msgr2=0x7f66101cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f66100a5bb0 0x7f66101cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 monclient: found mon.noname-b 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66100a4a30 con 0x7f6610138a50 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f66100a5bb0 0x7f66101cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6615ef5640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6610138a50 0x7f66101d0d40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f660800efc0 tx=0x7f660800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 client.?.objecter ms_handle_connect 0x7f6610138a50 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:59.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6608018020 con 0x7f6610138a50 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: dump: 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.748 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660800cd30 con 0x7f6610138a50 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 5 monclient: authenticate success, global_id 22736 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient: _renew_subs 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66100aacb0 con 0x7f6610138a50 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6608014d60 con 0x7f6610138a50 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: dump: 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.750 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 client.22736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66077fe640 4 set_mon_vals no callback set 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient: _renew_subs 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f661009ee60 con 0x7f6610138a50 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 librados: init done 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 0x7f66101d1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f661006d140 con 0x7f6610137380 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 0x7f66101d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f660801b020 con 0x7f6610138a50 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient: get_auth_request con 0x7f6610137380 auth_method 0 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65f4038e10 0x7f65f403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 8683+0+0 (secure 0 0 0) 0x7f660804ac60 con 0x7f6610138a50 2024-04-19T00:46:59.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 client.22736.objecter ms_dispatch 0x7f6610000ba0 osd_map(105..105 src has 1..105) 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 3 client.22736.objecter handle_osd_map got epochs [105,105] > 0 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 3 client.22736.objecter handle_osd_map decoding full epoch 105 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 10 monclient(hunting): handle_auth_done global_id 22594 payload 824 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6610137760 con 0x7f6610137380 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 20 client.22736.objecter dump_active .. 0 homeless 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f66156f4640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 0x7f66101d1280 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f660c011550 tx=0x7f660c011a10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 client.22736.objecter ms_handle_connect 0x7f6610137380 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 client.22736.objecter resend_mon_ops 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65f4038e10 0x7f65f403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 10 monclient: get_auth_request con 0x7f65f4038e10 auth_method 0 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f660c00fb30 con 0x7f6610137380 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: dump: 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.754 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f660c00f6c0 con 0x7f6610137380 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 msgr2=0x7f66101d1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 0x7f66101d1280 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f660c011550 tx=0x7f660c011a10 comp rx=0 tx=0).stop 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 msgr2=0x7f66101d1280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6610137380 0x7f66101d1280 secure :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f660c011550 tx=0x7f660c011a10 comp rx=0 tx=0).stop 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6614ef3640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65f4038e10 0x7f65f403b2d0 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f66000ad000 tx=0x7f66000a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.726+0000 7f6605ffb640 10 client.22736.objecter ms_handle_connect 0x7f65f4038e10 2024-04-19T00:46:59.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:59.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 0x7f66100a3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f66101d1bd0 con 0x7f66101d17c0 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 0x7f66100a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient: get_auth_request con 0x7f66101d17c0 auth_method 0 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 10 monclient(hunting): handle_auth_done global_id 22603 payload 824 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 1 -- 172.21.15.43:0/1995398985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66101ce3a0 con 0x7f66101d17c0 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f66156f4640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 0x7f66100a3310 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f660c011550 tx=0x7f660c0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 client.22736.objecter ms_handle_connect 0x7f66101d17c0 2024-04-19T00:46:59.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 client.22736.objecter resend_mon_ops 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f660c015d80 con 0x7f66101d17c0 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 monclient: dump: 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.759 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f660c015d80 con 0x7f66101d17c0 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 msgr2=0x7f66100a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 0x7f66100a3310 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f660c011550 tx=0x7f660c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 msgr2=0x7f66100a3310 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6605ffb640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f66101d17c0 0x7f66100a3310 secure :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f660c011550 tx=0x7f660c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163254, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:59.761 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.762 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.763 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:59.764 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:59.765 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:59.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:59.768 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.770 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.771 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.773 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.776 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:59.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.779 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.782 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:59.783 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 10 librados: watch_flush enter 2024-04-19T00:46:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 10 librados: watch_flush exit 2024-04-19T00:46:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 20 client.22736.objecter shutdown clearing up homeless session... 2024-04-19T00:46:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 10 client.22736.objecter successfully canceled tick 2024-04-19T00:46:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65f4038e10 msgr2=0x7f65f403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65f4038e10 0x7f65f403b2d0 secure :-1 s=READY pgs=3343 cs=0 l=1 rev1=1 crypto rx=0x7f66000ad000 tx=0x7f66000a6000 comp rx=0 tx=0).stop 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 10 monclient: shutdown 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6610138a50 msgr2=0x7f66101d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 --2- 172.21.15.43:0/1995398985 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6610138a50 0x7f66101d0d40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f660800efc0 tx=0x7f660800c490 comp rx=0 tx=0).stop 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6615ef5640 1 -- 172.21.15.43:0/1995398985 reap_dead start 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6615ef5640 1 -- 172.21.15.43:0/1995398985 reap_dead start 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 shutdown_connections 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 >> 172.21.15.43:0/1995398985 conn(0x7f66100a7cb0 msgr2=0x7f66100a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 shutdown_connections 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 -- 172.21.15.43:0/1995398985 wait complete. 2024-04-19T00:46:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.734+0000 7f6617958640 1 librados: shutdown 2024-04-19T00:46:59.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:59.787 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:46:59.787 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient: get_monmap_and_config 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient: monmap: 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:46:59.906573+0000 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:46:59.906573+0000 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:46:59.905 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:46:59.906 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 Processor -- start 2024-04-19T00:46:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 -- start start 2024-04-19T00:46:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient: init 2024-04-19T00:46:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient: _add_conns ranks=[5,1,7,2,10,9,3,8,4,0,6] 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 0x7f1820072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): picked mon.noname-h con 0x7f18200729d0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1820139100 addr v1:172.21.15.161:6789/0 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18200a51b0 0x7f1820140f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): picked mon.noname-i con 0x7f18200a51b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f182006f830 con 0x7f1820139100 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f182006f270 con 0x7f18200729d0 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f182006df80 con 0x7f18200a51b0 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f18271f9640 10 monclient(hunting): authenticate will time out at 1090.033791s 2024-04-19T00:46:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 0x7f1820072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 0x7f1820072db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:39862/0 (socket says 172.21.15.43:39862) 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 -- 172.21.15.43:0/629834794 learned_addr learned my addr 172.21.15.43:0/629834794 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): get_auth_request con 0x7f18200729d0 auth_method 0 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 1 --2- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18200a51b0 0x7f1820140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): get_auth_request con 0x7f18200a51b0 auth_method 0 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1825796640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient(hunting): handle_auth_done global_id 44267 payload 274 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 -- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18200a51b0 msgr2=0x7f1820140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 --2- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18200a51b0 0x7f1820140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 -- 172.21.15.43:0/629834794 >> v1:172.21.15.161:6789/0 conn(0x7f1820139100 legacy=0x7f18200a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 monclient: found mon.noname-h 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:46:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.902+0000 7f1824f95640 1 -- 172.21.15.43:0/629834794 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18200732f0 con 0x7f18200729d0 2024-04-19T00:46:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f1824f95640 1 --2- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 0x7f1820072db0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1818009e80 tx=0x7f181801c910 comp rx=0 tx=0).ready entity=mon.5 client_cookie=6035d471db47283d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 1 -- 172.21.15.43:0/629834794 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18180090d0 con 0x7f18200729d0 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: dump: 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 1 -- 172.21.15.43:0/629834794 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18180a5a00 con 0x7f18200729d0 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 5 monclient: authenticate success, global_id 44267 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:46:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 1 -- 172.21.15.43:0/629834794 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18180a5bf0 con 0x7f18200729d0 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 10 monclient: get_monmap_and_config success 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 4 set_mon_vals no callback set 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f181f7fe640 10 monclient: dump: 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 10 monclient: shutdown 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 -- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 msgr2=0x7f1820072db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 --2- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 0x7f1820072db0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1818009e80 tx=0x7f181801c910 comp rx=0 tx=0).stop 2024-04-19T00:46:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 -- 172.21.15.43:0/629834794 shutdown_connections 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 --2- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18200a51b0 0x7f1820140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 --2- 172.21.15.43:0/629834794 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18200729d0 0x7f1820072db0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 -- 172.21.15.43:0/629834794 >> 172.21.15.43:0/629834794 conn(0x7f18200a6410 msgr2=0x7f182006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 -- 172.21.15.43:0/629834794 shutdown_connections 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 -- 172.21.15.43:0/629834794 wait complete. 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 10 monclient: build_initial_monmap 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 10 monclient: monmap: 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:46:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 librados: starting msgr at 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 librados: starting objecter 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 Processor -- start 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 -- start start 2024-04-19T00:46:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 librados: setting wanted keys 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 1 librados: calling monclient init 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.906+0000 7f18271f9640 10 monclient: init 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient: _reopen_session rank -1 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient: _add_conns ranks=[8,6,0,3,4,2,1,5,7] 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18200729d0 0x7f1820134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): picked mon.noname-e con 0x7f18200729d0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a51b0 0x7f1820135150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): picked mon.noname-d con 0x7f18200a51b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1820139100 0x7f1820135690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): picked mon.noname-a con 0x7f1820139100 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1820144020 con 0x7f1820139100 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1820143ea0 con 0x7f18200a51b0 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f18201441a0 con 0x7f18200729d0 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): _renew_subs 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): authenticate will time out at 1090.040558s 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18200729d0 0x7f1820134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1820139100 0x7f1820135690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18200729d0 0x7f1820134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54948/0 (socket says 172.21.15.43:54948) 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 -- 172.21.15.43:0/3494263403 learned_addr learned my addr 172.21.15.43:0/3494263403 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): get_auth_request con 0x7f18200729d0 auth_method 0 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): get_auth_request con 0x7f1820139100 auth_method 0 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a51b0 0x7f1820135150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): get_auth_request con 0x7f18200a51b0 auth_method 0 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient(hunting): handle_auth_done global_id 29655 payload 824 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18200729d0 msgr2=0x7f1820134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18200729d0 0x7f1820134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a51b0 msgr2=0x7f1820135150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a51b0 0x7f1820135150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient: _finish_hunting 0 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 monclient: found mon.noname-a 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182013a1d0 con 0x7f1820139100 2024-04-19T00:46:59.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18200729d0 0x7f1820134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1825796640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1820139100 0x7f1820135690 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f181000e7d0 tx=0x7f181000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 client.?.objecter ms_handle_connect 0x7f1820139100 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1810017070 con 0x7f1820139100 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: dump: 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.929 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181000cdb0 con 0x7f1820139100 2024-04-19T00:46:59.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 5 monclient: authenticate success, global_id 29655 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient: _renew_subs 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f182006e9b0 con 0x7f1820139100 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18100138e0 con 0x7f1820139100 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181effd640 4 set_mon_vals no callback set 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: dump: 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.931 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 client.29655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient: _renew_subs 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1820139ef0 con 0x7f1820139100 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 librados: init done 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 0x7f18200a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f182006d140 con 0x7f18200a0490 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 0x7f18200a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f181001a070 con 0x7f1820139100 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient: get_auth_request con 0x7f18200a0490 auth_method 0 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 10 monclient(hunting): handle_auth_done global_id 22776 payload 824 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f180c038e10 0x7f180c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1820136e70 con 0x7f18200a0490 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 8683+0+0 (secure 0 0 0) 0x7f181000f040 con 0x7f1820139100 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 client.29655.objecter ms_dispatch 0x7f1820000ba0 osd_map(105..105 src has 1..105) 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 3 client.29655.objecter handle_osd_map got epochs [105,105] > 0 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 3 client.29655.objecter handle_osd_map decoding full epoch 105 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f1824f95640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 0x7f18200a0870 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f181801d1b0 tx=0x7f18180196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 20 client.29655.objecter dump_active .. 0 homeless 2024-04-19T00:46:59.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 client.29655.objecter ms_handle_connect 0x7f18200a0490 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 client.29655.objecter resend_mon_ops 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18180bd070 con 0x7f18200a0490 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f180c038e10 0x7f180c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: dump: 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.935 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181ffff640 10 monclient: get_auth_request con 0x7f180c038e10 auth_method 0 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1818004040 con 0x7f18200a0490 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 msgr2=0x7f18200a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 0x7f18200a0870 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f181801d1b0 tx=0x7f18180196b0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 msgr2=0x7f18200a0870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.910+0000 7f181d7fa640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0490 0x7f18200a0870 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.914+0000 7f181d7fa640 10 client.29655.objecter ms_handle_connect 0x7f180c038e10 2024-04-19T00:46:59.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.914+0000 7f181ffff640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f180c038e10 0x7f180c03b2d0 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f1814007fd0 tx=0x7f1814009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:46:59.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 0x7f18201db100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1820140b60 con 0x7f18200a0db0 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 0x7f18201db100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient: get_auth_request con 0x7f18200a0db0 auth_method 0 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 10 monclient(hunting): handle_auth_done global_id 22785 payload 824 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 1 -- 172.21.15.43:0/3494263403 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18200008e0 con 0x7f18200a0db0 2024-04-19T00:46:59.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1824f95640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 0x7f18201db100 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f18180a6fd0 tx=0x7f18180a7000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 client.29655.objecter ms_handle_connect 0x7f18200a0db0 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 client.29655.objecter resend_mon_ops 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18180a5d50 con 0x7f18200a0db0 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 monclient: dump: 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:46:59.940 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f18180a5d50 con 0x7f18200a0db0 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 msgr2=0x7f18201db100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 0x7f18201db100 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f18180a6fd0 tx=0x7f18180a7000 comp rx=0 tx=0).stop 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 msgr2=0x7f18201db100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f181d7fa640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18200a0db0 0x7f18201db100 secure :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f18180a6fd0 tx=0x7f18180a7000 comp rx=0 tx=0).stop 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163434, 2024-04-19T00:46:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:46:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:46:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:46:59.948 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.949 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:46:59.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:46:59.954 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:46:59.956 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.958 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.959 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:46:59.961 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:46:59.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.963 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:46:59.964 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:46:59.965 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:46:59.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:46:59.965 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:46:59.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:46:59.965 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:46:59.965 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 10 librados: watch_flush enter 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 10 librados: watch_flush exit 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 20 client.29655.objecter shutdown clearing up homeless session... 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 10 client.29655.objecter successfully canceled tick 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f180c038e10 msgr2=0x7f180c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f180c038e10 0x7f180c03b2d0 secure :-1 s=READY pgs=3344 cs=0 l=1 rev1=1 crypto rx=0x7f1814007fd0 tx=0x7f1814009040 comp rx=0 tx=0).stop 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 10 monclient: shutdown 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1820139100 msgr2=0x7f1820135690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:46:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1820139100 0x7f1820135690 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f181000e7d0 tx=0x7f181000ec90 comp rx=0 tx=0).stop 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f1825796640 1 -- 172.21.15.43:0/3494263403 reap_dead start 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 shutdown_connections 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 --2- 172.21.15.43:0/3494263403 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1820139100 0x7f1820135690 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 >> 172.21.15.43:0/3494263403 conn(0x7f18200a6410 msgr2=0x7f1820131aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 shutdown_connections 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 -- 172.21.15.43:0/3494263403 wait complete. 2024-04-19T00:46:59.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:46:59.918+0000 7f18271f9640 1 librados: shutdown 2024-04-19T00:46:59.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:46:59.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:46:59.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient: get_monmap_and_config 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient: monmap: 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:00.094712+0000 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:00.094712+0000 2024-04-19T00:47:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:00.094 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 Processor -- start 2024-04-19T00:47:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 -- start start 2024-04-19T00:47:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient: init 2024-04-19T00:47:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient: _add_conns ranks=[9,8,7,0,4,2,5,6,1,3,10] 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 0x7fcc680a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): picked mon.noname-a con 0x7fcc680a1f20 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 0x7fcc680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): picked mon.noname-e con 0x7fcc680a1270 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a4430 0x7fcc6813fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): picked mon.noname-i con 0x7fcc680a4430 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fcc680a0920 con 0x7fcc680a1f20 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fcc6806bf40 con 0x7fcc680a4430 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcc6809fff0 con 0x7fcc680a1270 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc7025e640 10 monclient(hunting): authenticate will time out at 1090.221925s 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 0x7fcc680a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 0x7fcc680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 0x7fcc680a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55516/0 (socket says 172.21.15.43:55516) 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 0x7fcc680a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54952/0 (socket says 172.21.15.43:54952) 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 1 -- 172.21.15.43:0/2520742891 learned_addr learned my addr 172.21.15.43:0/2520742891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): get_auth_request con 0x7fcc680a1f20 auth_method 0 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): get_auth_request con 0x7fcc680a1270 auth_method 0 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6e7fb640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a4430 0x7fcc6813fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6e7fb640 10 monclient(hunting): get_auth_request con 0x7fcc680a4430 auth_method 0 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6e7fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6e7fb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6e7fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient(hunting): handle_auth_done global_id 22355 payload 274 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 -- 172.21.15.43:0/2520742891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a4430 msgr2=0x7fcc6813fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a4430 0x7fcc6813fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 -- 172.21.15.43:0/2520742891 >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 msgr2=0x7fcc680a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 --2- 172.21.15.43:0/2520742891 >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 0x7fcc680a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 monclient: found mon.noname-e 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6d7f9640 1 -- 172.21.15.43:0/2520742891 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc680a02d0 con 0x7fcc680a1270 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6dffa640 1 --2- 172.21.15.43:0/2520742891 >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 0x7fcc680a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.090+0000 7fcc6e7fb640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a4430 0x7fcc6813fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6d7f9640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 0x7fcc680a3ef0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fcc580097f0 tx=0x7fcc5801c360 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a2a354af178ebb9a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 1 -- 172.21.15.43:0/2520742891 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc580b0070 con 0x7fcc680a1270 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:00.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: dump: 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 1 -- 172.21.15.43:0/2520742891 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc580aa810 con 0x7fcc680a1270 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 5 monclient: authenticate success, global_id 22355 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 1 -- 172.21.15.43:0/2520742891 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc580a9300 con 0x7fcc680a1270 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 4 set_mon_vals no callback set 2024-04-19T00:47:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6cff8640 10 monclient: dump: 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.105 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: shutdown 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- 172.21.15.43:0/2520742891 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 msgr2=0x7fcc680a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 0x7fcc680a3ef0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fcc580097f0 tx=0x7fcc5801c360 comp rx=0 tx=0).stop 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- 172.21.15.43:0/2520742891 shutdown_connections 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a4430 0x7fcc6813fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- 172.21.15.43:0/2520742891 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a1270 0x7fcc680a3ef0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- 172.21.15.43:0/2520742891 >> v2:172.21.15.43:3300/0 conn(0x7fcc680a1f20 0x7fcc680a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- 172.21.15.43:0/2520742891 >> 172.21.15.43:0/2520742891 conn(0x7fcc680a85a0 msgr2=0x7fcc680a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- 172.21.15.43:0/2520742891 shutdown_connections 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- 172.21.15.43:0/2520742891 wait complete. 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: monmap: 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.114 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 librados: starting msgr at 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 librados: starting objecter 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 Processor -- start 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- start start 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 librados: setting wanted keys 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 librados: calling monclient init 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: init 2024-04-19T00:47:00.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient: _add_conns ranks=[3,7,8,2,1,5,4,6,0] 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcc680a1270 0x7fcc681d1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): picked mon.noname-g con 0x7fcc680a1270 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a1f20 0x7fcc68122a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): picked mon.noname-i con 0x7fcc680a1f20 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a4430 0x7fcc68122f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): picked mon.noname-e con 0x7fcc680a4430 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fcc680a2480 con 0x7fcc680a1270 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fcc680a2300 con 0x7fcc680a1f20 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcc68142900 con 0x7fcc680a4430 2024-04-19T00:47:00.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc7025e640 10 monclient(hunting): authenticate will time out at 1090.227177s 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a4430 0x7fcc68122f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a4430 0x7fcc68122f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54968/0 (socket says 172.21.15.43:54968) 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 -- 172.21.15.43:0/3750408433 learned_addr learned my addr 172.21.15.43:0/3750408433 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): get_auth_request con 0x7fcc680a4430 auth_method 0 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6d7f9640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a1f20 0x7fcc68122a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6dffa640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcc680a1270 0x7fcc681d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6d7f9640 10 monclient(hunting): get_auth_request con 0x7fcc680a1f20 auth_method 0 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6d7f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6d7f9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6d7f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6dffa640 10 monclient(hunting): get_auth_request con 0x7fcc680a1270 auth_method 0 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6dffa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6dffa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6dffa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient(hunting): handle_auth_done global_id 22364 payload 824 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a1f20 msgr2=0x7fcc68122a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcc680a1f20 0x7fcc68122a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcc680a1270 msgr2=0x7fcc681d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fcc680a1270 0x7fcc681d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 monclient: found mon.noname-e 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc68073100 con 0x7fcc680a4430 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc6e7fb640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a4430 0x7fcc68122f40 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcc60007e70 tx=0x7fcc6000d4a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc5e7fc640 10 client.?.objecter ms_handle_connect 0x7fcc680a4430 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.094+0000 7fcc5e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc6001a070 con 0x7fcc680a4430 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: dump: 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.119 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 5 monclient: authenticate success, global_id 22364 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc60003e10 con 0x7fcc680a4430 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc600044a0 con 0x7fcc680a4430 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 monclient: _renew_subs 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc6809f6c0 con 0x7fcc680a4430 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5ffff640 4 set_mon_vals no callback set 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: dump: 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.121 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.122 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 client.22364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 monclient: _renew_subs 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc681263b0 con 0x7fcc680a4430 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 1 librados: init done 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 0x7fcc68123480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcc6806d140 con 0x7fcc681d1550 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 0x7fcc68123480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient: get_auth_request con 0x7fcc681d1550 auth_method 0 2024-04-19T00:47:00.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fcc60004640 con 0x7fcc680a4430 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_done global_id 22373 payload 824 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc6809ef70 con 0x7fcc681d1550 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6dffa640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 0x7fcc68123480 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcc64011600 tx=0x7fcc64011ac0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcc4c038e10 0x7fcc4c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 client.22364.objecter ms_handle_connect 0x7fcc681d1550 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 client.22364.objecter resend_mon_ops 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(105..105 src has 1..105) ==== 8683+0+0 (secure 0 0 0) 0x7fcc60049ff0 con 0x7fcc680a4430 2024-04-19T00:47:00.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 client.22364.objecter ms_dispatch 0x7fcc68000ba0 osd_map(105..105 src has 1..105) 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 3 client.22364.objecter handle_osd_map got epochs [105,105] > 0 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 3 client.22364.objecter handle_osd_map decoding full epoch 105 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 20 client.22364.objecter dump_active .. 0 homeless 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc64019070 con 0x7fcc681d1550 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: dump: 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.125 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6d7f9640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcc4c038e10 0x7fcc4c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6d7f9640 10 monclient: get_auth_request con 0x7fcc4c038e10 auth_method 0 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcc64004240 con 0x7fcc681d1550 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 msgr2=0x7fcc68123480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 0x7fcc68123480 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcc64011600 tx=0x7fcc64011ac0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 msgr2=0x7fcc68123480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681d1550 0x7fcc68123480 secure :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcc64011600 tx=0x7fcc64011ac0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc5e7fc640 10 client.22364.objecter ms_handle_connect 0x7fcc4c038e10 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.098+0000 7fcc6d7f9640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcc4c038e10 0x7fcc4c03b2d0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7fcc580ad000 tx=0x7fcc580a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 0x7fcc6813bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcc68123dd0 con 0x7fcc681239c0 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 0x7fcc6813bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient: get_auth_request con 0x7fcc681239c0 auth_method 0 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 10 monclient(hunting): handle_auth_done global_id 22382 payload 824 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 1 -- 172.21.15.43:0/3750408433 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcc6813b090 con 0x7fcc681239c0 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 client.22364.objecter ms_handle_connect 0x7fcc681239c0 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 client.22364.objecter resend_mon_ops 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6dffa640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 0x7fcc6813bda0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcc64011600 tx=0x7fcc640046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcc64015ca0 con 0x7fcc681239c0 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 monclient: dump: 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.128 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fcc64015830 con 0x7fcc681239c0 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 msgr2=0x7fcc6813bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 0x7fcc6813bda0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcc64011600 tx=0x7fcc640046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 msgr2=0x7fcc6813bda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc5e7fc640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc681239c0 0x7fcc6813bda0 secure :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fcc64011600 tx=0x7fcc640046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:00.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163618, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:00.131 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.132 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.133 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.134 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.135 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.136 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.137 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.138 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.139 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.140 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.141 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.142 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.143 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.144 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:00.145 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.146 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.148 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:00.150 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.152 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.153 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.154 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.154 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.154 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:00.154 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 10 librados: watch_flush enter 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 10 librados: watch_flush exit 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 20 client.22364.objecter shutdown clearing up homeless session... 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 10 client.22364.objecter successfully canceled tick 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcc4c038e10 msgr2=0x7fcc4c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcc4c038e10 0x7fcc4c03b2d0 secure :-1 s=READY pgs=3345 cs=0 l=1 rev1=1 crypto rx=0x7fcc580ad000 tx=0x7fcc580a6000 comp rx=0 tx=0).stop 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 10 monclient: shutdown 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a4430 msgr2=0x7fcc68122f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 --2- 172.21.15.43:0/3750408433 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcc680a4430 0x7fcc68122f40 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcc60007e70 tx=0x7fcc6000d4a0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6e7fb640 1 -- 172.21.15.43:0/3750408433 reap_dead start 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc6e7fb640 1 -- 172.21.15.43:0/3750408433 reap_dead start 2024-04-19T00:47:00.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 shutdown_connections 2024-04-19T00:47:00.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 >> 172.21.15.43:0/3750408433 conn(0x7fcc680a85a0 msgr2=0x7fcc680a7570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 shutdown_connections 2024-04-19T00:47:00.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 -- 172.21.15.43:0/3750408433 wait complete. 2024-04-19T00:47:00.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.106+0000 7fcc7025e640 1 librados: shutdown 2024-04-19T00:47:00.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:47:00.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:47:00.157 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:47:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient: get_monmap_and_config 2024-04-19T00:47:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient: monmap: 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:00.275971+0000 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:00.275971+0000 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:00.276 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 Processor -- start 2024-04-19T00:47:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 -- start start 2024-04-19T00:47:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient: init 2024-04-19T00:47:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient: _add_conns ranks=[8,5,0,4,6,2,3,10,7,9,1] 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4138a50 0x7f6ef4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): picked mon.noname-e con 0x7f6ef4138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef4139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): picked mon.noname-h con 0x7f6ef40a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ef4137960 addr v1:172.21.15.43:6789/0 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef40a0d40 con 0x7f6ef4137960 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6ef40a1020 con 0x7f6ef40a5bb0 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6ef41398b0 con 0x7f6ef4138a50 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6efb986640 10 monclient(hunting): authenticate will time out at 1090.403595s 2024-04-19T00:47:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9f23640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6ef4137960 0x7f6ef4137dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50486/0 (socket says 172.21.15.43:50486) 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9f23640 1 -- 172.21.15.43:0/4279247959 learned_addr learned my addr 172.21.15.43:0/4279247959 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4138a50 0x7f6ef4138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): get_auth_request con 0x7f6ef4138a50 auth_method 0 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef8f21640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef4139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.270+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): get_auth_request con 0x7f6ef40a5bb0 auth_method 0 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 1 -- 172.21.15.43:0/4279247959 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4118577219 0 0) 0x7f6ef40a0d40 con 0x7f6ef4137960 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient(hunting): _negotiate 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient(hunting): my global_id is 29673 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 1 -- 172.21.15.43:0/4279247959 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ee0003530 con 0x7f6ef4137960 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 10 monclient(hunting): handle_auth_done global_id 22391 payload 274 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 1 -- 172.21.15.43:0/4279247959 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 msgr2=0x7f6ef4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef4139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 1 -- 172.21.15.43:0/4279247959 >> v1:172.21.15.43:6789/0 conn(0x7f6ef4137960 legacy=0x7f6ef4137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 1 monclient: found mon.noname-e 2024-04-19T00:47:00.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 1 -- 172.21.15.43:0/4279247959 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef40a0a60 con 0x7f6ef4138a50 2024-04-19T00:47:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6ef9722640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4138a50 0x7f6ef4138e30 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009dc0 tx=0x7f6ef001c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8680e721d78b49b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 1 -- 172.21.15.43:0/4279247959 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef00b2070 con 0x7f6ef4138a50 2024-04-19T00:47:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: dump: 2024-04-19T00:47:00.289 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.290 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 1 -- 172.21.15.43:0/4279247959 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef00a5a50 con 0x7f6ef4138a50 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 5 monclient: authenticate success, global_id 22391 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 4 set_mon_vals no callback set 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 1 -- 172.21.15.43:0/4279247959 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef00a5c40 con 0x7f6ef4138a50 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6eebfff640 10 monclient: dump: 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.292 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 10 monclient: shutdown 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 -- 172.21.15.43:0/4279247959 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4138a50 msgr2=0x7f6ef4138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4138a50 0x7f6ef4138e30 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0009dc0 tx=0x7f6ef001c850 comp rx=0 tx=0).stop 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 -- 172.21.15.43:0/4279247959 shutdown_connections 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef4139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 --2- 172.21.15.43:0/4279247959 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4138a50 0x7f6ef4138e30 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 -- 172.21.15.43:0/4279247959 >> 172.21.15.43:0/4279247959 conn(0x7f6ef40a7cb0 msgr2=0x7f6ef40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 -- 172.21.15.43:0/4279247959 shutdown_connections 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 1 -- 172.21.15.43:0/4279247959 wait complete. 2024-04-19T00:47:00.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.274+0000 7f6efb986640 10 monclient: monmap: 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.294 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 librados: starting msgr at 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 librados: starting objecter 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 Processor -- start 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- start start 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 librados: setting wanted keys 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 librados: calling monclient init 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: init 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: _add_conns ranks=[5,8,0,4,7,2,6,3,1] 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): picked mon.noname-h con 0x7f6ef40a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:00.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4137960 0x7f6ef409bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): picked mon.noname-e con 0x7f6ef4137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ef4138a50 0x7f6ef409c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef4138a50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef4143bb0 con 0x7f6ef4138a50 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6ef4143a30 con 0x7f6ef40a5bb0 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6ef4143d30 con 0x7f6ef4137960 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): authenticate will time out at 1090.408934s 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ef4138a50 0x7f6ef409c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4137960 0x7f6ef409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4137960 0x7f6ef409bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:55002/0 (socket says 172.21.15.43:55002) 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ef4138a50 0x7f6ef409c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55520/0 (socket says 172.21.15.43:55520) 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 -- 172.21.15.43:0/3319617528 learned_addr learned my addr 172.21.15.43:0/3319617528 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): get_auth_request con 0x7f6ef4137960 auth_method 0 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): get_auth_request con 0x7f6ef4138a50 auth_method 0 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): get_auth_request con 0x7f6ef40a5bb0 auth_method 0 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient(hunting): handle_auth_done global_id 22400 payload 824 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 msgr2=0x7f6ef409ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6ef40a5bb0 0x7f6ef409ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ef4138a50 msgr2=0x7f6ef409c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ef4138a50 0x7f6ef409c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 monclient: found mon.noname-e 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef409ee60 con 0x7f6ef4137960 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ef4138a50 0x7f6ef409c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef8f21640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4137960 0x7f6ef409bfb0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400e900 tx=0x7f6ee400edc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 client.?.objecter ms_handle_connect 0x7f6ef4137960 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ee4017070 con 0x7f6ef4137960 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: dump: 2024-04-19T00:47:00.299 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.300 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee4013470 con 0x7f6ef4137960 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ee4013a90 con 0x7f6ef4137960 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 5 monclient: authenticate success, global_id 22400 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: _renew_subs 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef409e5d0 con 0x7f6ef4137960 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6eeb7fe640 4 set_mon_vals no callback set 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 monclient: dump: 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.302 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 client.22400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: _renew_subs 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef409cc80 con 0x7f6ef4137960 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 librados: init done 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 0x7f6ef412cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef409e010 con 0x7f6ef412c8b0 2024-04-19T00:47:00.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6ee401a070 con 0x7f6ef4137960 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ed8038e10 0x7f6ed803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 0x7f6ef412cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(105..105 src has 1..105) ==== 8683+0+0 (secure 0 0 0) 0x7f6ee400f040 con 0x7f6ef4137960 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 10 client.22400.objecter ms_dispatch 0x7f6ef4000ba0 osd_map(105..105 src has 1..105) 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 3 client.22400.objecter handle_osd_map got epochs [105,105] > 0 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 3 client.22400.objecter handle_osd_map decoding full epoch 105 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient: get_auth_request con 0x7f6ef412c8b0 auth_method 0 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ee9ffb640 20 client.22400.objecter dump_active .. 0 homeless 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ed8038e10 0x7f6ed803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9f23640 10 monclient: get_auth_request con 0x7f6ed8038e10 auth_method 0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 10 monclient(hunting): handle_auth_done global_id 23338 payload 824 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.278+0000 7f6ef9722640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ef40008e0 con 0x7f6ef412c8b0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 client.22400.objecter ms_handle_connect 0x7f6ef412c8b0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 client.22400.objecter resend_mon_ops 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ef9722640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 0x7f6ef412cc90 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a7500 tx=0x7f6ef00195f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 client.22400.objecter ms_handle_connect 0x7f6ed8038e10 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ef9f23640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ed8038e10 0x7f6ed803b2d0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a4c00 tx=0x7f6eec007f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef00bd070 con 0x7f6ef412c8b0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 monclient: dump: 2024-04-19T00:47:00.305 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.306 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6ef00b3040 con 0x7f6ef412c8b0 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 msgr2=0x7f6ef412cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 0x7f6ef412cc90 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a7500 tx=0x7f6ef00195f0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 msgr2=0x7f6ef412cc90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.282+0000 7f6ee9ffb640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412c8b0 0x7f6ef412cc90 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6efb986640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6efb986640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 0x7f6ef41310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6efb986640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef406d140 con 0x7f6ef412d1d0 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 0x7f6ef41310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient: get_auth_request con 0x7f6ef412d1d0 auth_method 0 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.286+0000 7f6ef9722640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ef9722640 10 monclient(hunting): handle_auth_done global_id 23347 payload 824 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ef9722640 1 -- 172.21.15.43:0/3319617528 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ef412d5e0 con 0x7f6ef412d1d0 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ef9722640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 0x7f6ef41310c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0000f90 tx=0x7f6ef00a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 client.22400.objecter ms_handle_connect 0x7f6ef412d1d0 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 client.22400.objecter resend_mon_ops 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ef00bd070 con 0x7f6ef412d1d0 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:47:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 monclient: dump: 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.311 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f6ef00a55e0 con 0x7f6ef412d1d0 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 msgr2=0x7f6ef41310c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 0x7f6ef41310c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0000f90 tx=0x7f6ef00a7000 comp rx=0 tx=0).stop 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 msgr2=0x7f6ef41310c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ee9ffb640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6ef412d1d0 0x7f6ef41310c0 secure :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0000f90 tx=0x7f6ef00a7000 comp rx=0 tx=0).stop 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163760, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:00.313 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.314 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.315 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:00.316 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.317 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.318 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.320 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.321 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.322 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.323 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:00.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.327 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:00.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.334 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.335 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.336 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:47:00.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.336 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.336 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:00.336 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:00.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 10 librados: watch_flush enter 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 10 librados: watch_flush exit 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 20 client.22400.objecter shutdown clearing up homeless session... 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 10 client.22400.objecter successfully canceled tick 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ed8038e10 msgr2=0x7f6ed803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ed8038e10 0x7f6ed803b2d0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f6ef40a4c00 tx=0x7f6eec007f30 comp rx=0 tx=0).stop 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 10 monclient: shutdown 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4137960 msgr2=0x7f6ef409bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 --2- 172.21.15.43:0/3319617528 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6ef4137960 0x7f6ef409bfb0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400e900 tx=0x7f6ee400edc0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ef9f23640 1 -- 172.21.15.43:0/3319617528 reap_dead start 2024-04-19T00:47:00.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6ef9f23640 1 -- 172.21.15.43:0/3319617528 reap_dead start 2024-04-19T00:47:00.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 shutdown_connections 2024-04-19T00:47:00.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 >> 172.21.15.43:0/3319617528 conn(0x7f6ef40a7cb0 msgr2=0x7f6ef40a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 shutdown_connections 2024-04-19T00:47:00.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 -- 172.21.15.43:0/3319617528 wait complete. 2024-04-19T00:47:00.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.290+0000 7f6efb986640 1 librados: shutdown 2024-04-19T00:47:00.339 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:47:00.339 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:47:00.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient: get_monmap_and_config 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient: monmap: 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:00.466756+0000 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:00.466756+0000 2024-04-19T00:47:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:00.466 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 Processor -- start 2024-04-19T00:47:00.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 -- start start 2024-04-19T00:47:00.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient: init 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient: _add_conns ranks=[8,2,3,9,4,1,6,5,0,7,10] 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34f8138a50 0x7f34f8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): picked mon.noname-e con 0x7f34f8138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 0x7f34f8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): picked mon.noname-b con 0x7f34f80a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8137960 0x7f34f8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): picked mon.noname-g con 0x7f34f8137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f34f80a1020 con 0x7f34f80a5bb0 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f34f81398b0 con 0x7f34f8137960 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f34f80a0780 con 0x7f34f8138a50 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f35005c7640 10 monclient(hunting): authenticate will time out at 1090.593972s 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 0x7f34f8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34f8138a50 0x7f34f8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 0x7f34f8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37394/0 (socket says 172.21.15.43:37394) 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 -- 172.21.15.43:0/1402028632 learned_addr learned my addr 172.21.15.43:0/1402028632 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): get_auth_request con 0x7f34f80a5bb0 auth_method 0 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): get_auth_request con 0x7f34f8138a50 auth_method 0 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34feb64640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8137960 0x7f34f8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient(hunting): handle_auth_done global_id 22745 payload 274 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 -- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34f8138a50 msgr2=0x7f34f8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34f8138a50 0x7f34f8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 -- 172.21.15.43:0/1402028632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8137960 msgr2=0x7f34f8137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8137960 0x7f34f8137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 monclient: found mon.noname-b 2024-04-19T00:47:00.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:47:00.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34fdb62640 1 -- 172.21.15.43:0/1402028632 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f80a0d40 con 0x7f34f80a5bb0 2024-04-19T00:47:00.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34feb64640 10 monclient: get_auth_request con 0x7f34f8137960 auth_method 0 2024-04-19T00:47:00.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.462+0000 7f34feb64640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8137960 0x7f34f8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 0x7f34f8139370 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f34e80098a0 tx=0x7f34e801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4b1ced5b8643ae server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 1 -- 172.21.15.43:0/1402028632 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34e80b0070 con 0x7f34f80a5bb0 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: dump: 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.479 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 1 -- 172.21.15.43:0/1402028632 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e80044a0 con 0x7f34f80a5bb0 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 5 monclient: authenticate success, global_id 22745 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 1 -- 172.21.15.43:0/1402028632 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34e8003a90 con 0x7f34f80a5bb0 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 4 set_mon_vals no callback set 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fd361640 10 monclient: dump: 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: shutdown 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 msgr2=0x7f34f8139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 0x7f34f8139370 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f34e80098a0 tx=0x7f34e801c410 comp rx=0 tx=0).stop 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- 172.21.15.43:0/1402028632 shutdown_connections 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8137960 0x7f34f8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f80a5bb0 0x7f34f8139370 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- 172.21.15.43:0/1402028632 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f34f8138a50 0x7f34f8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- 172.21.15.43:0/1402028632 >> 172.21.15.43:0/1402028632 conn(0x7f34f80a7cb0 msgr2=0x7f34f80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- 172.21.15.43:0/1402028632 shutdown_connections 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- 172.21.15.43:0/1402028632 wait complete. 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: monmap: 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:00.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 librados: starting msgr at 2024-04-19T00:47:00.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 librados: starting objecter 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 Processor -- start 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- start start 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 librados: setting wanted keys 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 librados: calling monclient init 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: init 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient: _add_conns ranks=[1,0,2,8,3,4,6,5,7] 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34f80a5bb0 0x7f34f81d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): picked mon.noname-f con 0x7f34f80a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34f8137960 0x7f34f81369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): picked mon.noname-a con 0x7f34f8137960 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f8138a50 0x7f34f813c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): picked mon.noname-b con 0x7f34f8138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f34f80a2100 con 0x7f34f8137960 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f34f80a1f80 con 0x7f34f80a5bb0 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f34f8142810 con 0x7f34f8138a50 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f35005c7640 10 monclient(hunting): authenticate will time out at 1090.599442s 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f8138a50 0x7f34f813c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34f8137960 0x7f34f81369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f8138a50 0x7f34f813c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37396/0 (socket says 172.21.15.43:37396) 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34f8137960 0x7f34f81369c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55534/0 (socket says 172.21.15.43:55534) 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 1 -- 172.21.15.43:0/3073038500 learned_addr learned my addr 172.21.15.43:0/3073038500 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): get_auth_request con 0x7f34f8138a50 auth_method 0 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): get_auth_request con 0x7f34f8137960 auth_method 0 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fdb62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fe363640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34f80a5bb0 0x7f34f81d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fe363640 10 monclient(hunting): get_auth_request con 0x7f34f80a5bb0 auth_method 0 2024-04-19T00:47:00.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fe363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fe363640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34fe363640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 10 monclient(hunting): handle_auth_done global_id 22754 payload 824 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.466+0000 7f34feb64640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34f80a5bb0 msgr2=0x7f34f81d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f34f80a5bb0 0x7f34f81d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34f8137960 msgr2=0x7f34f81369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34f8137960 0x7f34f81369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 1 monclient: found mon.noname-b 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f8122080 con 0x7f34f8138a50 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fdb62640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f34f8137960 0x7f34f81369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34feb64640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f8138a50 0x7f34f813c730 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f34f000cf00 tx=0x7f34f000a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 client.?.objecter ms_handle_connect 0x7f34f8138a50 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34f000dab0 con 0x7f34f8138a50 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: dump: 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.489 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f000dc90 con 0x7f34f8138a50 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34f000de40 con 0x7f34f8138a50 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 5 monclient: authenticate success, global_id 22754 2024-04-19T00:47:00.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 monclient: _renew_subs 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f8121800 con 0x7f34f8138a50 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34effff640 4 set_mon_vals no callback set 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: dump: 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 client.22754.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 monclient: _renew_subs 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f8121ac0 con 0x7f34f8138a50 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 1 librados: init done 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 0x7f34f813cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f34f806d140 con 0x7f34f8136f00 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f34f001c020 con 0x7f34f8138a50 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 0x7f34f813cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient: get_auth_request con 0x7f34f8136f00 auth_method 0 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34dc038e10 0x7f34dc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 8683+0+0 (secure 0 0 0) 0x7f34f004a610 con 0x7f34f8138a50 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 client.22754.objecter ms_dispatch 0x7f34f8000ba0 osd_map(105..105 src has 1..105) 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 3 client.22754.objecter handle_osd_map got epochs [105,105] > 0 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 3 client.22754.objecter handle_osd_map decoding full epoch 105 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 20 client.22754.objecter dump_active .. 0 homeless 2024-04-19T00:47:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fdb62640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34dc038e10 0x7f34dc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fdb62640 10 monclient: get_auth_request con 0x7f34dc038e10 auth_method 0 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 10 monclient(hunting): handle_auth_done global_id 18480 payload 824 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34f809ee60 con 0x7f34f8136f00 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 client.22754.objecter ms_handle_connect 0x7f34f8136f00 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fe363640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 0x7f34f813cc70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f34f4011490 tx=0x7f34f4011950 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 client.22754.objecter resend_mon_ops 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 client.22754.objecter ms_handle_connect 0x7f34dc038e10 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34fdb62640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34dc038e10 0x7f34dc03b2d0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f34e80ad000 tx=0x7f34e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34f40044a0 con 0x7f34f8136f00 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: dump: 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.495 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f34f4004030 con 0x7f34f8136f00 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 msgr2=0x7f34f813cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 0x7f34f813cc70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f34f4011490 tx=0x7f34f4011950 comp rx=0 tx=0).stop 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 msgr2=0x7f34f813cc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.470+0000 7f34eeffd640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f8136f00 0x7f34f813cc70 secure :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f34f4011490 tx=0x7f34f4011950 comp rx=0 tx=0).stop 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f35005c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f35005c7640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 0x7f34f813d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f35005c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f34f8121cc0 con 0x7f34f813d1b0 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 0x7f34f813d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient: get_auth_request con 0x7f34f813d1b0 auth_method 0 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 10 monclient(hunting): handle_auth_done global_id 18489 payload 824 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 1 -- 172.21.15.43:0/3073038500 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f34f813db00 con 0x7f34f813d1b0 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34eeffd640 10 client.22754.objecter ms_handle_connect 0x7f34f813d1b0 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34fe363640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 0x7f34f813d5c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f34f4011490 tx=0x7f34f40046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34eeffd640 10 client.22754.objecter resend_mon_ops 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34f4003d80 con 0x7f34f813d1b0 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34eeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.478+0000 7f34eeffd640 10 monclient: dump: 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.499 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f34f4003910 con 0x7f34f813d1b0 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:00.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 msgr2=0x7f34f813d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 0x7f34f813d5c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f34f4011490 tx=0x7f34f40046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 msgr2=0x7f34f813d5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34eeffd640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f34f813d1b0 0x7f34f813d5c0 secure :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f34f4011490 tx=0x7f34f40046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163964, 2024-04-19T00:47:00.501 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:00.502 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:00.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.504 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:00.505 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.506 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.508 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.511 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.513 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.516 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.518 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.521 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.522 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:00.523 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 10 librados: watch_flush enter 2024-04-19T00:47:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 10 librados: watch_flush exit 2024-04-19T00:47:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 20 client.22754.objecter shutdown clearing up homeless session... 2024-04-19T00:47:00.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 10 client.22754.objecter successfully canceled tick 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34dc038e10 msgr2=0x7f34dc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f34dc038e10 0x7f34dc03b2d0 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f34e80ad000 tx=0x7f34e8009040 comp rx=0 tx=0).stop 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 10 monclient: shutdown 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f8138a50 msgr2=0x7f34f813c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 --2- 172.21.15.43:0/3073038500 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f34f8138a50 0x7f34f813c730 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f34f000cf00 tx=0x7f34f000a710 comp rx=0 tx=0).stop 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34feb64640 1 -- 172.21.15.43:0/3073038500 reap_dead start 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f34feb64640 1 -- 172.21.15.43:0/3073038500 reap_dead start 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 shutdown_connections 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 >> 172.21.15.43:0/3073038500 conn(0x7f34f80a7cb0 msgr2=0x7f34f80a26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 shutdown_connections 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 -- 172.21.15.43:0/3073038500 wait complete. 2024-04-19T00:47:00.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.482+0000 7f35005c7640 1 librados: shutdown 2024-04-19T00:47:00.527 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:47:00.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:47:00.527 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:47:00.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient: get_monmap_and_config 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient: monmap: 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:00.647324+0000 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:00.647324+0000 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:00.646 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:00.647 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 Processor -- start 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 -- start start 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient: init 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient: _add_conns ranks=[9,2,7,1,5,10,0,4,3,8,6] 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7000138a50 0x7f7000138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): picked mon.noname-a con 0x7f7000138a50 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 0x7f7000139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): picked mon.noname-b con 0x7f70000a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7000137960 0x7f7000137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): picked mon.noname-i con 0x7f7000137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:00.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f70000a1020 con 0x7f7000138a50 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f70001398b0 con 0x7f70000a5bb0 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f70000a0780 con 0x7f7000137960 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f70081cc640 10 monclient(hunting): authenticate will time out at 1090.774950s 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 0x7f7000139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 0x7f7000139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:37398/0 (socket says 172.21.15.43:37398) 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 -- 172.21.15.43:0/360712667 learned_addr learned my addr 172.21.15.43:0/360712667 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): get_auth_request con 0x7f70000a5bb0 auth_method 0 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 1 --2- 172.21.15.43:0/360712667 >> v2:172.21.15.43:3300/0 conn(0x7f7000138a50 0x7f7000138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): get_auth_request con 0x7f7000138a50 auth_method 0 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 1 --2- 172.21.15.43:0/360712667 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7000137960 0x7f7000137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): get_auth_request con 0x7f7000137960 auth_method 0 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7006769640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient(hunting): handle_auth_done global_id 22763 payload 274 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 -- 172.21.15.43:0/360712667 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7000137960 msgr2=0x7f7000137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 --2- 172.21.15.43:0/360712667 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7000137960 0x7f7000137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 -- 172.21.15.43:0/360712667 >> v2:172.21.15.43:3300/0 conn(0x7f7000138a50 msgr2=0x7f7000138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 --2- 172.21.15.43:0/360712667 >> v2:172.21.15.43:3300/0 conn(0x7f7000138a50 0x7f7000138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 monclient: found mon.noname-b 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:47:00.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005767640 1 -- 172.21.15.43:0/360712667 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70000a0d40 con 0x7f70000a5bb0 2024-04-19T00:47:00.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.642+0000 7f7005f68640 1 --2- 172.21.15.43:0/360712667 >> v2:172.21.15.43:3300/0 conn(0x7f7000138a50 0x7f7000138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7005767640 1 --2- 172.21.15.43:0/360712667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 0x7f7000139370 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009870 tx=0x7f6ff001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=540208ce531993f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 1 -- 172.21.15.43:0/360712667 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ff00b0070 con 0x7f70000a5bb0 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: dump: 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.659 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 1 -- 172.21.15.43:0/360712667 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff00aa810 con 0x7f70000a5bb0 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 5 monclient: authenticate success, global_id 22763 2024-04-19T00:47:00.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 1 -- 172.21.15.43:0/360712667 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ff00a9300 con 0x7f70000a5bb0 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 4 set_mon_vals no callback set 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f7004f66640 10 monclient: dump: 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.661 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 10 monclient: shutdown 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 -- 172.21.15.43:0/360712667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 msgr2=0x7f7000139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 --2- 172.21.15.43:0/360712667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 0x7f7000139370 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009870 tx=0x7f6ff001c3f0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 -- 172.21.15.43:0/360712667 shutdown_connections 2024-04-19T00:47:00.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 --2- 172.21.15.43:0/360712667 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7000137960 0x7f7000137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 --2- 172.21.15.43:0/360712667 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70000a5bb0 0x7f7000139370 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 --2- 172.21.15.43:0/360712667 >> v2:172.21.15.43:3300/0 conn(0x7f7000138a50 0x7f7000138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 -- 172.21.15.43:0/360712667 >> 172.21.15.43:0/360712667 conn(0x7f70000a7cb0 msgr2=0x7f70000a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 -- 172.21.15.43:0/360712667 shutdown_connections 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 -- 172.21.15.43:0/360712667 wait complete. 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 10 monclient: monmap: 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:00.663 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 librados: starting msgr at 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 librados: starting objecter 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 Processor -- start 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 -- start start 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 librados: setting wanted keys 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 1 librados: calling monclient init 2024-04-19T00:47:00.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.646+0000 7f70081cc640 10 monclient: init 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient: _add_conns ranks=[3,8,4,2,7,5,0,6,1] 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70000a5bb0 0x7f70001d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): picked mon.noname-g con 0x7f70000a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7000137960 0x7f7000136a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): picked mon.noname-e con 0x7f7000137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7000138a50 0x7f7000136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): picked mon.noname-c con 0x7f7000138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70000a2100 con 0x7f70000a5bb0 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f70000a1f80 con 0x7f7000138a50 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7000142810 con 0x7f7000137960 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): authenticate will time out at 1090.780630s 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7000137960 0x7f7000136a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7006769640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7000138a50 0x7f7000136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7000137960 0x7f7000136a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:55024/0 (socket says 172.21.15.43:55024) 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 -- 172.21.15.43:0/3557749444 learned_addr learned my addr 172.21.15.43:0/3557749444 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): get_auth_request con 0x7f7000137960 auth_method 0 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7006769640 10 monclient(hunting): get_auth_request con 0x7f7000138a50 auth_method 0 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7006769640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7006769640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7006769640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient(hunting): handle_auth_done global_id 22418 payload 824 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005f68640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70000a5bb0 0x7f70001d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7000138a50 msgr2=0x7f7000136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7000138a50 0x7f7000136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70000a5bb0 msgr2=0x7f70001d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70000a5bb0 0x7f70001d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 monclient: found mon.noname-e 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70001223a0 con 0x7f7000137960 2024-04-19T00:47:00.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005f68640 10 monclient: get_auth_request con 0x7f70000a5bb0 auth_method 0 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005f68640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70000a5bb0 0x7f70001d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f7005767640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7000137960 0x7f7000136a00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00ae040 tx=0x7f6ff0019190 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 client.?.objecter ms_handle_connect 0x7f7000137960 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ff00b9070 con 0x7f7000137960 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: dump: 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.668 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 5 monclient: authenticate success, global_id 22418 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient: _renew_subs 2024-04-19T00:47:00.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f700009ee60 con 0x7f7000137960 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff00aaca0 con 0x7f7000137960 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ff00b0040 con 0x7f7000137960 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff7fff640 4 set_mon_vals no callback set 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 client.22418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 monclient: dump: 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.670 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient: _renew_subs 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f7000121b00 con 0x7f7000137960 2024-04-19T00:47:00.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 librados: init done 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 0x7f700013c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f700006d140 con 0x7f7000121d40 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6ff0004040 con 0x7f7000137960 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6fe4038e10 0x7f6fe403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(105..105 src has 1..105) ==== 8683+0+0 (secure 0 0 0) 0x7f6ff00e3280 con 0x7f7000137960 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 10 client.22418.objecter ms_dispatch 0x7f7000000ba0 osd_map(105..105 src has 1..105) 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 3 client.22418.objecter handle_osd_map got epochs [105,105] > 0 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 3 client.22418.objecter handle_osd_map decoding full epoch 105 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.650+0000 7f6ff67fc640 20 client.22418.objecter dump_active .. 0 homeless 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7006769640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6fe4038e10 0x7f6fe403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7006769640 10 monclient: get_auth_request con 0x7f6fe4038e10 auth_method 0 2024-04-19T00:47:00.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 0x7f700013c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient: get_auth_request con 0x7f7000121d40 auth_method 0 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.654+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f7005f68640 10 monclient(hunting): handle_auth_done global_id 44294 payload 824 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f7005f68640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70000a55e0 con 0x7f7000121d40 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f7006769640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6fe4038e10 0x7f6fe403b2d0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f7000122890 tx=0x7f6ff800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f7005f68640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 0x7f700013c7a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0107c0 tx=0x7f6ffc010c80 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f6ff67fc640 10 client.22418.objecter ms_handle_connect 0x7f6fe4038e10 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f6ff67fc640 10 client.22418.objecter ms_handle_connect 0x7f7000121d40 2024-04-19T00:47:00.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.658+0000 7f6ff67fc640 10 client.22418.objecter resend_mon_ops 2024-04-19T00:47:00.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ffc00f040 con 0x7f7000121d40 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 10 monclient: dump: 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.683 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.684 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.684 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.684 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.684 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.684 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6ffc01c400 con 0x7f7000121d40 2024-04-19T00:47:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 msgr2=0x7f700013c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 0x7f700013c7a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0107c0 tx=0x7f6ffc010c80 comp rx=0 tx=0).stop 2024-04-19T00:47:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 msgr2=0x7f700013c7a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.678+0000 7f6ff67fc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7000121d40 0x7f700013c7a0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0107c0 tx=0x7f6ffc010c80 comp rx=0 tx=0).stop 2024-04-19T00:47:00.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f70081cc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:00.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f70081cc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 0x7f700013d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f70081cc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f700013d630 con 0x7f700013cce0 2024-04-19T00:47:00.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 0x7f700013d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient: get_auth_request con 0x7f700013cce0 auth_method 0 2024-04-19T00:47:00.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 10 monclient(hunting): handle_auth_done global_id 44303 payload 824 2024-04-19T00:47:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.690+0000 7f7005f68640 1 -- 172.21.15.43:0/3557749444 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7000121800 con 0x7f700013cce0 2024-04-19T00:47:00.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f7005f68640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 0x7f700013d0f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0107c0 tx=0x7f6ffc0046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 client.22418.objecter ms_handle_connect 0x7f700013cce0 2024-04-19T00:47:00.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 client.22418.objecter resend_mon_ops 2024-04-19T00:47:00.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ffc0219e0 con 0x7f700013cce0 2024-04-19T00:47:00.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 monclient: dump: 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.698 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f6ffc029410 con 0x7f700013cce0 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 msgr2=0x7f700013d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 0x7f700013d0f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0107c0 tx=0x7f6ffc0046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 msgr2=0x7f700013d0f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f6ff67fc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f700013cce0 0x7f700013d0f0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0107c0 tx=0x7f6ffc0046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.699 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 20423, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:00.700 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.701 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.702 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:00.703 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.704 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:00.705 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.708 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:00.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.713 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.716 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:00.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.720 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:00.721 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 10 librados: watch_flush enter 2024-04-19T00:47:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 10 librados: watch_flush exit 2024-04-19T00:47:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 20 client.22418.objecter shutdown clearing up homeless session... 2024-04-19T00:47:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 10 client.22418.objecter successfully canceled tick 2024-04-19T00:47:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6fe4038e10 msgr2=0x7f6fe403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6fe4038e10 0x7f6fe403b2d0 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7f7000122890 tx=0x7f6ff800b040 comp rx=0 tx=0).stop 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 10 monclient: shutdown 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7000137960 msgr2=0x7f7000136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 --2- 172.21.15.43:0/3557749444 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7000137960 0x7f7000136a00 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00ae040 tx=0x7f6ff0019190 comp rx=0 tx=0).stop 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f7006769640 1 -- 172.21.15.43:0/3557749444 reap_dead start 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f7006769640 1 -- 172.21.15.43:0/3557749444 reap_dead start 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 shutdown_connections 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 >> 172.21.15.43:0/3557749444 conn(0x7f70000a7cb0 msgr2=0x7f700006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 shutdown_connections 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 -- 172.21.15.43:0/3557749444 wait complete. 2024-04-19T00:47:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.694+0000 7f70081cc640 1 librados: shutdown 2024-04-19T00:47:00.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:47:00.725 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:47:00.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:47:00.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient: get_monmap_and_config 2024-04-19T00:47:00.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient: monmap: 2024-04-19T00:47:00.842 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.842 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:00.843896+0000 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:00.843896+0000 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:00.844 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 1 Processor -- start 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 1 -- start start 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient: init 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient: _add_conns ranks=[0,7,1,4,9,5,2,10,6,3,8] 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb04c138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 0x7fb04c13e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient(hunting): picked mon.noname-i con 0x7fb04c0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb04c1397c0 addr v1:172.21.15.161:6789/0 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04c0a3dc0 con 0x7fb04c138cd0 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04c0a40a0 con 0x7fb04c1397c0 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb04c0a4410 con 0x7fb04c0a5b80 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb0521a5640 10 monclient(hunting): authenticate will time out at 1090.971195s 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04b7fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb04c138cd0 0x7fb04c1390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:50494/0 (socket says 172.21.15.43:50494) 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04b7fe640 1 -- 172.21.15.43:0/3064705142 learned_addr learned my addr 172.21.15.43:0/3064705142 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 0x7fb04c13e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): get_auth_request con 0x7fb04c0a5b80 auth_method 0 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 298172428 0 0) 0x7fb04c0a3dc0 con 0x7fb04c138cd0 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient(hunting): _negotiate 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient(hunting): my global_id is 29709 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb034003530 con 0x7fb04c138cd0 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3129403129 0 0) 0x7fb034003530 con 0x7fb04c138cd0 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 msgr2=0x7fb04c13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 0x7fb04c13e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 >> v1:172.21.15.161:6789/0 conn(0x7fb04c1397c0 legacy=0x7fb04c1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:47:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c0a2ae0 con 0x7fb04c138cd0 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.838+0000 7fb04a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fb040001970 con 0x7fb04c138cd0 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: dump: 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:47:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 >> v1:172.21.15.43:6789/0 conn(0x7fb04c138cd0 legacy=0x7fb04c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: _add_conns ranks=[0,7,5,3,6,1,2,4,8] 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb034004240 0x7fb034004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): picked mon.a con 0x7fb034004240 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 0x7fb034004e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): picked mon.i con 0x7fb04c0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb0340099c0 0x7fb03400bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): picked mon.h con 0x7fb0340099c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c0a4410 con 0x7fb034004240 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb03400c420 con 0x7fb0340099c0 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb03400c5d0 con 0x7fb04c0a5b80 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb034004240 0x7fb034004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): get_auth_request con 0x7fb034004240 auth_method 0 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04affd640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 0x7fb034004e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04bfff640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb0340099c0 0x7fb03400bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04affd640 10 monclient(hunting): get_auth_request con 0x7fb04c0a5b80 auth_method 0 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_done global_id 29709 payload 293 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 -- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 msgr2=0x7fb034004e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c0a5b80 0x7fb034004e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 -- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb0340099c0 msgr2=0x7fb03400bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb0340099c0 0x7fb03400bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 monclient: found mon.a 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 -- 172.21.15.43:0/3064705142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb034003530 con 0x7fb034004240 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04bfff640 10 monclient: get_auth_request con 0x7fb0340099c0 auth_method 0 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04bfff640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb0340099c0 0x7fb03400bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 5 monclient: authenticate success, global_id 29709 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb034004240 0x7fb034004810 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fb040007f00 tx=0x7fb04001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d53f3eb9016b65e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0400ac070 con 0x7fb034004240 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: dump: 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb040005d50 con 0x7fb034004240 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 1 -- 172.21.15.43:0/3064705142 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb040007b20 con 0x7fb034004240 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 4 set_mon_vals no callback set 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04a7fc640 10 monclient: dump: 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 10 monclient: shutdown 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 1 -- 172.21.15.43:0/3064705142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb034004240 msgr2=0x7fb034004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 1 --2- 172.21.15.43:0/3064705142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb034004240 0x7fb034004810 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7fb040007f00 tx=0x7fb04001c900 comp rx=0 tx=0).stop 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb04bfff640 1 -- 172.21.15.43:0/3064705142 reap_dead start 2024-04-19T00:47:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 1 -- 172.21.15.43:0/3064705142 shutdown_connections 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 1 -- 172.21.15.43:0/3064705142 >> 172.21.15.43:0/3064705142 conn(0x7fb04c0ab2f0 msgr2=0x7fb04c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 1 -- 172.21.15.43:0/3064705142 shutdown_connections 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.842+0000 7fb0521a5640 1 -- 172.21.15.43:0/3064705142 wait complete. 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: build_initial_monmap 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: monmap: 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 librados: starting msgr at 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 librados: starting objecter 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 Processor -- start 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- start start 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 librados: setting wanted keys 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 librados: calling monclient init 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: init 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: _add_conns ranks=[3,8,5,7,0,4,6,2,1] 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c1d0c20 0x7fb04c137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): picked mon.noname-g con 0x7fb04c1d0c20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb04c138210 0x7fb04c09fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): picked mon.noname-e con 0x7fb04c138210 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb04c0a00c0 0x7fb04c0a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): picked mon.noname-h con 0x7fb04c0a00c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb04c142900 con 0x7fb04c1d0c20 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb04c0a20a0 con 0x7fb04c0a00c0 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb04c142a80 con 0x7fb04c138210 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): authenticate will time out at 1090.977180s 2024-04-19T00:47:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb04c138210 0x7fb04c09fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb04c138210 0x7fb04c09fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:55036/0 (socket says 172.21.15.43:55036) 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 -- 172.21.15.43:0/2071135887 learned_addr learned my addr 172.21.15.43:0/2071135887 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): get_auth_request con 0x7fb04c138210 auth_method 0 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb04c0a00c0 0x7fb04c0a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c1d0c20 0x7fb04c137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 10 monclient(hunting): get_auth_request con 0x7fb04c0a00c0 auth_method 0 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): get_auth_request con 0x7fb04c1d0c20 auth_method 0 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient(hunting): handle_auth_done global_id 22427 payload 824 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb04c0a00c0 msgr2=0x7fb04c0a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb04c0a00c0 0x7fb04c0a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c1d0c20 msgr2=0x7fb04c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb04c1d0c20 0x7fb04c137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient: _finish_hunting 0 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 monclient: found mon.noname-e 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c0a3040 con 0x7fb04c138210 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb04c0a00c0 0x7fb04c0a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04affd640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb04c138210 0x7fb04c09fb80 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb03c011630 tx=0x7fb03c011af0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 client.?.objecter ms_handle_connect 0x7fb04c138210 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb03c00fb90 con 0x7fb04c138210 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: dump: 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.871 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: _finish_auth 0 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: _check_auth_tickets 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03c00fd70 con 0x7fb04c138210 2024-04-19T00:47:00.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb03c0158d0 con 0x7fb04c138210 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 5 monclient: authenticate success, global_id 22427 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: _renew_subs 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04c0a1820 con 0x7fb04c138210 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb049ffb640 4 set_mon_vals no callback set 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 client.22427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 monclient: dump: 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.873 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: _renew_subs 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:00.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb04c123670 con 0x7fb04c138210 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 librados: init done 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 0x7fb04c09be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb04c06d140 con 0x7fb04c09ba70 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb03c015460 con 0x7fb04c138210 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb038038e10 0x7fb03803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 0x7fb04c09be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(106..106 src has 1..106) ==== 8307+0+0 (secure 0 0 0) 0x7fb03c04a6b0 con 0x7fb04c138210 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient: get_auth_request con 0x7fb04c09ba70 auth_method 0 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 10 client.22427.objecter ms_dispatch 0x7fb04c000ba0 osd_map(106..106 src has 1..106) 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 3 client.22427.objecter handle_osd_map got epochs [106,106] > 0 2024-04-19T00:47:00.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 3 client.22427.objecter handle_osd_map decoding full epoch 106 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb033fff640 20 client.22427.objecter dump_active .. 0 homeless 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb038038e10 0x7fb03803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04bfff640 10 monclient: get_auth_request con 0x7fb038038e10 auth_method 0 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.846+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_done global_id 18961 payload 824 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb04b7fe640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb04c1d1000 con 0x7fb04c09ba70 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 0x7fb04c09be50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb040009000 tx=0x7fb034005840 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 client.22427.objecter ms_handle_connect 0x7fb04c09ba70 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 client.22427.objecter resend_mon_ops 2024-04-19T00:47:00.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 client.22427.objecter ms_handle_connect 0x7fb038038e10 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb04bfff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb038038e10 0x7fb03803b2d0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0a4de0 tx=0x7fb044009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0400b5020 con 0x7fb04c09ba70 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 monclient: dump: 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.877 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb040004e50 con 0x7fb04c09ba70 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 msgr2=0x7fb04c09be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 0x7fb04c09be50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb040009000 tx=0x7fb034005840 comp rx=0 tx=0).stop 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 msgr2=0x7fb04c09be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.850+0000 7fb033fff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09ba70 0x7fb04c09be50 secure :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb040009000 tx=0x7fb034005840 comp rx=0 tx=0).stop 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb0521a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb0521a5640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 0x7fb04c1dbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb0521a5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb04c068240 con 0x7fb04c09c390 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 0x7fb04c1dbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb04b7fe640 10 monclient: get_auth_request con 0x7fb04c09c390 auth_method 0 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb04b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb04b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:00.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.854+0000 7fb04b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04b7fe640 10 monclient(hunting): handle_auth_done global_id 18970 payload 824 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04b7fe640 1 -- 172.21.15.43:0/2071135887 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb04c0693b0 con 0x7fb04c09c390 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04b7fe640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 0x7fb04c1dbf70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb040000fc0 tx=0x7fb0400a7000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 client.22427.objecter ms_handle_connect 0x7fb04c09c390 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 client.22427.objecter resend_mon_ops 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb0400b5070 con 0x7fb04c09c390 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 monclient: dump: 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:00.881 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:00.882 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fb0400070c0 con 0x7fb04c09c390 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 msgr2=0x7fb04c1dbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 0x7fb04c1dbf70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb040000fc0 tx=0x7fb0400a7000 comp rx=0 tx=0).stop 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 msgr2=0x7fb04c1dbf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb033fff640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb04c09c390 0x7fb04c1dbf70 secure :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fb040000fc0 tx=0x7fb0400a7000 comp rx=0 tx=0).stop 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:00.883 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 24, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 164316, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:00.884 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.885 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.886 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:00.887 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:00.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:00.895 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:00.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:00.957 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 10 librados: watch_flush enter 2024-04-19T00:47:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 10 librados: watch_flush exit 2024-04-19T00:47:00.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 20 client.22427.objecter shutdown clearing up homeless session... 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 10 client.22427.objecter successfully canceled tick 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb038038e10 msgr2=0x7fb03803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb038038e10 0x7fb03803b2d0 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0a4de0 tx=0x7fb044009040 comp rx=0 tx=0).stop 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 10 monclient: shutdown 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb04c138210 msgr2=0x7fb04c09fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 --2- 172.21.15.43:0/2071135887 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb04c138210 0x7fb04c09fb80 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb03c011630 tx=0x7fb03c011af0 comp rx=0 tx=0).stop 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04bfff640 1 -- 172.21.15.43:0/2071135887 reap_dead start 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb04bfff640 1 -- 172.21.15.43:0/2071135887 reap_dead start 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 shutdown_connections 2024-04-19T00:47:00.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 >> 172.21.15.43:0/2071135887 conn(0x7fb04c0ab2f0 msgr2=0x7fb04c139ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 shutdown_connections 2024-04-19T00:47:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 -- 172.21.15.43:0/2071135887 wait complete. 2024-04-19T00:47:00.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:00.858+0000 7fb0521a5640 1 librados: shutdown 2024-04-19T00:47:00.961 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:47:00.962 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-19T00:47:00.962 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['g'] 2024-04-19T00:47:00.962 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-19T00:47:00.962 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a store 2024-04-19T00:47:00.962 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a sync_force --yes-i-really-mean-it 2024-04-19T00:47:01.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient: get_monmap_and_config 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient: build_initial_monmap 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient: monmap: 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:01.096148+0000 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:01.096148+0000 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:01.096 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:01.096 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 Processor -- start 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 -- start start 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient: init 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient: _add_conns ranks=[8,2,7,3,10,0,9,4,1,6,5] 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 0x7f891c1370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): picked mon.noname-e con 0x7f891c0a0b10 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:01.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f891c0a0ef0 0x7f891c0a1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): picked mon.noname-b con 0x7f891c0a0ef0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f891c0a2f50 0x7f891c0a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): picked mon.noname-i con 0x7f891c0a2f50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f891c0a19c0 con 0x7f891c0a0ef0 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f891c09ed40 con 0x7f891c0a2f50 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f891c09f020 con 0x7f891c0a0b10 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f89227de640 10 monclient(hunting): authenticate will time out at 1091.223253s 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 0x7f891c1370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 0x7f891c1370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:41708/0 (socket says 172.21.15.43:41708) 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 -- 172.21.15.43:0/1422509679 learned_addr learned my addr 172.21.15.43:0/1422509679 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): get_auth_request con 0x7f891c0a0b10 auth_method 0 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f891c0a0ef0 0x7f891c0a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): get_auth_request con 0x7f891c0a0ef0 auth_method 0 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f8920d7b640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f891c0a2f50 0x7f891c0a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f8920d7b640 10 monclient(hunting): get_auth_request con 0x7f891c0a2f50 auth_method 0 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f8920d7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f8920d7b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f8920d7b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient(hunting): handle_auth_done global_id 22436 payload 274 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 -- 172.21.15.43:0/1422509679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f891c0a2f50 msgr2=0x7f891c0a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f891c0a2f50 0x7f891c0a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 -- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f891c0a0ef0 msgr2=0x7f891c0a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f891c0a0ef0 0x7f891c0a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 monclient: found mon.noname-e 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:01.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.090+0000 7f891bfff640 1 -- 172.21.15.43:0/1422509679 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c0767a0 con 0x7f891c0a0b10 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891bfff640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 0x7f891c1370a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f890c009e80 tx=0x7f890c01c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=2d268a1945cf67d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 1 -- 172.21.15.43:0/1422509679 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f890c0090d0 con 0x7f891c0a0b10 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: dump: 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.109 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: _finish_auth 0 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:01.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 5 monclient: authenticate success, global_id 22436 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 1 -- 172.21.15.43:0/1422509679 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890c0a6970 con 0x7f891c0a0b10 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 4 set_mon_vals no callback set 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 1 -- 172.21.15.43:0/1422509679 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f890c0af480 con 0x7f891c0a0b10 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f891affd640 10 monclient: dump: 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.111 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 10 monclient: shutdown 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:01.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 -- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 msgr2=0x7f891c1370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 0x7f891c1370a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f890c009e80 tx=0x7f890c01c910 comp rx=0 tx=0).stop 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 -- 172.21.15.43:0/1422509679 shutdown_connections 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f891c0a2f50 0x7f891c0a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f891c0a0ef0 0x7f891c0a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 --2- 172.21.15.43:0/1422509679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a0b10 0x7f891c1370a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 -- 172.21.15.43:0/1422509679 >> 172.21.15.43:0/1422509679 conn(0x7f891c0a4040 msgr2=0x7f891c0a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 -- 172.21.15.43:0/1422509679 shutdown_connections 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 -- 172.21.15.43:0/1422509679 wait complete. 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 10 monclient: build_initial_monmap 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 10 monclient: monmap: 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:01.113 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 librados: starting msgr at 2024-04-19T00:47:01.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 librados: starting objecter 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 Processor -- start 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 -- start start 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 librados: setting wanted keys 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 1 librados: calling monclient init 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.094+0000 7f89227de640 10 monclient: init 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient: _add_conns ranks=[6,1,8,5,2,7,3,0,4] 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 0x7f891c134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): picked mon.noname-d con 0x7f891c0a0b10 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f891c0a0ef0 0x7f891c09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): picked mon.noname-f con 0x7f891c0a0ef0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a2f50 0x7f891c09ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): picked mon.noname-e con 0x7f891c0a2f50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f891c140860 con 0x7f891c0a0ef0 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f891c1406e0 con 0x7f891c0a0b10 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f891c1409e0 con 0x7f891c0a2f50 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): authenticate will time out at 1091.228365s 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 0x7f891c134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 0x7f891c134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:41086/0 (socket says 172.21.15.43:41086) 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 -- 172.21.15.43:0/3801679723 learned_addr learned my addr 172.21.15.43:0/3801679723 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): get_auth_request con 0x7f891c0a0b10 auth_method 0 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a2f50 0x7f891c09ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): get_auth_request con 0x7f891c0a2f50 auth_method 0 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:01.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f891c0a0ef0 0x7f891c09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient(hunting): handle_auth_done global_id 22794 payload 824 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a2f50 msgr2=0x7f891c09ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f891c0a2f50 0x7f891c09ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f891c0a0ef0 msgr2=0x7f891c09b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f891c0a0ef0 0x7f891c09b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 monclient: found mon.noname-d 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c13c800 con 0x7f891c0a0b10 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient: get_auth_request con 0x7f891c0a0ef0 auth_method 0 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f891c0a0ef0 0x7f891c09b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891bfff640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 0x7f891c134130 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f890c01c740 tx=0x7f890c0196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:01.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 client.?.objecter ms_handle_connect 0x7f891c0a0b10 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f890c009270 con 0x7f891c0a0b10 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: dump: 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:01.119 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: _finish_auth 0 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890c0a6710 con 0x7f891c0a0b10 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 5 monclient: authenticate success, global_id 22794 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient: _renew_subs 2024-04-19T00:47:01.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891c13cae0 con 0x7f891c0a0b10 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f890c003e10 con 0x7f891c0a0b10 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891a7fc640 4 set_mon_vals no callback set 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 client.22794.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: dump: 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:01.121 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient: _renew_subs 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:47:01.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f891c13c520 con 0x7f891c0a0b10 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 librados: init done 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 0x7f891c09bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f891c074f70 con 0x7f891c13aed0 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 0x7f891c09bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient: get_auth_request con 0x7f891c13aed0 auth_method 0 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f890c0039a0 con 0x7f891c0a0b10 2024-04-19T00:47:01.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 10 monclient(hunting): handle_auth_done global_id 29718 payload 824 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f891c0693b0 con 0x7f891c13aed0 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f88fc038e40 0x7f88fc03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(106..106 src has 1..106) ==== 8307+0+0 (secure 0 0 0) 0x7f890c0b5070 con 0x7f891c0a0b10 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 client.22794.objecter ms_dispatch 0x7f891c000ba0 osd_map(106..106 src has 1..106) 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 3 client.22794.objecter handle_osd_map got epochs [106,106] > 0 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 3 client.22794.objecter handle_osd_map decoding full epoch 106 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f891b7fe640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 0x7f891c09bfc0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f891c09cbd0 tx=0x7f8914010d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 20 client.22794.objecter dump_active .. 0 homeless 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 client.22794.objecter ms_handle_connect 0x7f891c13aed0 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 client.22794.objecter resend_mon_ops 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f891400f040 con 0x7f891c13aed0 2024-04-19T00:47:01.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f88fc038e40 0x7f88fc03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: dump: 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:01.125 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 10 monclient: get_auth_request con 0x7f88fc038e40 auth_method 0 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f891401c400 con 0x7f891c13aed0 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 msgr2=0x7f891c09bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 0x7f891c09bfc0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f891c09cbd0 tx=0x7f8914010d60 comp rx=0 tx=0).stop 2024-04-19T00:47:01.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 msgr2=0x7f891c09bfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c13aed0 0x7f891c09bfc0 secure :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f891c09cbd0 tx=0x7f8914010d60 comp rx=0 tx=0).stop 2024-04-19T00:47:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8920d7b640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f88fc038e40 0x7f88fc03b300 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7f891c13b420 tx=0x7f8904008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:01.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.098+0000 7f8918ff9640 10 client.22794.objecter ms_handle_connect 0x7f88fc038e40 2024-04-19T00:47:01.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f89227de640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f89227de640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 0x7f891c137fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f89227de640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f891c06d140 con 0x7f891c137c00 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 0x7f891c137fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient: get_auth_request con 0x7f891c137c00 auth_method 0 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 10 monclient(hunting): handle_auth_done global_id 29727 payload 824 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 1 -- 172.21.15.43:0/3801679723 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f891c13cf00 con 0x7f891c137c00 2024-04-19T00:47:01.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f891b7fe640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 0x7f891c137fe0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f8914011010 tx=0x7f89140046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 client.22794.objecter ms_handle_connect 0x7f891c137c00 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 client.22794.objecter resend_mon_ops 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f891401cda0 con 0x7f891c137c00 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 monclient: dump: 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:01.130 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f8914005bb0 con 0x7f891c137c00 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:01.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 msgr2=0x7f891c137fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 0x7f891c137fe0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f8914011010 tx=0x7f89140046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 msgr2=0x7f891c137fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f8918ff9640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f891c137c00 0x7f891c137fe0 secure :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f8914011010 tx=0x7f89140046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:47:01.132 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.106+0000 7f89227de640 10 librados: watch_flush enter 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 10 librados: watch_flush exit 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 20 client.22794.objecter shutdown clearing up homeless session... 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 10 client.22794.objecter successfully canceled tick 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f88fc038e40 msgr2=0x7f88fc03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f88fc038e40 0x7f88fc03b300 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7f891c13b420 tx=0x7f8904008040 comp rx=0 tx=0).stop 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 10 monclient: shutdown 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 msgr2=0x7f891c134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 0x7f891c134130 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f890c01c740 tx=0x7f890c0196b0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f8920d7b640 1 -- 172.21.15.43:0/3801679723 reap_dead start 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 shutdown_connections 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 --2- 172.21.15.43:0/3801679723 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f891c0a0b10 0x7f891c134130 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 >> 172.21.15.43:0/3801679723 conn(0x7f891c0a4040 msgr2=0x7f891c134780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 shutdown_connections 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 -- 172.21.15.43:0/3801679723 wait complete. 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:01.110+0000 7f89227de640 1 librados: shutdown 2024-04-19T00:47:01.135 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T00:47:01.135 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T00:47:01.135 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:47:01.148 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T00:47:01.148 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-04-19T00:47:01.149 INFO:tasks.ceph.mon.g:Sent signal 19 2024-04-19T00:47:01.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:47:16.150 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-04-19T00:47:16.151 INFO:tasks.ceph.mon.g:Sent signal 18 2024-04-19T00:47:16.151 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-04-19T00:47:16.151 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:47:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient: get_monmap_and_config 2024-04-19T00:47:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient: build_initial_monmap 2024-04-19T00:47:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient: monmap: 2024-04-19T00:47:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:16.286 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:16.287995+0000 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:16.287995+0000 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:16.287 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:16.288 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:16.288 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:16.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 1 Processor -- start 2024-04-19T00:47:16.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 1 -- start start 2024-04-19T00:47:16.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient: init 2024-04-19T00:47:16.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:16.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient: _add_conns ranks=[1,9,8,7,3,5,6,2,4,0,10] 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.282+0000 7f2786429640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f27800a4b10 addr v1:172.21.15.161:6789/0 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 0x7f27800a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 10 monclient(hunting): picked mon.noname-a con 0x7f27800a4ef0 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 10 monclient(hunting): picked mon.noname-e con 0x7f278013e170 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27801398f0 con 0x7f27800a4b10 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f278006e750 con 0x7f27800a4ef0 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f278006f250 con 0x7f278013e170 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f2786429640 10 monclient(hunting): authenticate will time out at 1106.416128s 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27777fe640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 msgr2=0x7f27800a5370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:16.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27777fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 0x7f27800a5370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:60934/0 (socket says 172.21.15.43:60934) 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 learned_addr learned my addr 172.21.15.43:0/2148742397 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f278013e170 auth_method 0 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:16.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.286+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:16.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 msgr2=0x7f27800a5370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:16.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 0x7f27800a5370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f278013e170 auth_method 0 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:16.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.490+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:47:16.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 msgr2=0x7f27800a5370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:16.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 0x7f27800a5370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:47:16.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f278013e170 auth_method 0 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:16.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:16.894+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:47:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 msgr2=0x7f27800a5370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 0x7f27800a5370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:47:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f278013e170 auth_method 0 2024-04-19T00:47:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:17.698+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): tick 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 msgr2=0x7f2780140560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f278013e170 0x7f2780140560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:19.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 msgr2=0x7f27800a5370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.43:3300/0 conn(0x7f27800a4ef0 0x7f27800a5370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 >> v1:172.21.15.161:6789/0 conn(0x7f27800a4b10 legacy=0x7f27801391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient: _add_conns ranks=[7,3,5,6,0,10,4,2] 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): picked mon.noname-i con 0x7f2760003820 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): picked mon.noname-g con 0x7f27600062f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): picked mon.noname-h con 0x7f2760008e70 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f278006e750 con 0x7f27600062f0 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:19.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f278006f250 con 0x7f2760008e70 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f276000baa0 con 0x7f2760003820 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277e7fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f27600062f0 auth_method 0 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760008e70 auth_method 0 2024-04-19T00:47:19.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760003820 auth_method 0 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:19.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:19.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:47:19.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.286+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760003820 auth_method 0 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760008e70 auth_method 0 2024-04-19T00:47:19.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f27600062f0 auth_method 0 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:47:19.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:47:19.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:47:19.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.490+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760008e70 auth_method 0 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760003820 auth_method 0 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f27600062f0 auth_method 0 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:19.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:47:19.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:47:19.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:19.894+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760008e70 auth_method 0 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:20.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f27600062f0 auth_method 0 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760003820 auth_method 0 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:47:20.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:47:20.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:20.698+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:47:21.581 INFO:tasks.workunit.client.0.smithi161.stdout: pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-19T00:47:21.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2057 ms) 2024-04-19T00:47:21.581 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9059 ms total) 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4073 ms) 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3011 ms) 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-19T00:47:21.582 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4013 ms) 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11097 ms total) 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (115193 ms total) 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-19T00:47:21.583 INFO:tasks.workunit.client.0.smithi161.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-19T00:47:22.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): tick 2024-04-19T00:47:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:47:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:47:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:47:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 msgr2=0x7f2760005c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2760003820 0x7f2760005c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 msgr2=0x7f276000b2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760008e70 0x7f276000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 msgr2=0x7f2760008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f27600062f0 0x7f2760008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 reap_dead start 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 1 -- 172.21.15.43:0/2148742397 reap_dead start 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient: _add_conns ranks=[4,10,6,0,2] 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760006770 0x7f27600092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f2760006770 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.161:3300/0 conn(0x7f2760009810 0x7f2760005c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): picked mon.noname-f con 0x7f2760009810 addr v2:172.21.15.161:3300/0 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f276000a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): picked mon.noname-d con 0x7f2760009bf0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f278006e750 con 0x7f2760009810 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f278006f250 con 0x7f2760006770 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 1 -- 172.21.15.43:0/2148742397 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f276000baa0 con 0x7f2760009bf0 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277e7fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760006770 0x7f27600092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760006770 auth_method 0 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f276000a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.161:3300/0 conn(0x7f2760009810 0x7f2760005c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760009bf0 auth_method 0 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f2760009810 auth_method 0 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27849c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient(hunting): handle_auth_done global_id 22630 payload 274 2024-04-19T00:47:22.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 msgr2=0x7f276000a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f276000a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 >> v2:172.21.15.161:3300/0 conn(0x7f2760009810 msgr2=0x7f2760005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.161:3300/0 conn(0x7f2760009810 0x7f2760005c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 monclient: found mon.noname-c 2024-04-19T00:47:22.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.286+0000 7f277ffff640 1 -- 172.21.15.43:0/2148742397 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278006f530 con 0x7f2760006770 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.290+0000 7f277ffff640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760006770 0x7f27600092d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f2770007c80 tx=0x7f277001ccb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=233b73449b2c06bd server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 1 -- 172.21.15.43:0/2148742397 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2770006d10 con 0x7f2760006770 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: dump: 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.319 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 1 -- 172.21.15.43:0/2148742397 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27700a9070 con 0x7f2760006770 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 5 monclient: authenticate success, global_id 22630 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 1 -- 172.21.15.43:0/2148742397 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f27700074a0 con 0x7f2760006770 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f277f7fe640 10 monclient: dump: 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.321 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 4 set_mon_vals no callback set 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 10 monclient: shutdown 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:22.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 -- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760006770 msgr2=0x7f27600092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760006770 0x7f27600092d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f2770007c80 tx=0x7f277001ccb0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 -- 172.21.15.43:0/2148742397 shutdown_connections 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f276000a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 --2- 172.21.15.43:0/2148742397 >> v2:172.21.15.161:3300/0 conn(0x7f2760009810 0x7f2760005c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 --2- 172.21.15.43:0/2148742397 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760006770 0x7f27600092d0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 -- 172.21.15.43:0/2148742397 >> 172.21.15.43:0/2148742397 conn(0x7f27800a98c0 msgr2=0x7f27800a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 -- 172.21.15.43:0/2148742397 shutdown_connections 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 -- 172.21.15.43:0/2148742397 wait complete. 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 10 monclient: build_initial_monmap 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 10 monclient: monmap: 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:22.323 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 librados: starting msgr at 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 librados: starting objecter 2024-04-19T00:47:22.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 Processor -- start 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 -- start start 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 librados: setting wanted keys 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 1 librados: calling monclient init 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.294+0000 7f2786429640 10 monclient: init 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _add_conns ranks=[8,4,6,5,0,7,1,3,2] 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2760006770 0x7f27800a21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): picked mon.noname-e con 0x7f2760006770 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f2780134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): picked mon.noname-c con 0x7f2760009810 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f2780135150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): picked mon.noname-d con 0x7f2760009bf0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f2780135c40 con 0x7f2760009810 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2780135ac0 con 0x7f2760009bf0 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f2780135dc0 con 0x7f2760006770 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient(hunting): authenticate will time out at 1112.428673s 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f2780134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2760006770 0x7f27800a21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f2780134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:32936/0 (socket says 172.21.15.43:32936) 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 1 -- 172.21.15.43:0/2819555021 learned_addr learned my addr 172.21.15.43:0/2819555021 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f2760009810 auth_method 0 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2760006770 0x7f27800a21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53118/0 (socket says 172.21.15.43:53118) 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f2780135150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760006770 auth_method 0 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760009bf0 auth_method 0 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient(hunting): handle_auth_done global_id 22839 payload 824 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2760006770 msgr2=0x7f27800a21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2760006770 0x7f27800a21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 msgr2=0x7f2780134c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f2780134c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 monclient: found mon.noname-d 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278011fe90 con 0x7f2760009bf0 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27777fe640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f2780134c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f2780135150 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f27800739b0 tx=0x7f2778018790 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 client.?.objecter ms_handle_connect 0x7f2760009bf0 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2778023870 con 0x7f2760009bf0 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: dump: 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.329 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2778023a50 con 0x7f2760009bf0 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 5 monclient: authenticate success, global_id 22839 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _renew_subs 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2780121060 con 0x7f2760009bf0 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2778023c00 con 0x7f2760009bf0 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277e7fc640 4 set_mon_vals no callback set 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 client.22839.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: dump: 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.331 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _renew_subs 2024-04-19T00:47:22.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f27801220c0 con 0x7f2760009bf0 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 librados: init done 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 msgr2=0x7f2780135150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f277802a020 con 0x7f2760009bf0 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 msgr2=0x7f2780135150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f2780135150 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f27800739b0 tx=0x7f2778018790 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f2780135150 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f27800739b0 tx=0x7f2778018790 comp rx=0 tx=0).stop 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2780069630 con 0x7f2760009bf0 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f276003b080 0x7f276003d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:47:22.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 msgr2=0x7f2780135150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2760009bf0 0x7f2780135150 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient: _add_conns ranks=[4,5,2,7,0,8,6,1,3] 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f276003fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): picked mon.c con 0x7f2760009810 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760040440 0x7f2760042870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): picked mon.h con 0x7f2760040440 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2760043290 0x7f2760045720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): picked mon.b con 0x7f2760043290 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f27600462d0 con 0x7f2760043290 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f2760046480 con 0x7f2760009810 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f2760046630 con 0x7f2760040440 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.298+0000 7f277d7fa640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f276003fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): get_auth_request con 0x7f2760009810 auth_method 0 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2760043290 0x7f2760045720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): get_auth_request con 0x7f2760043290 auth_method 0 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f276003b080 0x7f276003d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27849c6640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760040440 0x7f2760042870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27849c6640 10 monclient(hunting): get_auth_request con 0x7f2760040440 auth_method 0 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27849c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27849c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27849c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient(hunting): handle_auth_done global_id 22839 payload 293 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760040440 msgr2=0x7f2760042870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2760040440 0x7f2760042870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2760043290 msgr2=0x7f2760045720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2760043290 0x7f2760045720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 monclient: found mon.c 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f2760046830 con 0x7f2760009810 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f276800b580 con 0x7f2760009810 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 10 monclient: get_auth_request con 0x7f276003b080 auth_method 0 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f27777fe640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f276003fcf0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f276800bbc0 tx=0x7f2778009040 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 client.22839.objecter ms_handle_connect 0x7f2760009810 2024-04-19T00:47:22.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 client.22839.objecter resend_mon_ops 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2768009d20 con 0x7f2760009810 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 monclient: dump: 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.338 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f276800bde0 con 0x7f2760009810 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277effd640 4 set_mon_vals no callback set 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 client.22839.objecter ms_handle_connect 0x7f276003b080 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277ffff640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f276003b080 0x7f276003d540 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f27700ad000 tx=0x7f27700a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f276800ac40 con 0x7f2760009810 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.4 v2:172.21.15.43:3302/0 4 ==== osd_map(109..109 src has 1..109) ==== 7695+0+0 (secure 0 0 0) 0x7f2768014040 con 0x7f2760009810 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 10 client.22839.objecter ms_dispatch 0x7f2780000ba0 osd_map(109..109 src has 1..109) 2024-04-19T00:47:22.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 3 client.22839.objecter handle_osd_map got epochs [109,109] > 0 2024-04-19T00:47:22.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 3 client.22839.objecter handle_osd_map decoding full epoch 109 2024-04-19T00:47:22.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.302+0000 7f277d7fa640 20 client.22839.objecter dump_active .. 0 homeless 2024-04-19T00:47:22.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.310+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.4 v2:172.21.15.43:3302/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f276801d030 con 0x7f2760009810 2024-04-19T00:47:22.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.310+0000 7f277d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.310+0000 7f277d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f2786429640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:47:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f2786429640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:47:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f2786429640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2780069630 con 0x7f2760009810 2024-04-19T00:47:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f277d7fa640 1 -- 172.21.15.43:0/2819555021 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f2768047090 con 0x7f2760009810 2024-04-19T00:47:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f277d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:47:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.562+0000 7f277d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:22.566 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":30,"quorum":[1,2,3,4,5,6,7,8],"quorum_names":["f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:47:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 10 librados: watch_flush enter 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 10 librados: watch_flush exit 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 20 client.22839.objecter shutdown clearing up homeless session... 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 10 client.22839.objecter successfully canceled tick 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f276003b080 msgr2=0x7f276003d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f276003b080 0x7f276003d540 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f27700ad000 tx=0x7f27700a7070 comp rx=0 tx=0).stop 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 10 monclient: shutdown 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 msgr2=0x7f276003fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f2786429640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f276003fcf0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f276800bbc0 tx=0x7f2778009040 comp rx=0 tx=0).stop 2024-04-19T00:47:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.566+0000 7f27849c6640 1 -- 172.21.15.43:0/2819555021 reap_dead start 2024-04-19T00:47:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.570+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 shutdown_connections 2024-04-19T00:47:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.570+0000 7f2786429640 1 --2- 172.21.15.43:0/2819555021 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2760009810 0x7f276003fcf0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.570+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 >> 172.21.15.43:0/2819555021 conn(0x7f27800a98c0 msgr2=0x7f27801c3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:22.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.574+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 shutdown_connections 2024-04-19T00:47:22.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.574+0000 7f2786429640 1 -- 172.21.15.43:0/2819555021 wait complete. 2024-04-19T00:47:22.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.574+0000 7f2786429640 1 librados: shutdown 2024-04-19T00:47:22.605 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-04-19T00:47:22.605 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-04-19T00:47:22.606 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:47:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.718+0000 7f8baa456640 10 monclient: get_monmap_and_config 2024-04-19T00:47:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.718+0000 7f8baa456640 10 monclient: build_initial_monmap 2024-04-19T00:47:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient: monmap: 2024-04-19T00:47:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.728 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:22.726506+0000 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:22.726506+0000 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:22.729 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 Processor -- start 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 -- start start 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient: init 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient: _add_conns ranks=[9,5,7,8,3,0,2,1,10,4,6] 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8ba4138cd0 0x7f8ba41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba4138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ba40a5b80 0x7f8ba41395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): picked mon.noname-h con 0x7f8ba40a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba41358b0 0x7f8ba4135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): picked mon.noname-i con 0x7f8ba41358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba40a4410 con 0x7f8ba4138cd0 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8ba4139b30 con 0x7f8ba40a5b80 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8ba40a3dc0 con 0x7f8ba41358b0 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8baa456640 10 monclient(hunting): authenticate will time out at 1112.853637s 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba3fff640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7f8ba4138cd0 msgr2=0x7f8ba41390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba3fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8ba4138cd0 0x7f8ba41390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ba40a5b80 0x7f8ba41395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba41358b0 0x7f8ba4135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ba40a5b80 0x7f8ba41395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:53426/0 (socket says 172.21.15.43:53426) 2024-04-19T00:47:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 1 -- 172.21.15.43:0/1165090793 learned_addr learned my addr 172.21.15.43:0/1165090793 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request con 0x7f8ba40a5b80 auth_method 0 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): get_auth_request con 0x7f8ba41358b0 auth_method 0 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_done global_id 18997 payload 274 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ba40a5b80 msgr2=0x7f8ba41395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 --2- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ba40a5b80 0x7f8ba41395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1165090793 >> v2:172.21.15.43:3300/0 conn(0x7f8ba4138cd0 msgr2=0x7f8ba41390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 --2- 172.21.15.43:0/1165090793 >> v2:172.21.15.43:3300/0 conn(0x7f8ba4138cd0 0x7f8ba41390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 monclient: found mon.noname-i 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:47:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.722+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1165090793 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba40a40a0 con 0x7f8ba41358b0 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba89f3640 1 --2- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba41358b0 0x7f8ba4135d40 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009d90 tx=0x7f8b9401c7f0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=c2fb2325df97b1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 1 -- 172.21.15.43:0/1165090793 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b940b2070 con 0x7f8ba41358b0 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: dump: 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 1 -- 172.21.15.43:0/1165090793 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b940a59b0 con 0x7f8ba41358b0 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 1 -- 172.21.15.43:0/1165090793 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b940a5be0 con 0x7f8ba41358b0 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8ba2ffd640 10 monclient: dump: 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 5 monclient: authenticate success, global_id 18997 2024-04-19T00:47:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 4 set_mon_vals no callback set 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: shutdown 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba41358b0 msgr2=0x7f8ba4135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba41358b0 0x7f8ba4135d40 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009d90 tx=0x7f8b9401c7f0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- 172.21.15.43:0/1165090793 shutdown_connections 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba41358b0 0x7f8ba4135d40 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- 172.21.15.43:0/1165090793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ba40a5b80 0x7f8ba41395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- 172.21.15.43:0/1165090793 >> v2:172.21.15.43:3300/0 conn(0x7f8ba4138cd0 0x7f8ba41390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- 172.21.15.43:0/1165090793 >> 172.21.15.43:0/1165090793 conn(0x7f8ba40ab2f0 msgr2=0x7f8ba40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- 172.21.15.43:0/1165090793 shutdown_connections 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- 172.21.15.43:0/1165090793 wait complete. 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: build_initial_monmap 2024-04-19T00:47:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: monmap: 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 librados: starting msgr at 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 librados: starting objecter 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 Processor -- start 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- start start 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 librados: setting wanted keys 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 librados: calling monclient init 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: init 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient: _add_conns ranks=[7,4,2,8,5,0,1,6,3] 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba40a5b80 0x7f8ba4134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): picked mon.noname-i con 0x7f8ba40a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba41358b0 0x7f8ba41351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba41358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 0x7f8ba41d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba4138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba40a2010 con 0x7f8ba4138cd0 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8ba40a1e90 con 0x7f8ba41358b0 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8ba4142aa0 con 0x7f8ba40a5b80 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.726+0000 7f8baa456640 10 monclient(hunting): authenticate will time out at 1112.859972s 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 0x7f8ba41d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba41358b0 0x7f8ba41351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 0x7f8ba41d0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35782/0 (socket says 172.21.15.43:35782) 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1422111315 learned_addr learned my addr 172.21.15.43:0/1422111315 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): get_auth_request con 0x7f8ba4138cd0 auth_method 0 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request con 0x7f8ba41358b0 auth_method 0 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba40a5b80 0x7f8ba4134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): get_auth_request con 0x7f8ba40a5b80 auth_method 0 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient(hunting): handle_auth_done global_id 22790 payload 824 2024-04-19T00:47:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba40a5b80 msgr2=0x7f8ba4134c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ba40a5b80 0x7f8ba4134c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba41358b0 msgr2=0x7f8ba41351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba41358b0 0x7f8ba41351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 monclient: found mon.noname-b 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba41356e0 con 0x7f8ba4138cd0 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba41358b0 0x7f8ba41351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba89f3640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 0x7f8ba41d0b30 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f8b9401c620 tx=0x7f8b940195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 client.?.objecter ms_handle_connect 0x7f8ba4138cd0 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b940b2050 con 0x7f8ba4138cd0 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: dump: 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b940be070 con 0x7f8ba4138cd0 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 5 monclient: authenticate success, global_id 22790 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 monclient: _renew_subs 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba40a3590 con 0x7f8ba4138cd0 2024-04-19T00:47:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b94003e10 con 0x7f8ba4138cd0 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba27fc640 4 set_mon_vals no callback set 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: dump: 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.754 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 client.22790.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 monclient: _renew_subs 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba40a3040 con 0x7f8ba4138cd0 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 1 librados: init done 2024-04-19T00:47:22.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 0x7f8ba41d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba409eeb0 con 0x7f8ba40a0290 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 0x7f8ba41d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient: get_auth_request con 0x7f8ba40a0290 auth_method 0 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8b940039a0 con 0x7f8ba4138cd0 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b84038e10 0x7f8b8403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_done global_id 22799 payload 824 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(110..110 src has 1..110) ==== 8505+0+0 (secure 0 0 0) 0x7f8b940e4d40 con 0x7f8ba4138cd0 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 client.22790.objecter ms_dispatch 0x7f8ba4000ba0 osd_map(110..110 src has 1..110) 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 3 client.22790.objecter handle_osd_map got epochs [110,110] > 0 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 3 client.22790.objecter handle_osd_map decoding full epoch 110 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ba4139800 con 0x7f8ba40a0290 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 20 client.22790.objecter dump_active .. 0 homeless 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 client.22790.objecter ms_handle_connect 0x7f8ba40a0290 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 client.22790.objecter resend_mon_ops 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba3fff640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 0x7f8ba41d1070 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c003990 tx=0x7f8b9c00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b84038e10 0x7f8b8403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b9c018070 con 0x7f8ba40a0290 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: dump: 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.758 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 10 monclient: get_auth_request con 0x7f8b84038e10 auth_method 0 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8b9c005a50 con 0x7f8ba40a0290 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 msgr2=0x7f8ba41d1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 0x7f8ba41d1070 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c003990 tx=0x7f8b9c00db90 comp rx=0 tx=0).stop 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 msgr2=0x7f8ba41d1070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba40a0290 0x7f8ba41d1070 secure :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c003990 tx=0x7f8b9c00db90 comp rx=0 tx=0).stop 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba0ff9640 10 client.22790.objecter ms_handle_connect 0x7f8b84038e10 2024-04-19T00:47:22.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.730+0000 7f8ba37fe640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b84038e10 0x7f8b8403b2d0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409dfb0 tx=0x7f8b8c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 0x7f8ba413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba41d19c0 con 0x7f8ba41d15b0 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 0x7f8ba413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient: get_auth_request con 0x7f8ba41d15b0 auth_method 0 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_done global_id 22808 payload 824 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 1 -- 172.21.15.43:0/1422111315 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ba40693b0 con 0x7f8ba41d15b0 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 client.22790.objecter ms_handle_connect 0x7f8ba41d15b0 2024-04-19T00:47:22.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 client.22790.objecter resend_mon_ops 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba3fff640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 0x7f8ba413bcf0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b130 tx=0x7f8b9c0059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b9c018040 con 0x7f8ba41d15b0 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 monclient: dump: 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.763 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9434 (secure 0 0 0) 0x7f8b9c003d80 con 0x7f8ba41d15b0 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 msgr2=0x7f8ba413bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 0x7f8ba413bcf0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b130 tx=0x7f8b9c0059e0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 msgr2=0x7f8ba413bcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba0ff9640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba41d15b0 0x7f8ba413bcf0 secure :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b130 tx=0x7f8b9c0059e0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 186237, 2024-04-19T00:47:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.776 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.777 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:22.783 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.785 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.787 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.788 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:22.789 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 10 librados: watch_flush enter 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 10 librados: watch_flush exit 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 20 client.22790.objecter shutdown clearing up homeless session... 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 10 client.22790.objecter successfully canceled tick 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b84038e10 msgr2=0x7f8b8403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b84038e10 0x7f8b8403b2d0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409dfb0 tx=0x7f8b8c009240 comp rx=0 tx=0).stop 2024-04-19T00:47:22.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 10 monclient: shutdown 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 msgr2=0x7f8ba41d0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8ba89f3640 1 -- 172.21.15.43:0/1422111315 reap_dead start 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 0x7f8ba41d0b30 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f8b9401c620 tx=0x7f8b940195f0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 shutdown_connections 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 --2- 172.21.15.43:0/1422111315 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba4138cd0 0x7f8ba41d0b30 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 >> 172.21.15.43:0/1422111315 conn(0x7f8ba40ab2f0 msgr2=0x7f8ba41368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 shutdown_connections 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 -- 172.21.15.43:0/1422111315 wait complete. 2024-04-19T00:47:22.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.738+0000 7f8baa456640 1 librados: shutdown 2024-04-19T00:47:22.793 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:22.793 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon c 2024-04-19T00:47:22.793 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:47:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient: get_monmap_and_config 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient: build_initial_monmap 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient: monmap: 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:22.915364+0000 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:22.915364+0000 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:22.915 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 Processor -- start 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 -- start start 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient: init 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient: _add_conns ranks=[10,8,9,1,6,3,0,5,2,4,7] 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa480138cd0 0x7fa4801390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): picked mon.noname-f con 0x7fa480138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 0x7fa4801395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): picked mon.noname-e con 0x7fa4800a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa4801358b0 0x7fa480135d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): picked mon.noname-a con 0x7fa4801358b0 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fa4800a4410 con 0x7fa4801358b0 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa480139b30 con 0x7fa480138cd0 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa4800a3dc0 con 0x7fa4800a5b80 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa4852e4640 10 monclient(hunting): authenticate will time out at 1113.042591s 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47f7fe640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7fa4801358b0 msgr2=0x7fa480135d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47f7fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa4801358b0 0x7fa480135d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 0x7fa4801395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 0x7fa4801395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53130/0 (socket says 172.21.15.43:53130) 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 -- 172.21.15.43:0/2724620533 learned_addr learned my addr 172.21.15.43:0/2724620533 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): get_auth_request con 0x7fa4800a5b80 auth_method 0 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47effd640 1 --2- 172.21.15.43:0/2724620533 >> v2:172.21.15.161:3300/0 conn(0x7fa480138cd0 0x7fa4801390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47effd640 10 monclient(hunting): get_auth_request con 0x7fa480138cd0 auth_method 0 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_done global_id 22472 payload 274 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 -- 172.21.15.43:0/2724620533 >> v2:172.21.15.161:3300/0 conn(0x7fa480138cd0 msgr2=0x7fa4801390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/2724620533 >> v2:172.21.15.161:3300/0 conn(0x7fa480138cd0 0x7fa4801390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 -- 172.21.15.43:0/2724620533 >> v2:172.21.15.43:3300/0 conn(0x7fa4801358b0 msgr2=0x7fa480135d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/2724620533 >> v2:172.21.15.43:3300/0 conn(0x7fa4801358b0 0x7fa480135d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 monclient: found mon.noname-e 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.910+0000 7fa47e7fc640 1 -- 172.21.15.43:0/2724620533 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4800a40a0 con 0x7fa4800a5b80 2024-04-19T00:47:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/2724620533 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 0x7fa4801395f0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa4800a7450 tx=0x7fa47401c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=2ff5e368b072ea33 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 1 -- 172.21.15.43:0/2724620533 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4740b0070 con 0x7fa4800a5b80 2024-04-19T00:47:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: dump: 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 1 -- 172.21.15.43:0/2724620533 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4740aa810 con 0x7fa4800a5b80 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 1 -- 172.21.15.43:0/2724620533 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4740a9300 con 0x7fa4800a5b80 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 5 monclient: authenticate success, global_id 22472 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 4 set_mon_vals no callback set 2024-04-19T00:47:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa47dffb640 10 monclient: dump: 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 10 monclient: shutdown 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 -- 172.21.15.43:0/2724620533 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 msgr2=0x7fa4801395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 --2- 172.21.15.43:0/2724620533 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 0x7fa4801395f0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa4800a7450 tx=0x7fa47401c3a0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 -- 172.21.15.43:0/2724620533 shutdown_connections 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 --2- 172.21.15.43:0/2724620533 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa4800a5b80 0x7fa4801395f0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 --2- 172.21.15.43:0/2724620533 >> v2:172.21.15.43:3300/0 conn(0x7fa4801358b0 0x7fa480135d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 --2- 172.21.15.43:0/2724620533 >> v2:172.21.15.161:3300/0 conn(0x7fa480138cd0 0x7fa4801390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 -- 172.21.15.43:0/2724620533 >> 172.21.15.43:0/2724620533 conn(0x7fa4800ab2f0 msgr2=0x7fa4800ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 -- 172.21.15.43:0/2724620533 shutdown_connections 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 -- 172.21.15.43:0/2724620533 wait complete. 2024-04-19T00:47:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 10 monclient: build_initial_monmap 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 10 monclient: monmap: 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:22.932 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 librados: starting msgr at 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 librados: starting objecter 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 Processor -- start 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 -- start start 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 librados: setting wanted keys 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 1 librados: calling monclient init 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.914+0000 7fa4852e4640 10 monclient: init 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient: _add_conns ranks=[4,3,8,1,7,2,5,6,0] 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4800a5b80 0x7fa4801d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): picked mon.noname-c con 0x7fa4800a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4801358b0 0x7fa4801d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): picked mon.noname-g con 0x7fa4801358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa480138cd0 0x7fa4801d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): picked mon.noname-e con 0x7fa480138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa4800a2010 con 0x7fa4801358b0 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa4800a1e90 con 0x7fa4800a5b80 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa480142aa0 con 0x7fa480138cd0 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): authenticate will time out at 1113.048285s 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4800a5b80 0x7fa4801d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4800a5b80 0x7fa4801d0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:32964/0 (socket says 172.21.15.43:32964) 2024-04-19T00:47:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 -- 172.21.15.43:0/874951218 learned_addr learned my addr 172.21.15.43:0/874951218 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): get_auth_request con 0x7fa4800a5b80 auth_method 0 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa480138cd0 0x7fa4801d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): get_auth_request con 0x7fa480138cd0 auth_method 0 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4801358b0 0x7fa4801d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): get_auth_request con 0x7fa4801358b0 auth_method 0 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient(hunting): handle_auth_done global_id 22657 payload 824 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa480138cd0 msgr2=0x7fa4801d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa480138cd0 0x7fa4801d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4801358b0 msgr2=0x7fa4801d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa4801358b0 0x7fa4801d1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient: _finish_hunting 0 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 monclient: found mon.noname-c 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48013e930 con 0x7fa4800a5b80 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47effd640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4800a5b80 0x7fa4801d0b30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa47000d620 tx=0x7fa47000dae0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 client.?.objecter ms_handle_connect 0x7fa4800a5b80 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa470007380 con 0x7fa4800a5b80 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: dump: 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.937 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: _finish_auth 0 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: _check_auth_tickets 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa470019070 con 0x7fa4800a5b80 2024-04-19T00:47:22.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa470014430 con 0x7fa4800a5b80 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 5 monclient: authenticate success, global_id 22657 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient: _renew_subs 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4800a3590 con 0x7fa4800a5b80 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47d7fa640 4 set_mon_vals no callback set 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: dump: 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.939 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 client.22657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient: _renew_subs 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:22.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48013ced0 con 0x7fa4800a5b80 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 librados: init done 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 0x7fa480134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa48013e2d0 con 0x7fa4801348a0 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 0x7fa480134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient: get_auth_request con 0x7fa4801348a0 auth_method 0 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa47000b620 con 0x7fa4800a5b80 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_done global_id 22666 payload 824 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4800a3040 con 0x7fa4801348a0 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa46c038e40 0x7fa46c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(110..110 src has 1..110) ==== 8505+0+0 (secure 0 0 0) 0x7fa47000f040 con 0x7fa4800a5b80 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 client.22657.objecter ms_dispatch 0x7fa480000ba0 osd_map(110..110 src has 1..110) 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 3 client.22657.objecter handle_osd_map got epochs [110,110] > 0 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 3 client.22657.objecter handle_osd_map decoding full epoch 110 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 0x7fa480134c80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa4800a3520 tx=0x7fa4740191a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 20 client.22657.objecter dump_active .. 0 homeless 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 client.22657.objecter ms_handle_connect 0x7fa4801348a0 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 client.22657.objecter resend_mon_ops 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4740b9030 con 0x7fa4801348a0 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: dump: 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.943 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa46c038e40 0x7fa46c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa47f7fe640 10 monclient: get_auth_request con 0x7fa46c038e40 auth_method 0 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa4740b1040 con 0x7fa4801348a0 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 msgr2=0x7fa480134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 0x7fa480134c80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa4800a3520 tx=0x7fa4740191a0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 msgr2=0x7fa480134c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.918+0000 7fa4677fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801348a0 0x7fa480134c80 secure :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa4800a3520 tx=0x7fa4740191a0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.922+0000 7fa4677fe640 10 client.22657.objecter ms_handle_connect 0x7fa46c038e40 2024-04-19T00:47:22.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.922+0000 7fa47f7fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa46c038e40 0x7fa46c03b300 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7fa478003cc0 tx=0x7fa47800d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:22.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 0x7fa48009da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:22.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:22.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa48006d140 con 0x7fa4801351c0 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 0x7fa48009da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient: get_auth_request con 0x7fa4801351c0 auth_method 0 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 10 monclient(hunting): handle_auth_done global_id 22675 payload 824 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 1 -- 172.21.15.43:0/874951218 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4800693b0 con 0x7fa4801351c0 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47e7fc640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 0x7fa48009da80 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa4740097c0 tx=0x7fa4740a8000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 client.22657.objecter ms_handle_connect 0x7fa4801351c0 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 client.22657.objecter resend_mon_ops 2024-04-19T00:47:22.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa4740b9040 con 0x7fa4801351c0 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 monclient: dump: 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:22.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7fa4740be460 con 0x7fa4801351c0 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 msgr2=0x7fa48009da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 0x7fa48009da80 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa4740097c0 tx=0x7fa4740a8000 comp rx=0 tx=0).stop 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 msgr2=0x7fa48009da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4677fe640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4801351c0 0x7fa48009da80 secure :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa4740097c0 tx=0x7fa4740a8000 comp rx=0 tx=0).stop 2024-04-19T00:47:22.949 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 186445, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:22.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.961 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:22.964 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:22.966 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:22.968 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:22.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:22.970 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:22.971 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 10 librados: watch_flush enter 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 10 librados: watch_flush exit 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 20 client.22657.objecter shutdown clearing up homeless session... 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 10 client.22657.objecter successfully canceled tick 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa46c038e40 msgr2=0x7fa46c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa46c038e40 0x7fa46c03b300 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7fa478003cc0 tx=0x7fa47800d040 comp rx=0 tx=0).stop 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 10 monclient: shutdown 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:22.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4800a5b80 msgr2=0x7fa4801d0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 --2- 172.21.15.43:0/874951218 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa4800a5b80 0x7fa4801d0b30 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa47000d620 tx=0x7fa47000dae0 comp rx=0 tx=0).stop 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47f7fe640 1 -- 172.21.15.43:0/874951218 reap_dead start 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa47f7fe640 1 -- 172.21.15.43:0/874951218 reap_dead start 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 shutdown_connections 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 >> 172.21.15.43:0/874951218 conn(0x7fa4800ab2f0 msgr2=0x7fa4801368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 shutdown_connections 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 -- 172.21.15.43:0/874951218 wait complete. 2024-04-19T00:47:22.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:22.926+0000 7fa4852e4640 1 librados: shutdown 2024-04-19T00:47:22.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:22.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-04-19T00:47:22.975 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:47:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.094+0000 7f0f022c7640 10 monclient: get_monmap_and_config 2024-04-19T00:47:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.094+0000 7f0f022c7640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.094+0000 7f0f022c7640 10 monclient: monmap: 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:23.102292+0000 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:23.102292+0000 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:23.102 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 Processor -- start 2024-04-19T00:47:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 -- start start 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient: init 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient: _add_conns ranks=[0,10,8,7,4,6,1,5,2,3,9] 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0efc138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0efc0a5b80 0x7f0efc13e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): picked mon.noname-f con 0x7f0efc0a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): picked mon.noname-e con 0x7f0efc1397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0efc0a3dc0 con 0x7f0efc138cd0 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0efc0a4410 con 0x7f0efc0a5b80 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0efc0a40a0 con 0x7f0efc1397c0 2024-04-19T00:47:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f022c7640 10 monclient(hunting): authenticate will time out at 1113.229743s 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0efbfff640 1 -- >> v1:172.21.15.43:6789/0 conn(0x7f0efc138cd0 legacy=0x7f0efc1390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc1407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53148/0 (socket says 172.21.15.43:53148) 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 -- 172.21.15.43:0/1028731436 learned_addr learned my addr 172.21.15.43:0/1028731436 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): get_auth_request con 0x7f0efc1397c0 auth_method 0 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0efb7fe640 1 --2- 172.21.15.43:0/1028731436 >> v2:172.21.15.161:3300/0 conn(0x7f0efc0a5b80 0x7f0efc13e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0efb7fe640 10 monclient(hunting): get_auth_request con 0x7f0efc0a5b80 auth_method 0 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0efb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0efb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient(hunting): handle_auth_done global_id 22490 payload 274 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 -- 172.21.15.43:0/1028731436 >> v2:172.21.15.161:3300/0 conn(0x7f0efc0a5b80 msgr2=0x7f0efc13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 --2- 172.21.15.43:0/1028731436 >> v2:172.21.15.161:3300/0 conn(0x7f0efc0a5b80 0x7f0efc13e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 -- 172.21.15.43:0/1028731436 >> v1:172.21.15.43:6789/0 conn(0x7f0efc138cd0 legacy=0x7f0efc1390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 monclient: found mon.noname-e 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.098+0000 7f0f00864640 1 -- 172.21.15.43:0/1028731436 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc0a2ae0 con 0x7f0efc1397c0 2024-04-19T00:47:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f00864640 1 --2- 172.21.15.43:0/1028731436 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc1407c0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0a7360 tx=0x7f0eec01c400 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e7f2bb0341e057ec server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 1 -- 172.21.15.43:0/1028731436 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0eec0b1070 con 0x7f0efc1397c0 2024-04-19T00:47:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: dump: 2024-04-19T00:47:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 1 -- 172.21.15.43:0/1028731436 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec0ab810 con 0x7f0efc1397c0 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 5 monclient: authenticate success, global_id 22490 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 4 set_mon_vals no callback set 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 1 -- 172.21.15.43:0/1028731436 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0efc0a20a0 con 0x7f0efc1397c0 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0efaffd640 10 monclient: dump: 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 10 monclient: shutdown 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 -- 172.21.15.43:0/1028731436 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 msgr2=0x7f0efc1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1028731436 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc1407c0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f0efc0a7360 tx=0x7f0eec01c400 comp rx=0 tx=0).stop 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 -- 172.21.15.43:0/1028731436 shutdown_connections 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1028731436 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc1407c0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1028731436 >> v2:172.21.15.161:3300/0 conn(0x7f0efc0a5b80 0x7f0efc13e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 -- 172.21.15.43:0/1028731436 >> 172.21.15.43:0/1028731436 conn(0x7f0efc0ab2f0 msgr2=0x7f0efc0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 -- 172.21.15.43:0/1028731436 shutdown_connections 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 -- 172.21.15.43:0/1028731436 wait complete. 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 10 monclient: monmap: 2024-04-19T00:47:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.119 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 librados: starting msgr at 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 librados: starting objecter 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 Processor -- start 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 -- start start 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 librados: setting wanted keys 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 1 librados: calling monclient init 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.102+0000 7f0f022c7640 10 monclient: init 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient: _add_conns ranks=[7,0,8,1,6,3,4,2,5] 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0efc0a5b80 0x7f0efc12da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): picked mon.noname-i con 0x7f0efc0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0efc138cd0 0x7f0efc12dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): picked mon.noname-a con 0x7f0efc138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc09ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): picked mon.noname-e con 0x7f0efc1397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc1438b0 con 0x7f0efc138cd0 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0efc143730 con 0x7f0efc0a5b80 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0efc143a30 con 0x7f0efc1397c0 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): authenticate will time out at 1113.236163s 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efb7fe640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0efc138cd0 msgr2=0x7f0efc12dfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efb7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0efc138cd0 0x7f0efc12dfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc09ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53154/0 (socket says 172.21.15.43:53154) 2024-04-19T00:47:23.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 -- 172.21.15.43:0/1802341506 learned_addr learned my addr 172.21.15.43:0/1802341506 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): get_auth_request con 0x7f0efc1397c0 auth_method 0 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0efc0a5b80 0x7f0efc12da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient(hunting): handle_auth_done global_id 22499 payload 824 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0efc0a5b80 msgr2=0x7f0efc12da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0efc0a5b80 0x7f0efc12da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0efc138cd0 msgr2=0x7f0efc12dfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0efc138cd0 0x7f0efc12dfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 monclient: found mon.noname-e 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc09f490 con 0x7f0efc1397c0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient: get_auth_request con 0x7f0efc0a5b80 auth_method 0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0efc0a5b80 0x7f0efc12da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f00864640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc09ba70 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0af040 tx=0x7f0eec0191a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 client.?.objecter ms_handle_connect 0x7f0efc1397c0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0eec0b1040 con 0x7f0efc1397c0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: dump: 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.123 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec0044b0 con 0x7f0efc1397c0 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 5 monclient: authenticate success, global_id 22499 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient: _renew_subs 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc09fe00 con 0x7f0efc1397c0 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0eec0abc10 con 0x7f0efc1397c0 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 client.22499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efa7fc640 4 set_mon_vals no callback set 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: dump: 2024-04-19T00:47:23.125 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.126 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient: _renew_subs 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc09e870 con 0x7f0efc1397c0 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 librados: init done 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 0x7f0efc09c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0efc06d140 con 0x7f0efc09bfb0 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 0x7f0efc09c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0eec0be060 con 0x7f0efc1397c0 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient: get_auth_request con 0x7f0efc09bfb0 auth_method 0 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038e40 0x7f0ee403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(110..110 src has 1..110) ==== 8505+0+0 (secure 0 0 0) 0x7f0eec0e3570 con 0x7f0efc1397c0 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 client.22499.objecter ms_dispatch 0x7f0efc000ba0 osd_map(110..110 src has 1..110) 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 3 client.22499.objecter handle_osd_map got epochs [110,110] > 0 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 3 client.22499.objecter handle_osd_map decoding full epoch 110 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 10 monclient(hunting): handle_auth_done global_id 22848 payload 824 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0efc0a3590 con 0x7f0efc09bfb0 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 20 client.22499.objecter dump_active .. 0 homeless 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 client.22499.objecter ms_handle_connect 0x7f0efc09bfb0 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 client.22499.objecter resend_mon_ops 2024-04-19T00:47:23.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efbfff640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 0x7f0efc09c390 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400ccd0 tx=0x7f0ef400a5f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efb7fe640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038e40 0x7f0ee403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efb7fe640 10 monclient: get_auth_request con 0x7f0ee4038e40 auth_method 0 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ef4013070 con 0x7f0efc09bfb0 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: dump: 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.129 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ef4018450 con 0x7f0efc09bfb0 2024-04-19T00:47:23.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 msgr2=0x7f0efc09c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 0x7f0efc09c390 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400ccd0 tx=0x7f0ef400a5f0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 msgr2=0x7f0efc09c390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09bfb0 0x7f0efc09c390 secure :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400ccd0 tx=0x7f0ef400a5f0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0efb7fe640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038e40 0x7f0ee403b300 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8000f00 tx=0x7f0ee8006e20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.106+0000 7f0ef8ff9640 10 client.22499.objecter ms_handle_connect 0x7f0ee4038e40 2024-04-19T00:47:23.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 0x7f0efc1331e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0efc068240 con 0x7f0efc09c8d0 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 0x7f0efc1331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient: get_auth_request con 0x7f0efc09c8d0 auth_method 0 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 10 monclient(hunting): handle_auth_done global_id 22857 payload 824 2024-04-19T00:47:23.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 1 -- 172.21.15.43:0/1802341506 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0efc0a3040 con 0x7f0efc09c8d0 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0efbfff640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 0x7f0efc1331e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4015000 tx=0x7f0ef4005d60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 client.22499.objecter ms_handle_connect 0x7f0efc09c8d0 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 client.22499.objecter resend_mon_ops 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ef400faa0 con 0x7f0efc09c8d0 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 monclient: dump: 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.137 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f0ef400f630 con 0x7f0efc09c8d0 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 msgr2=0x7f0efc1331e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 0x7f0efc1331e0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4015000 tx=0x7f0ef4005d60 comp rx=0 tx=0).stop 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 msgr2=0x7f0efc1331e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0ef8ff9640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0efc09c8d0 0x7f0efc1331e0 secure :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4015000 tx=0x7f0ef4005d60 comp rx=0 tx=0).stop 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 186629, 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.140 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.141 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:23.142 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:23.143 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.144 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:23.145 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.146 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.147 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.148 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.149 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:23.151 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.153 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.155 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.156 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:23.157 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.159 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.161 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:23.161 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:23.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 10 librados: watch_flush enter 2024-04-19T00:47:23.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 10 librados: watch_flush exit 2024-04-19T00:47:23.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 20 client.22499.objecter shutdown clearing up homeless session... 2024-04-19T00:47:23.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 10 client.22499.objecter successfully canceled tick 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038e40 msgr2=0x7f0ee403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0ee4038e40 0x7f0ee403b300 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8000f00 tx=0x7f0ee8006e20 comp rx=0 tx=0).stop 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 10 monclient: shutdown 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 msgr2=0x7f0efc09ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 --2- 172.21.15.43:0/1802341506 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0efc1397c0 0x7f0efc09ba70 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f0eec0af040 tx=0x7f0eec0191a0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f00864640 1 -- 172.21.15.43:0/1802341506 reap_dead start 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f00864640 1 -- 172.21.15.43:0/1802341506 reap_dead start 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 shutdown_connections 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 >> 172.21.15.43:0/1802341506 conn(0x7f0efc0ab2f0 msgr2=0x7f0efc131ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 shutdown_connections 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 -- 172.21.15.43:0/1802341506 wait complete. 2024-04-19T00:47:23.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.114+0000 7f0f022c7640 1 librados: shutdown 2024-04-19T00:47:23.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:23.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-04-19T00:47:23.164 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:47:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient: get_monmap_and_config 2024-04-19T00:47:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient: monmap: 2024-04-19T00:47:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:23.283052+0000 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:23.283052+0000 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:23.283 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 Processor -- start 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 -- start start 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient: init 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient: _add_conns ranks=[3,7,1,10,8,4,2,6,5,0,9] 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de80a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): picked mon.noname-g con 0x7f6de80a1f20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a1270 0x7f6de80a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): picked mon.noname-i con 0x7f6de80a1270 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6de80a4430 addr v1:172.21.15.161:6789/0 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de809fff0 con 0x7f6de80a4430 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6de80a0920 con 0x7f6de80a1f20 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6de806bf40 con 0x7f6de80a1270 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dee422640 10 monclient(hunting): authenticate will time out at 1113.410175s 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dec9bf640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6de80a4430 0x7f6de813fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:37236/0 (socket says 172.21.15.43:37236) 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6dec9bf640 1 -- 172.21.15.43:0/2723484885 learned_addr learned my addr 172.21.15.43:0/2723484885 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:47:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a1270 0x7f6de80a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de80a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): get_auth_request con 0x7f6de80a1f20 auth_method 0 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): get_auth_request con 0x7f6de80a1270 auth_method 0 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient(hunting): handle_auth_done global_id 18516 payload 274 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de6ffd640 1 -- 172.21.15.43:0/2723484885 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 378107666 0 0) 0x7f6de809fff0 con 0x7f6de80a4430 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 1 -- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a1270 msgr2=0x7f6de80a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a1270 0x7f6de80a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 1 -- 172.21.15.43:0/2723484885 >> v1:172.21.15.161:6789/0 conn(0x7f6de80a4430 legacy=0x7f6de813fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 1 monclient: found mon.noname-g 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de7fff640 1 -- 172.21.15.43:0/2723484885 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de80a02d0 con 0x7f6de80a1f20 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:47:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.278+0000 7f6de77fe640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a1270 0x7f6de80a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de7fff640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de80a39b0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80097f0 tx=0x7f6dd801c3a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=acfc0cb17b5d3e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 1 -- 172.21.15.43:0/2723484885 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd80b0070 con 0x7f6de80a1f20 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: dump: 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 1 -- 172.21.15.43:0/2723484885 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd80aa810 con 0x7f6de80a1f20 2024-04-19T00:47:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 1 -- 172.21.15.43:0/2723484885 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd80a9300 con 0x7f6de80a1f20 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 5 monclient: authenticate success, global_id 18516 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 4 set_mon_vals no callback set 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6de6ffd640 10 monclient: dump: 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 10 monclient: shutdown 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 -- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 msgr2=0x7f6de80a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de80a39b0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6dd80097f0 tx=0x7f6dd801c3a0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 -- 172.21.15.43:0/2723484885 shutdown_connections 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a1270 0x7f6de80a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 --2- 172.21.15.43:0/2723484885 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de80a39b0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 -- 172.21.15.43:0/2723484885 >> 172.21.15.43:0/2723484885 conn(0x7f6de80a85a0 msgr2=0x7f6de80a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 -- 172.21.15.43:0/2723484885 shutdown_connections 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 -- 172.21.15.43:0/2723484885 wait complete. 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 10 monclient: monmap: 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 librados: starting msgr at 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 librados: starting objecter 2024-04-19T00:47:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 Processor -- start 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 -- start start 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 librados: setting wanted keys 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 1 librados: calling monclient init 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.282+0000 7f6dee422640 10 monclient: init 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient: _add_conns ranks=[2,3,7,5,8,0,1,6,4] 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6de80a1270 0x7f6de81d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): picked mon.noname-b con 0x7f6de80a1270 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de81d11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): picked mon.noname-g con 0x7f6de80a1f20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a4430 0x7f6de813f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): picked mon.noname-i con 0x7f6de80a4430 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6de80a2480 con 0x7f6de80a1270 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6de80a2300 con 0x7f6de80a1f20 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6de8143ae0 con 0x7f6de80a4430 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): authenticate will time out at 1113.416109s 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6de80a1270 0x7f6de81d0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6de80a1270 0x7f6de81d0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35812/0 (socket says 172.21.15.43:35812) 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 -- 172.21.15.43:0/1004323980 learned_addr learned my addr 172.21.15.43:0/1004323980 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): get_auth_request con 0x7f6de80a1270 auth_method 0 2024-04-19T00:47:23.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de81d11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a4430 0x7f6de813f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 10 monclient(hunting): get_auth_request con 0x7f6de80a4430 auth_method 0 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): get_auth_request con 0x7f6de80a1f20 auth_method 0 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient(hunting): handle_auth_done global_id 22817 payload 824 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a4430 msgr2=0x7f6de813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6de80a4430 0x7f6de813f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 msgr2=0x7f6de81d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6de80a1f20 0x7f6de81d11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 monclient: found mon.noname-b 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de8125090 con 0x7f6de80a1270 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de7fff640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6de80a1270 0x7f6de81d0c80 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009710 tx=0x7f6dd8019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 client.?.objecter ms_handle_connect 0x7f6de80a1270 2024-04-19T00:47:23.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd80b0040 con 0x7f6de80a1270 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: dump: 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.306 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd80044b0 con 0x7f6de80a1270 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 5 monclient: authenticate success, global_id 22817 2024-04-19T00:47:23.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient: _renew_subs 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de8073100 con 0x7f6de80a1270 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6dd80aacf0 con 0x7f6de80a1270 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: dump: 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.308 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de67fc640 4 set_mon_vals no callback set 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 client.22817.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:23.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient: _renew_subs 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de8124db0 con 0x7f6de80a1270 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 librados: init done 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 0x7f6de813fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6de806d140 con 0x7f6de81254b0 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6dd8004040 con 0x7f6de80a1270 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 0x7f6de813fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient: get_auth_request con 0x7f6de81254b0 auth_method 0 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6dd4038e10 0x7f6dd403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(110..110 src has 1..110) ==== 8505+0+0 (secure 0 0 0) 0x7f6dd80e3980 con 0x7f6de80a1270 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 client.22817.objecter ms_dispatch 0x7f6de8000ba0 osd_map(110..110 src has 1..110) 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 3 client.22817.objecter handle_osd_map got epochs [110,110] > 0 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 3 client.22817.objecter handle_osd_map decoding full epoch 110 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 10 monclient(hunting): handle_auth_done global_id 22508 payload 824 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6de8076110 con 0x7f6de81254b0 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 20 client.22817.objecter dump_active .. 0 homeless 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 client.22817.objecter ms_handle_connect 0x7f6de81254b0 2024-04-19T00:47:23.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 client.22817.objecter resend_mon_ops 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de77fe640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 0x7f6de813fa50 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6de80a1650 tx=0x7f6ddc011a60 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6dd4038e10 0x7f6dd403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 10 monclient: get_auth_request con 0x7f6dd4038e10 auth_method 0 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ddc015ae0 con 0x7f6de81254b0 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: dump: 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.312 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6ddc015670 con 0x7f6de81254b0 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 msgr2=0x7f6de813fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 0x7f6de813fa50 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6de80a1650 tx=0x7f6ddc011a60 comp rx=0 tx=0).stop 2024-04-19T00:47:23.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 msgr2=0x7f6de813fa50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de81254b0 0x7f6de813fa50 secure :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6de80a1650 tx=0x7f6ddc011a60 comp rx=0 tx=0).stop 2024-04-19T00:47:23.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6de4ff9640 10 client.22817.objecter ms_handle_connect 0x7f6dd4038e10 2024-04-19T00:47:23.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.286+0000 7f6dec9bf640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6dd4038e10 0x7f6dd403b2d0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7f6de8140d40 tx=0x7f6de0005b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 0x7f6de81403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6de8124960 con 0x7f6de813ff90 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 0x7f6de81403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient: get_auth_request con 0x7f6de813ff90 auth_method 0 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 10 monclient(hunting): handle_auth_done global_id 22517 payload 824 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 1 -- 172.21.15.43:0/1004323980 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6de81408e0 con 0x7f6de813ff90 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de77fe640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 0x7f6de81403a0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc01c000 tx=0x7f6ddc004740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 client.22817.objecter ms_handle_connect 0x7f6de813ff90 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 client.22817.objecter resend_mon_ops 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ddc0222a0 con 0x7f6de813ff90 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 monclient: dump: 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.317 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.318 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f6ddc0222a0 con 0x7f6de813ff90 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 msgr2=0x7f6de81403a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 0x7f6de81403a0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc01c000 tx=0x7f6ddc004740 comp rx=0 tx=0).stop 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 msgr2=0x7f6de81403a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6de4ff9640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6de813ff90 0x7f6de81403a0 secure :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc01c000 tx=0x7f6ddc004740 comp rx=0 tx=0).stop 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:23.319 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 186805, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.320 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.321 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.322 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:23.323 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.324 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.325 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.326 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.328 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.331 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.333 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.336 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:23.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.340 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.341 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.342 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.342 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:23.342 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 10 librados: watch_flush enter 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 10 librados: watch_flush exit 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 20 client.22817.objecter shutdown clearing up homeless session... 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 10 client.22817.objecter successfully canceled tick 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6dd4038e10 msgr2=0x7f6dd403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6dd4038e10 0x7f6dd403b2d0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7f6de8140d40 tx=0x7f6de0005b80 comp rx=0 tx=0).stop 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 10 monclient: shutdown 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6de80a1270 msgr2=0x7f6de81d0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 --2- 172.21.15.43:0/1004323980 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6de80a1270 0x7f6de81d0c80 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009710 tx=0x7f6dd8019140 comp rx=0 tx=0).stop 2024-04-19T00:47:23.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dec9bf640 1 -- 172.21.15.43:0/1004323980 reap_dead start 2024-04-19T00:47:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dec9bf640 1 -- 172.21.15.43:0/1004323980 reap_dead start 2024-04-19T00:47:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 shutdown_connections 2024-04-19T00:47:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 >> 172.21.15.43:0/1004323980 conn(0x7f6de80a85a0 msgr2=0x7f6de8072480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 shutdown_connections 2024-04-19T00:47:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 -- 172.21.15.43:0/1004323980 wait complete. 2024-04-19T00:47:23.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.294+0000 7f6dee422640 1 librados: shutdown 2024-04-19T00:47:23.345 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:23.345 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-04-19T00:47:23.345 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:47:23.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient: get_monmap_and_config 2024-04-19T00:47:23.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient: monmap: 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:23.467325+0000 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:23.467325+0000 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.466 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.467 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.467 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.467 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.467 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:23.467 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:23.467 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 Processor -- start 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 -- start start 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient: init 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient: _add_conns ranks=[2,8,10,5,4,3,6,9,0,1,7] 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 0x7f74441390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): picked mon.noname-b con 0x7f7444138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 0x7f74441395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): picked mon.noname-e con 0x7f74440a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f74441358b0 0x7f7444135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): picked mon.noname-f con 0x7f74441358b0 addr v2:172.21.15.161:3300/0 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f74440a4410 con 0x7f74441358b0 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7444139b30 con 0x7f7444138cd0 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f74440a3dc0 con 0x7f74440a5b80 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7448a0b640 10 monclient(hunting): authenticate will time out at 1113.594379s 2024-04-19T00:47:23.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 0x7f74441390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 0x7f74441395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 0x7f74441390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35826/0 (socket says 172.21.15.43:35826) 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 0x7f74441395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53172/0 (socket says 172.21.15.43:53172) 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 1 -- 172.21.15.43:0/1385951512 learned_addr learned my addr 172.21.15.43:0/1385951512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): get_auth_request con 0x7f7444138cd0 auth_method 0 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): get_auth_request con 0x7f74440a5b80 auth_method 0 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7442d9d640 1 --2- 172.21.15.43:0/1385951512 >> v2:172.21.15.161:3300/0 conn(0x7f74441358b0 0x7f7444135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7442d9d640 10 monclient(hunting): get_auth_request con 0x7f74441358b0 auth_method 0 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7442d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7442d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7442d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient(hunting): handle_auth_done global_id 22526 payload 274 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 -- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 msgr2=0x7f74441390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 --2- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 0x7f74441390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 -- 172.21.15.43:0/1385951512 >> v2:172.21.15.161:3300/0 conn(0x7f74441358b0 msgr2=0x7f7444135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 --2- 172.21.15.43:0/1385951512 >> v2:172.21.15.161:3300/0 conn(0x7f74441358b0 0x7f7444135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 monclient: found mon.noname-e 2024-04-19T00:47:23.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:23.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f7441d9b640 1 -- 172.21.15.43:0/1385951512 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74440a40a0 con 0x7f74440a5b80 2024-04-19T00:47:23.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.462+0000 7f744259c640 1 --2- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 0x7f74441390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:47:23.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7441d9b640 1 --2- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 0x7f74441395f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f74380097c0 tx=0x7f743801c300 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d5d72946ea54af5c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 1 -- 172.21.15.43:0/1385951512 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74380b0070 con 0x7f74440a5b80 2024-04-19T00:47:23.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: dump: 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.480 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 1 -- 172.21.15.43:0/1385951512 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74380aa810 con 0x7f74440a5b80 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 5 monclient: authenticate success, global_id 22526 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 1 -- 172.21.15.43:0/1385951512 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74380a9300 con 0x7f74440a5b80 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:23.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 4 set_mon_vals no callback set 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744159a640 10 monclient: dump: 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.482 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: shutdown 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 msgr2=0x7f74441395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 0x7f74441395f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f74380097c0 tx=0x7f743801c300 comp rx=0 tx=0).stop 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- 172.21.15.43:0/1385951512 shutdown_connections 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- 172.21.15.43:0/1385951512 >> v2:172.21.15.161:3300/0 conn(0x7f74441358b0 0x7f7444135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f74440a5b80 0x7f74441395f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- 172.21.15.43:0/1385951512 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7444138cd0 0x7f74441390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- 172.21.15.43:0/1385951512 >> 172.21.15.43:0/1385951512 conn(0x7f74440ab2f0 msgr2=0x7f74440ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- 172.21.15.43:0/1385951512 shutdown_connections 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- 172.21.15.43:0/1385951512 wait complete. 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: monmap: 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 librados: starting msgr at 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 librados: starting objecter 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 Processor -- start 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- start start 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 librados: setting wanted keys 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 librados: calling monclient init 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: init 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient: _add_conns ranks=[0,5,1,4,8,2,7,3,6] 2024-04-19T00:47:23.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74440a5b80 0x7f74441d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): picked mon.noname-a con 0x7f74440a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f74441358b0 0x7f744413ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): picked mon.noname-h con 0x7f74441358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444138cd0 0x7f744413f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): picked mon.noname-f con 0x7f7444138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f74440a2010 con 0x7f74440a5b80 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f74440a1e90 con 0x7f7444138cd0 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7444142aa0 con 0x7f74441358b0 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f7448a0b640 10 monclient(hunting): authenticate will time out at 1113.599763s 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744259c640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74440a5b80 msgr2=0x7f74441d0c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.466+0000 7f744259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74440a5b80 0x7f74441d0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444138cd0 0x7f744413f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444138cd0 0x7f744413f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:58392/0 (socket says 172.21.15.43:58392) 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 -- 172.21.15.43:0/576078649 learned_addr learned my addr 172.21.15.43:0/576078649 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): get_auth_request con 0x7f7444138cd0 auth_method 0 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f74441358b0 0x7f744413ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): get_auth_request con 0x7f74441358b0 auth_method 0 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient(hunting): handle_auth_done global_id 23464 payload 824 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f74441358b0 msgr2=0x7f744413ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f74441358b0 0x7f744413ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74440a5b80 msgr2=0x7f74441d0c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f74440a5b80 0x7f74441d0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 monclient: found mon.noname-f 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744413b3d0 con 0x7f7444138cd0 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7442d9d640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444138cd0 0x7f744413f140 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f743c00d800 tx=0x7f743c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 client.?.objecter ms_handle_connect 0x7f7444138cd0 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f743c00bb30 con 0x7f7444138cd0 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: dump: 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.489 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 5 monclient: authenticate success, global_id 23464 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 monclient: _renew_subs 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74440a3590 con 0x7f7444138cd0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743c019070 con 0x7f7444138cd0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 client.23464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 monclient: _renew_subs 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7440d99640 4 set_mon_vals no callback set 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744413f680 con 0x7f7444138cd0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f743c00bcd0 con 0x7f7444138cd0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 1 librados: init done 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: dump: 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.492 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 0x7f74441337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7444139f90 con 0x7f7444133410 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 0x7f74441337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient: get_auth_request con 0x7f7444133410 auth_method 0 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f743c00b6c0 con 0x7f7444138cd0 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7424038e10 0x7f742403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 8505+0+0 (secure 0 0 0) 0x7f743c00f040 con 0x7f7444138cd0 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 client.23464.objecter ms_dispatch 0x7f7444000ba0 osd_map(110..110 src has 1..110) 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 3 client.23464.objecter handle_osd_map got epochs [110,110] > 0 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 3 client.23464.objecter handle_osd_map decoding full epoch 110 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 10 monclient(hunting): handle_auth_done global_id 23473 payload 824 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7444135670 con 0x7f7444133410 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 20 client.23464.objecter dump_active .. 0 homeless 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7424038e10 0x7f742403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7441d9b640 10 monclient: get_auth_request con 0x7f7424038e10 auth_method 0 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 client.23464.objecter ms_handle_connect 0x7f7444133410 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 client.23464.objecter resend_mon_ops 2024-04-19T00:47:23.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f744259c640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 0x7f74441337f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f74340072a0 tx=0x7f743400b480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74340090d0 con 0x7f7444133410 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.470+0000 7f7432ffd640 10 monclient: dump: 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.495 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7441d9b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7424038e10 0x7f742403b2d0 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f74380ad000 tx=0x7f74380a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 10 client.23464.objecter ms_handle_connect 0x7f7424038e10 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7434019070 con 0x7f7444133410 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 msgr2=0x7f74441337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 0x7f74441337f0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f74340072a0 tx=0x7f743400b480 comp rx=0 tx=0).stop 2024-04-19T00:47:23.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 msgr2=0x7f74441337f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.474+0000 7f7432ffd640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133410 0x7f74441337f0 secure :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f74340072a0 tx=0x7f743400b480 comp rx=0 tx=0).stop 2024-04-19T00:47:23.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f7448a0b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:23.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f7448a0b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 0x7f74441377c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f7448a0b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f744406d140 con 0x7f7444133d30 2024-04-19T00:47:23.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 0x7f74441377c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient: get_auth_request con 0x7f7444133d30 auth_method 0 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 10 monclient(hunting): handle_auth_done global_id 23482 payload 824 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.478+0000 7f744259c640 1 -- 172.21.15.43:0/576078649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7444134fb0 con 0x7f7444133d30 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f744259c640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 0x7f74441377c0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f7434018000 tx=0x7f7434004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 client.23464.objecter ms_handle_connect 0x7f7444133d30 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 client.23464.objecter resend_mon_ops 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f74340092a0 con 0x7f7444133d30 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 monclient: dump: 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.500 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f74340092a0 con 0x7f7444133d30 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 msgr2=0x7f74441377c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 0x7f74441377c0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f7434018000 tx=0x7f7434004ac0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 msgr2=0x7f74441377c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7432ffd640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444133d30 0x7f74441377c0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:23.501 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 186952, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:23.502 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.503 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.504 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.505 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:23.507 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.510 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.515 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:23.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:23.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:23.522 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.523 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:23.524 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 10 librados: watch_flush enter 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 10 librados: watch_flush exit 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 20 client.23464.objecter shutdown clearing up homeless session... 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 10 client.23464.objecter successfully canceled tick 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7424038e10 msgr2=0x7f742403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7424038e10 0x7f742403b2d0 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7f74380ad000 tx=0x7f74380a6000 comp rx=0 tx=0).stop 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 10 monclient: shutdown 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444138cd0 msgr2=0x7f744413f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 --2- 172.21.15.43:0/576078649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7444138cd0 0x7f744413f140 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f743c00d800 tx=0x7f743c00dcc0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7442d9d640 1 -- 172.21.15.43:0/576078649 reap_dead start 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7442d9d640 1 -- 172.21.15.43:0/576078649 reap_dead start 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 shutdown_connections 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 >> 172.21.15.43:0/576078649 conn(0x7f74440ab2f0 msgr2=0x7f74440a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 shutdown_connections 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 -- 172.21.15.43:0/576078649 wait complete. 2024-04-19T00:47:23.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.482+0000 7f7448a0b640 1 librados: shutdown 2024-04-19T00:47:23.527 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:23.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-04-19T00:47:23.528 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.654+0000 7f977938f640 10 monclient: get_monmap_and_config 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.654+0000 7f977938f640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.654+0000 7f977938f640 10 monclient: monmap: 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:23.661958+0000 2024-04-19T00:47:23.660 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:23.661958+0000 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:23.661 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 Processor -- start 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 -- start start 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient: init 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient: _add_conns ranks=[7,8,9,1,4,3,2,5,10,6,0] 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a1ec0 0x7f97740a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): picked mon.noname-i con 0x7f97740a1ec0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 0x7f97740a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): picked mon.noname-e con 0x7f97740a22a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f97740a12e0 0x7f97740a4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): picked mon.noname-a con 0x7f97740a12e0 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f977406c4b0 con 0x7f97740a12e0 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f977406bfc0 con 0x7f97740a1ec0 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f97740a0060 con 0x7f97740a22a0 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f977938f640 10 monclient(hunting): authenticate will time out at 1113.788873s 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f97737fe640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7f97740a12e0 msgr2=0x7f97740a4430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f97737fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f97740a12e0 0x7f97740a4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 0x7f97740a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 0x7f97740a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53180/0 (socket says 172.21.15.43:53180) 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 -- 172.21.15.43:0/3807442585 learned_addr learned my addr 172.21.15.43:0/3807442585 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): get_auth_request con 0x7f97740a22a0 auth_method 0 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f9772ffd640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a1ec0 0x7f97740a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f9772ffd640 10 monclient(hunting): get_auth_request con 0x7f97740a1ec0 auth_method 0 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f9772ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f9772ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f9772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient(hunting): handle_auth_done global_id 22535 payload 274 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 -- 172.21.15.43:0/3807442585 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a1ec0 msgr2=0x7f97740a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a1ec0 0x7f97740a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 -- 172.21.15.43:0/3807442585 >> v2:172.21.15.43:3300/0 conn(0x7f97740a12e0 msgr2=0x7f97740a4430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 --2- 172.21.15.43:0/3807442585 >> v2:172.21.15.43:3300/0 conn(0x7f97740a12e0 0x7f97740a4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 monclient: found mon.noname-e 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 -- 172.21.15.43:0/3807442585 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97740a0340 con 0x7f97740a22a0 2024-04-19T00:47:23.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f9772ffd640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a1ec0 0x7f97740a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:47:23.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.658+0000 7f976bfff640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 0x7f97740a3ef0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f975c009870 tx=0x7f975c01c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e3ee2119bdc959ce server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 1 -- 172.21.15.43:0/3807442585 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f975c0b0070 con 0x7f97740a22a0 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: dump: 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.676 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 1 -- 172.21.15.43:0/3807442585 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975c0aa810 con 0x7f97740a22a0 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 1 -- 172.21.15.43:0/3807442585 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f975c0a9300 con 0x7f97740a22a0 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 5 monclient: authenticate success, global_id 22535 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:23.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 4 set_mon_vals no callback set 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97727fc640 10 monclient: dump: 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.678 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: shutdown 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- 172.21.15.43:0/3807442585 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 msgr2=0x7f97740a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 0x7f97740a3ef0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f975c009870 tx=0x7f975c01c440 comp rx=0 tx=0).stop 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- 172.21.15.43:0/3807442585 shutdown_connections 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- 172.21.15.43:0/3807442585 >> v2:172.21.15.43:3300/0 conn(0x7f97740a12e0 0x7f97740a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f97740a22a0 0x7f97740a3ef0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- 172.21.15.43:0/3807442585 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a1ec0 0x7f97740a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- 172.21.15.43:0/3807442585 >> 172.21.15.43:0/3807442585 conn(0x7f97740a8600 msgr2=0x7f97740a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- 172.21.15.43:0/3807442585 shutdown_connections 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- 172.21.15.43:0/3807442585 wait complete. 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: monmap: 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:23.680 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 librados: starting msgr at 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 librados: starting objecter 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 Processor -- start 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- start start 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 librados: setting wanted keys 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 librados: calling monclient init 2024-04-19T00:47:23.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: init 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient: _add_conns ranks=[7,4,5,3,6,1,8,2,0] 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a12e0 0x7f9774135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): picked mon.noname-i con 0x7f97740a12e0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f97740a1ec0 0x7f9774136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): picked mon.noname-c con 0x7f97740a1ec0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f97741d0de0 0x7f97741d1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): picked mon.noname-h con 0x7f97741d0de0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9774142b60 con 0x7f97740a1ec0 2024-04-19T00:47:23.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f97741429e0 con 0x7f97741d0de0 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9774142ce0 con 0x7f97740a12e0 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f977938f640 10 monclient(hunting): authenticate will time out at 1113.794642s 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f97740a1ec0 0x7f9774136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f97740a1ec0 0x7f9774136430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:32984/0 (socket says 172.21.15.43:32984) 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 -- 172.21.15.43:0/1335653154 learned_addr learned my addr 172.21.15.43:0/1335653154 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): get_auth_request con 0x7f97740a1ec0 auth_method 0 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97737fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f97741d0de0 0x7f97741d1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f9772ffd640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a12e0 0x7f9774135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient(hunting): handle_auth_done global_id 22684 payload 824 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a12e0 msgr2=0x7f9774135ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a12e0 0x7f9774135ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f97741d0de0 msgr2=0x7f97741d1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f97741d0de0 0x7f97741d1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 monclient: found mon.noname-c 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f977409f4a0 con 0x7f97740a1ec0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97737fe640 10 monclient: get_auth_request con 0x7f97741d0de0 auth_method 0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f9772ffd640 10 monclient: get_auth_request con 0x7f97740a12e0 auth_method 0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97737fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f97741d0de0 0x7f97741d1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f9772ffd640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f97740a12e0 0x7f9774135ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976bfff640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f97740a1ec0 0x7f9774136430 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f975c0ae040 tx=0x7f975c0191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 client.?.objecter ms_handle_connect 0x7f97740a1ec0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f975c0b8020 con 0x7f97740a1ec0 2024-04-19T00:47:23.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: dump: 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.686 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f975c0044b0 con 0x7f97740a1ec0 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f975c0b0040 con 0x7f97740a1ec0 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f976b7fe640 10 monclient: dump: 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.688 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.662+0000 7f97717fa640 4 set_mon_vals no callback set 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 5 monclient: authenticate success, global_id 22684 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 monclient: _renew_subs 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f977409d530 con 0x7f97740a1ec0 2024-04-19T00:47:23.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 client.22684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 monclient: _renew_subs 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f977413f5e0 con 0x7f97740a1ec0 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 1 librados: init done 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 0x7f97741d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f977406d140 con 0x7f97740a22a0 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f975c0bb020 con 0x7f97740a1ec0 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 0x7f97741d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient: get_auth_request con 0x7f97740a22a0 auth_method 0 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9754038e10 0x7f975403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(111..111 src has 1..111) ==== 8130+0+0 (secure 0 0 0) 0x7f975c0e37f0 con 0x7f97740a1ec0 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 client.22684.objecter ms_dispatch 0x7f9774000ba0 osd_map(111..111 src has 1..111) 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 3 client.22684.objecter handle_osd_map got epochs [111,111] > 0 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 3 client.22684.objecter handle_osd_map decoding full epoch 111 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 20 client.22684.objecter dump_active .. 0 homeless 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f97737fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9754038e10 0x7f975403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f97737fe640 10 monclient: get_auth_request con 0x7f9754038e10 auth_method 0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 10 monclient(hunting): handle_auth_done global_id 18543 payload 824 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f977409ef70 con 0x7f97740a22a0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 client.22684.objecter ms_handle_connect 0x7f9754038e10 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f97737fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9754038e10 0x7f975403b2d0 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f9774070de0 tx=0x7f976c007340 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 client.22684.objecter ms_handle_connect 0x7f97740a22a0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 client.22684.objecter resend_mon_ops 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f9772ffd640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 0x7f97741d17a0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f976400f4b0 tx=0x7f976400f970 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9764018070 con 0x7f97740a22a0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 monclient: dump: 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.693 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9764014690 con 0x7f97740a22a0 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 msgr2=0x7f97741d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 0x7f97741d17a0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f976400f4b0 tx=0x7f976400f970 comp rx=0 tx=0).stop 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 msgr2=0x7f97741d17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.666+0000 7f976b7fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f97740a22a0 0x7f97741d17a0 secure :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f976400f4b0 tx=0x7f976400f970 comp rx=0 tx=0).stop 2024-04-19T00:47:23.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:23.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 0x7f9774071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9774136970 con 0x7f9774070f70 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 0x7f9774071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient: get_auth_request con 0x7f9774070f70 auth_method 0 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 10 monclient(hunting): handle_auth_done global_id 18552 payload 824 2024-04-19T00:47:23.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 1 -- 172.21.15.43:0/1335653154 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97740008e0 con 0x7f9774070f70 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f9772ffd640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 0x7f9774071350 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f976401a000 tx=0x7f97640046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 client.22684.objecter ms_handle_connect 0x7f9774070f70 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 client.22684.objecter resend_mon_ops 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f976401dd20 con 0x7f9774070f70 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 monclient: dump: 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.698 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f976401dd20 con 0x7f9774070f70 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:23.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 msgr2=0x7f9774071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 0x7f9774071350 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f976401a000 tx=0x7f97640046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 msgr2=0x7f9774071350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f976b7fe640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9774070f70 0x7f9774071350 secure :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f976401a000 tx=0x7f97640046d0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:23.700 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 187160, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.701 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.702 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:23.704 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.705 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:23.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.716 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:23.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.721 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:47:23.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.723 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:23.724 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:23.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 10 librados: watch_flush enter 2024-04-19T00:47:23.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 10 librados: watch_flush exit 2024-04-19T00:47:23.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 20 client.22684.objecter shutdown clearing up homeless session... 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 10 client.22684.objecter successfully canceled tick 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9754038e10 msgr2=0x7f975403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9754038e10 0x7f975403b2d0 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f9774070de0 tx=0x7f976c007340 comp rx=0 tx=0).stop 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 10 monclient: shutdown 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f97740a1ec0 msgr2=0x7f9774136430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 --2- 172.21.15.43:0/1335653154 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f97740a1ec0 0x7f9774136430 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f975c0ae040 tx=0x7f975c0191e0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f97737fe640 1 -- 172.21.15.43:0/1335653154 reap_dead start 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f97737fe640 1 -- 172.21.15.43:0/1335653154 reap_dead start 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 shutdown_connections 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 >> 172.21.15.43:0/1335653154 conn(0x7f97740a8600 msgr2=0x7f9774070410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 shutdown_connections 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 -- 172.21.15.43:0/1335653154 wait complete. 2024-04-19T00:47:23.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.674+0000 7f977938f640 1 librados: shutdown 2024-04-19T00:47:23.727 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:23.727 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-04-19T00:47:23.727 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient: get_monmap_and_config 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient: monmap: 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:23.847000+0000 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:23.847000+0000 2024-04-19T00:47:23.845 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:23.846 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 Processor -- start 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 -- start start 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient: init 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient: _add_conns ranks=[4,5,3,1,8,6,2,0,10,9,7] 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 0x7f93c00a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): picked mon.noname-c con 0x7f93c00a1e00 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c00a1190 0x7f93c00a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): picked mon.noname-h con 0x7f93c00a1190 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:23.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a4350 0x7f93c013fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): picked mon.noname-g con 0x7f93c00a4350 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f93c006c690 con 0x7f93c00a4350 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f93c00735a0 con 0x7f93c00a1e00 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f93c009fff0 con 0x7f93c00a1190 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c82bf640 10 monclient(hunting): authenticate will time out at 1113.973779s 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 0x7f93c00a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 0x7f93c00a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:33000/0 (socket says 172.21.15.43:33000) 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 -- 172.21.15.43:0/521820283 learned_addr learned my addr 172.21.15.43:0/521820283 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): get_auth_request con 0x7f93c00a1e00 auth_method 0 2024-04-19T00:47:23.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c685c640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a4350 0x7f93c013fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c585a640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c00a1190 0x7f93c00a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c685c640 10 monclient(hunting): get_auth_request con 0x7f93c00a4350 auth_method 0 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c685c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c685c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c685c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c585a640 10 monclient(hunting): get_auth_request con 0x7f93c00a1190 auth_method 0 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c585a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c585a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c585a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient(hunting): handle_auth_done global_id 22693 payload 274 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 -- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c00a1190 msgr2=0x7f93c00a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c00a1190 0x7f93c00a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 -- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a4350 msgr2=0x7f93c013fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a4350 0x7f93c013fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 monclient: found mon.noname-c 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c605b640 1 -- 172.21.15.43:0/521820283 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c006c330 con 0x7f93c00a1e00 2024-04-19T00:47:23.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.842+0000 7f93c685c640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a4350 0x7f93c013fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c605b640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 0x7f93c00a38d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0098a0 tx=0x7f93bc01c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a30fe2f543bdae45 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 1 -- 172.21.15.43:0/521820283 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93bc0b0070 con 0x7f93c00a1e00 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: dump: 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.859 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.860 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 1 -- 172.21.15.43:0/521820283 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93bc0aa810 con 0x7f93c00a1e00 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 1 -- 172.21.15.43:0/521820283 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93bc0a9300 con 0x7f93c00a1e00 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 5 monclient: authenticate success, global_id 22693 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 4 set_mon_vals no callback set 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c5059640 10 monclient: dump: 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.861 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.862 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: shutdown 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- 172.21.15.43:0/521820283 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 msgr2=0x7f93c00a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 0x7f93c00a38d0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f93bc0098a0 tx=0x7f93bc01c440 comp rx=0 tx=0).stop 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- 172.21.15.43:0/521820283 shutdown_connections 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a4350 0x7f93c013fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c00a1190 0x7f93c00a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- 172.21.15.43:0/521820283 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a1e00 0x7f93c00a38d0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- 172.21.15.43:0/521820283 >> 172.21.15.43:0/521820283 conn(0x7f93c00a84c0 msgr2=0x7f93c00a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- 172.21.15.43:0/521820283 shutdown_connections 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- 172.21.15.43:0/521820283 wait complete. 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: build_initial_monmap 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: monmap: 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:23.863 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:23.864 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 librados: starting msgr at 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 librados: starting objecter 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 Processor -- start 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- start start 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 librados: setting wanted keys 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 librados: calling monclient init 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: init 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient: _add_conns ranks=[3,0,4,2,1,7,6,8,5] 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a1190 0x7f93c0122e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): picked mon.noname-g con 0x7f93c00a1190 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c00a1e00 0x7f93c0124e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): picked mon.noname-a con 0x7f93c00a1e00 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a4350 0x7f93c013acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): picked mon.noname-c con 0x7f93c00a4350 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f93c0142820 con 0x7f93c00a1e00 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f93c00a23f0 con 0x7f93c00a1190 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f93c01429a0 con 0x7f93c00a4350 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c82bf640 10 monclient(hunting): authenticate will time out at 1113.979864s 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c585a640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c00a1e00 msgr2=0x7f93c0124e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c585a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c00a1e00 0x7f93c0124e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.846+0000 7f93c685c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a4350 0x7f93c013acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a4350 0x7f93c013acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:33012/0 (socket says 172.21.15.43:33012) 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 -- 172.21.15.43:0/1042912364 learned_addr learned my addr 172.21.15.43:0/1042912364 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:23.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): get_auth_request con 0x7f93c00a4350 auth_method 0 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a1190 0x7f93c0122e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): get_auth_request con 0x7f93c00a1190 auth_method 0 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient(hunting): handle_auth_done global_id 22702 payload 824 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a1190 msgr2=0x7f93c0122e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93c00a1190 0x7f93c0122e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c00a1e00 msgr2=0x7f93c0124e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93c00a1e00 0x7f93c0124e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient: _finish_hunting 0 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 monclient: found mon.noname-c 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c01242f0 con 0x7f93c00a4350 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c685c640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a4350 0x7f93c013acb0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f93b8007f50 tx=0x7f93b800d560 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 client.?.objecter ms_handle_connect 0x7f93c00a4350 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93b801a070 con 0x7f93c00a4350 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: dump: 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.868 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93b8014810 con 0x7f93c00a4350 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 5 monclient: authenticate success, global_id 22702 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 monclient: _renew_subs 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93c0123ce0 con 0x7f93c00a4350 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93b8014e30 con 0x7f93c00a4350 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: dump: 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.870 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.871 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b7fff640 4 set_mon_vals no callback set 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 client.22702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 monclient: _renew_subs 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f93c013b1f0 con 0x7f93c00a4350 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 1 librados: init done 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 0x7f93c01d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f93c009eef0 con 0x7f93c0073050 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f93b80149c0 con 0x7f93c00a4350 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 0x7f93c01d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93a4038e10 0x7f93a403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient: get_auth_request con 0x7f93c0073050 auth_method 0 2024-04-19T00:47:23.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(111..111 src has 1..111) ==== 8130+0+0 (secure 0 0 0) 0x7f93b8049d80 con 0x7f93c00a4350 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 client.22702.objecter ms_dispatch 0x7f93c0000ba0 osd_map(111..111 src has 1..111) 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 3 client.22702.objecter handle_osd_map got epochs [111,111] > 0 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 3 client.22702.objecter handle_osd_map decoding full epoch 111 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 20 client.22702.objecter dump_active .. 0 homeless 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c585a640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93a4038e10 0x7f93a403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c585a640 10 monclient: get_auth_request con 0x7f93a4038e10 auth_method 0 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 10 monclient(hunting): handle_auth_done global_id 44366 payload 824 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93c00a7810 con 0x7f93c0073050 2024-04-19T00:47:23.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 client.22702.objecter ms_handle_connect 0x7f93c0073050 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 client.22702.objecter resend_mon_ops 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c605b640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 0x7f93c01d0c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009870 tx=0x7f93bc0191e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 client.22702.objecter ms_handle_connect 0x7f93a4038e10 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93c585a640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93a4038e10 0x7f93a403b2d0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f93c0068c20 tx=0x7f93b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93bc0b9030 con 0x7f93c0073050 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: dump: 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.874 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f93bc0b1040 con 0x7f93c0073050 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 msgr2=0x7f93c01d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 0x7f93c01d0c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009870 tx=0x7f93bc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 msgr2=0x7f93c01d0c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.850+0000 7f93b6ffd640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c0073050 0x7f93c01d0c40 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009870 tx=0x7f93bc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c82bf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c82bf640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 0x7f93c01d1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c82bf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f93c01d1ad0 con 0x7f93c01d1180 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 0x7f93c01d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient: get_auth_request con 0x7f93c01d1180 auth_method 0 2024-04-19T00:47:23.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 10 monclient(hunting): handle_auth_done global_id 44375 payload 824 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 1 -- 172.21.15.43:0/1042912364 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93c006c8d0 con 0x7f93c01d1180 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93c605b640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 0x7f93c01d1590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93bc01c480 tx=0x7f93bc0a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93b6ffd640 10 client.22702.objecter ms_handle_connect 0x7f93c01d1180 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93b6ffd640 10 client.22702.objecter resend_mon_ops 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93bc0b9040 con 0x7f93c01d1180 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93b6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:47:23.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.858+0000 7f93b6ffd640 10 monclient: dump: 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:23.879 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f93bc003b80 con 0x7f93c01d1180 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 msgr2=0x7f93c01d1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 0x7f93c01d1590 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93bc01c480 tx=0x7f93bc0a8000 comp rx=0 tx=0).stop 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 msgr2=0x7f93c01d1590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93b6ffd640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93c01d1180 0x7f93c01d1590 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f93bc01c480 tx=0x7f93bc0a8000 comp rx=0 tx=0).stop 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:23.880 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 43591, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.882 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.883 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:23.884 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:23.885 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:23.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.890 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:23.893 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.895 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:23.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:23.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.900 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:23.901 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 10 librados: watch_flush enter 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 10 librados: watch_flush exit 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 20 client.22702.objecter shutdown clearing up homeless session... 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 10 client.22702.objecter successfully canceled tick 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93a4038e10 msgr2=0x7f93a403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93a4038e10 0x7f93a403b2d0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f93c0068c20 tx=0x7f93b0008040 comp rx=0 tx=0).stop 2024-04-19T00:47:23.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 10 monclient: shutdown 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a4350 msgr2=0x7f93c013acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 --2- 172.21.15.43:0/1042912364 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f93c00a4350 0x7f93c013acb0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f93b8007f50 tx=0x7f93b800d560 comp rx=0 tx=0).stop 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c685c640 1 -- 172.21.15.43:0/1042912364 reap_dead start 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c685c640 1 -- 172.21.15.43:0/1042912364 reap_dead start 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 shutdown_connections 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 >> 172.21.15.43:0/1042912364 conn(0x7f93c00a84c0 msgr2=0x7f93c0075900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 shutdown_connections 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 -- 172.21.15.43:0/1042912364 wait complete. 2024-04-19T00:47:23.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:23.862+0000 7f93c82bf640 1 librados: shutdown 2024-04-19T00:47:23.904 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:23.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-04-19T00:47:23.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:47:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.022+0000 7feca079d640 10 monclient: get_monmap_and_config 2024-04-19T00:47:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.022+0000 7feca079d640 10 monclient: build_initial_monmap 2024-04-19T00:47:24.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.022+0000 7feca079d640 10 monclient: monmap: 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:47:24.030155+0000 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:47:24.030155+0000 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:24.029 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:24.030 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:24.030 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:47:24.030 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:47:24.030 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 Processor -- start 2024-04-19T00:47:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 -- start start 2024-04-19T00:47:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient: init 2024-04-19T00:47:24.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient: _add_conns ranks=[3,9,5,6,0,7,4,2,1,8,10] 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec98138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): picked mon.noname-g con 0x7fec98138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fec980a5bb0 0x7fec98139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): picked mon.noname-a con 0x7fec980a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec98137960 0x7fec98137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): picked mon.noname-h con 0x7fec98137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fec980a1020 con 0x7fec980a5bb0 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fec981398b0 con 0x7fec98138a50 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fec980a0780 con 0x7fec98137960 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:24.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7feca079d640 10 monclient(hunting): authenticate will time out at 1114.157597s 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9dd38640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7fec980a5bb0 msgr2=0x7fec98139370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9dd38640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fec980a5bb0 0x7fec98139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec98138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec98137960 0x7fec98137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec98138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:38770/0 (socket says 172.21.15.43:38770) 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 -- 172.21.15.43:0/2034632771 learned_addr learned my addr 172.21.15.43:0/2034632771 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): get_auth_request con 0x7fec98138a50 auth_method 0 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): get_auth_request con 0x7fec98137960 auth_method 0 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9ed3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient(hunting): handle_auth_done global_id 18579 payload 274 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 -- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec98137960 msgr2=0x7fec98137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 --2- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec98137960 0x7fec98137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 -- 172.21.15.43:0/2034632771 >> v2:172.21.15.43:3300/0 conn(0x7fec980a5bb0 msgr2=0x7fec98139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 --2- 172.21.15.43:0/2034632771 >> v2:172.21.15.43:3300/0 conn(0x7fec980a5bb0 0x7fec98139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient: _finish_hunting 0 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 monclient: found mon.noname-g 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:24.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:47:24.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.026+0000 7fec9e539640 1 -- 172.21.15.43:0/2034632771 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec980a0d40 con 0x7fec98138a50 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 1 --2- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec98138e30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec94009870 tx=0x7fec9401c410 comp rx=0 tx=0).ready entity=mon.3 client_cookie=29c5b7a64021b621 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 1 -- 172.21.15.43:0/2034632771 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec940b2070 con 0x7fec98138a50 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: dump: 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.042 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: _finish_auth 0 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: _check_auth_tickets 2024-04-19T00:47:24.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 1 -- 172.21.15.43:0/2034632771 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec940ab7b0 con 0x7fec98138a50 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 1 -- 172.21.15.43:0/2034632771 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec940abe50 con 0x7fec98138a50 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 5 monclient: authenticate success, global_id 18579 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: get_monmap_and_config success 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 4 set_mon_vals no callback set 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9d537640 10 monclient: dump: 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.044 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: shutdown 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 msgr2=0x7fec98138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec98138e30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec94009870 tx=0x7fec9401c410 comp rx=0 tx=0).stop 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- 172.21.15.43:0/2034632771 shutdown_connections 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec98137960 0x7fec98137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- 172.21.15.43:0/2034632771 >> v2:172.21.15.43:3300/0 conn(0x7fec980a5bb0 0x7fec98139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- 172.21.15.43:0/2034632771 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec98138e30 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- 172.21.15.43:0/2034632771 >> 172.21.15.43:0/2034632771 conn(0x7fec980a7cb0 msgr2=0x7fec980a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- 172.21.15.43:0/2034632771 shutdown_connections 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- 172.21.15.43:0/2034632771 wait complete. 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: build_initial_monmap 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: monmap: 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:47:24.046 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 librados: starting msgr at 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 librados: starting objecter 2024-04-19T00:47:24.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 Processor -- start 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- start start 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 librados: setting wanted keys 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 librados: calling monclient init 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: init 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: _reopen_session rank -1 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient: _add_conns ranks=[2,1,3,5,6,7,4,8,0] 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec980a5bb0 0x7fec981d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): picked mon.noname-b con 0x7fec980a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec98137960 0x7fec98136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): picked mon.noname-f con 0x7fec98137960 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec9813c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): picked mon.noname-g con 0x7fec98138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fec980a2100 con 0x7fec98137960 2024-04-19T00:47:24.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fec980a1f80 con 0x7fec980a5bb0 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fec98142810 con 0x7fec98138a50 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): _renew_subs 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7feca079d640 10 monclient(hunting): authenticate will time out at 1114.163526s 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec980a5bb0 0x7fec981d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec980a5bb0 0x7fec981d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35836/0 (socket says 172.21.15.43:35836) 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 1 -- 172.21.15.43:0/457869914 learned_addr learned my addr 172.21.15.43:0/457869914 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): get_auth_request con 0x7fec980a5bb0 auth_method 0 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:24.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9e539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9ed3a640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec9813c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.030+0000 7fec9ed3a640 10 monclient(hunting): get_auth_request con 0x7fec98138a50 auth_method 0 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9ed3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9ed3a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9ed3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec98137960 0x7fec98136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): get_auth_request con 0x7fec98137960 auth_method 0 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 10 monclient(hunting): handle_auth_done global_id 22835 payload 824 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 msgr2=0x7fec9813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fec98138a50 0x7fec9813c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec98137960 msgr2=0x7fec98136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec98137960 0x7fec98136b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 10 monclient: _finish_hunting 0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 monclient: found mon.noname-b 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9809fc90 con 0x7fec980a5bb0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9e539640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec980a5bb0 0x7fec981d09b0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fec940b0040 tx=0x7fec940191b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 client.?.objecter ms_handle_connect 0x7fec980a5bb0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec940b2040 con 0x7fec980a5bb0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:47:24.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: dump: 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:24.052 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: _finish_auth 0 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: _check_auth_tickets 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec940abdd0 con 0x7fec980a5bb0 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 5 monclient: authenticate success, global_id 22835 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 monclient: _renew_subs 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9809ee60 con 0x7fec980a5bb0 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec940043f0 con 0x7fec980a5bb0 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8ffff640 4 set_mon_vals no callback set 2024-04-19T00:47:24.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 client.22835.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: dump: 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:24.054 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 monclient: _renew_subs 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec981224f0 con 0x7fec980a5bb0 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 1 librados: init done 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 0x7fec9813ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fec98122170 con 0x7fec98137090 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fec940ab780 con 0x7fec980a5bb0 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 0x7fec9813ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient: get_auth_request con 0x7fec98137090 auth_method 0 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec7c038e10 0x7fec7c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(111..111 src has 1..111) ==== 8130+0+0 (secure 0 0 0) 0x7fec940e39b0 con 0x7fec980a5bb0 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 client.22835.objecter ms_dispatch 0x7fec98000ba0 osd_map(111..111 src has 1..111) 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 3 client.22835.objecter handle_osd_map got epochs [111,111] > 0 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 3 client.22835.objecter handle_osd_map decoding full epoch 111 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 20 client.22835.objecter dump_active .. 0 homeless 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9ed3a640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec7c038e10 0x7fec7c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9ed3a640 10 monclient: get_auth_request con 0x7fec7c038e10 auth_method 0 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 10 monclient(hunting): handle_auth_done global_id 19060 payload 824 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec98121800 con 0x7fec98137090 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 client.22835.objecter ms_handle_connect 0x7fec98137090 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 client.22835.objecter resend_mon_ops 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9dd38640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 0x7fec9813ceb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fec9806e8c0 tx=0x7fec8800db90 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec88003970 con 0x7fec98137090 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec9ed3a640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec7c038e10 0x7fec7c03b2d0 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7fec9813eff0 tx=0x7fec9000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: dump: 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.057 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 client.22835.objecter ms_handle_connect 0x7fec7c038e10 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec88005a20 con 0x7fec98137090 2024-04-19T00:47:24.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:47:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:47:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 msgr2=0x7fec9813ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 0x7fec9813ceb0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fec9806e8c0 tx=0x7fec8800db90 comp rx=0 tx=0).stop 2024-04-19T00:47:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 msgr2=0x7fec9813ceb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:24.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.034+0000 7fec8effd640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec98137090 0x7fec9813ceb0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7feca079d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7feca079d640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 0x7fec980a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7feca079d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fec9806d140 con 0x7fec9813d3f0 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 0x7fec980a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient: get_auth_request con 0x7fec9813d3f0 auth_method 0 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 10 monclient(hunting): handle_auth_done global_id 19069 payload 824 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 1 -- 172.21.15.43:0/457869914 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec9813d800 con 0x7fec9813d3f0 2024-04-19T00:47:24.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec9dd38640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 0x7fec980a3140 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fec88000c10 tx=0x7fec880059b0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec8effd640 10 client.22835.objecter ms_handle_connect 0x7fec9813d3f0 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec8effd640 10 client.22835.objecter resend_mon_ops 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec8801bde0 con 0x7fec9813d3f0 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec8effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.042+0000 7fec8effd640 10 monclient: dump: 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:47:24.062 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9262 (secure 0 0 0) 0x7fec88005110 con 0x7fec9813d3f0 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 msgr2=0x7fec980a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 0x7fec980a3140 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fec88000c10 tx=0x7fec880059b0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 msgr2=0x7fec980a3140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:47:24.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec8effd640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec9813d3f0 0x7fec980a3140 secure :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fec88000c10 tx=0x7fec880059b0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 30, 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 187504, 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:47:24.064 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:24.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:24.066 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:47:24.067 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:47:24.068 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:47:24.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:47:24.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:47:24.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:47:24.075 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:47:24.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.078 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.080 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:47:24.082 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:47:24.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:47:24.085 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 10 librados: watch_flush enter 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 10 librados: watch_flush exit 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 20 client.22835.objecter shutdown clearing up homeless session... 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 10 client.22835.objecter successfully canceled tick 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec7c038e10 msgr2=0x7fec7c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec7c038e10 0x7fec7c03b2d0 secure :-1 s=READY pgs=3363 cs=0 l=1 rev1=1 crypto rx=0x7fec9813eff0 tx=0x7fec9000b040 comp rx=0 tx=0).stop 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 10 monclient: shutdown 2024-04-19T00:47:24.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec980a5bb0 msgr2=0x7fec981d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 --2- 172.21.15.43:0/457869914 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec980a5bb0 0x7fec981d09b0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fec940b0040 tx=0x7fec940191b0 comp rx=0 tx=0).stop 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec9ed3a640 1 -- 172.21.15.43:0/457869914 reap_dead start 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7fec9ed3a640 1 -- 172.21.15.43:0/457869914 reap_dead start 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 shutdown_connections 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 >> 172.21.15.43:0/457869914 conn(0x7fec980a7cb0 msgr2=0x7fec980a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 shutdown_connections 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 -- 172.21.15.43:0/457869914 wait complete. 2024-04-19T00:47:24.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:47:24.046+0000 7feca079d640 1 librados: shutdown 2024-04-19T00:47:24.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:47:24.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T00:47:26.630 INFO:tasks.workunit.client.0.smithi161.stdout:70045 ms total) 2024-04-19T00:47:26.630 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (148 ms) 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (148 ms total) 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-19T00:47:26.631 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (29 ms) 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (4 ms) 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (33 ms total) 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-19T00:47:26.632 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (48 ms) 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (52 ms total) 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (21 ms) 2024-04-19T00:47:26.633 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (6 ms) 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (27 ms total) 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1055 ms) 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1055 ms total) 2024-04-19T00:47:26.634 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: 2024-04-19T00:47:26.635 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-19T00:47:26.635 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (120257 ms total) 2024-04-19T00:47:26.635 INFO:tasks.workunit.client.0.smithi161.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-19T00:47:29.654 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-04-19T00:47:29.654 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-19T00:47:29.654 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2371 ms) 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (19894 ms) 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6054 ms) 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2989 ms) 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3015 ms) 2024-04-19T00:47:29.655 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2965 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3000 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3024 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2974 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3028 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3033 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2935 ms) 2024-04-19T00:47:29.656 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3033 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3004 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3018 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2975 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3037 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3039 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (7856 ms) 2024-04-19T00:47:29.657 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3058 ms) 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3024 ms) 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2741 ms) 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2309 ms) 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (22580 ms) 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2349 ms) 2024-04-19T00:47:29.658 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-19T00:47:29.659 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3022 ms) 2024-04-19T00:47:29.659 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-19T00:47:56.765 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAi : cursor()=9:89d3ae78:::11:head expected=9:89d3ae78:::11:head 2024-04-19T00:47:56.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:89d3ae78:::11:head -> 11 2024-04-19T00:47:56.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=11 expected=11 2024-04-19T00:47:56.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:02547ec2:::1:head 2024-04-19T00:47:56.766 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:02547ec2:::1:head expected=9:02547ec2:::1:head 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:02547ec2:::1:head -> 1 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=1 expected=1 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:e960b815:::9:head 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:e960b815:::9:head expected=9:e960b815:::9:head 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:e960b815:::9:head -> 9 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=9 expected=9 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:863748b0:::15:head 2024-04-19T00:47:56.767 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:863748b0:::15:head expected=9:863748b0:::15:head 2024-04-19T00:47:56.768 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:863748b0:::15:head -> 15 2024-04-19T00:47:56.768 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=15 expected=15 2024-04-19T00:47:56.768 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:6cac518f:::0:head 2024-04-19T00:47:56.768 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:6cac518f:::0:head expected=9:6cac518f:::0:head 2024-04-19T00:47:56.768 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:6cac518f:::0:head -> 0 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=0 expected=0 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:62a1935d:::14:head 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:62a1935d:::14:head expected=9:62a1935d:::14:head 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:62a1935d:::14:head -> 14 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=14 expected=14 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:5c6b0b28:::7:head 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:5c6b0b28:::7:head expected=9:5c6b0b28:::7:head 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:5c6b0b28:::7:head -> 7 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=7 expected=7 2024-04-19T00:47:56.769 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : seek to 9:566253c9:::13:head 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : cursor()=9:566253c9:::13:head expected=9:566253c9:::13:head 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: > 9:566253c9:::13:head -> 13 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: : entry=13 expected=13 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (345 ms) 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (113232 ms) 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-19T00:47:56.770 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: split 0/5 -> MIN 9:33333333::::head 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: split 1/5 -> 9:33333333::::head 9:66666666::::head 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: split 2/5 -> 9:66666666::::head 9:99999999::::head 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: split 3/5 -> 9:99999999::::head 9:cccccccc::::head 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: split 4/5 -> 9:cccccccc::::head MAX 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (30008 ms) 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] 7 tests from LibRadosList (144049 ms total) 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 2024-04-19T00:47:56.771 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1087 ms) 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset foo1,foo2,foo3 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo1 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo2 2024-04-19T00:47:56.772 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo3 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset foo1,foo4,foo5 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo4 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo5 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo1 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset foo6,foo7 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo7 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: foo6 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-19T00:47:56.773 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns1:foo4 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns1:foo5 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns2:foo7 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns2:foo6 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: ns1:foo1 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: :foo1 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: :foo2 2024-04-19T00:47:56.774 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: :foo3 2024-04-19T00:47:56.775 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (48 ms) 2024-04-19T00:47:56.775 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-19T00:47:56.775 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 1 0 2024-04-19T00:47:56.775 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 10 0 2024-04-19T00:47:56.775 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 13 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 7 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 14 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 0 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 15 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 11 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 5 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 8 0 2024-04-19T00:48:02.594 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 6 0 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 3 0 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 4 0 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 12 0 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 9 0 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 2 0 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (91 ms) 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] 3 tests from LibRadosListEC (1226 ms total) 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-19T00:48:02.595 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2993 ms) 2024-04-19T00:48:02.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] 1 test from LibRadosListNP (2993 ms total) 2024-04-19T00:48:02.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: 2024-04-19T00:48:02.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [----------] Global test environment tear-down 2024-04-19T00:48:02.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [==========] 11 tests from 3 test suites ran. (156258 ms total) 2024-04-19T00:48:02.596 INFO:tasks.workunit.client.0.smithi161.stdout: api_list: [ PASSED ] 11 tests. 2024-04-19T00:48:54.090 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T00:48:54.090 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-19T00:48:54.090 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T00:48:54.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T00:48:54.094 INFO:tasks.ceph.mon.a:Started 2024-04-19T00:48:54.094 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-04-19T00:48:54.095 INFO:tasks.ceph.mon.g:Sent signal 19 2024-04-19T00:48:54.095 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:48:54.195 INFO:tasks.ceph.mon.a.smithi043.stderr:2024-04-19T00:48:54.193+0000 7f430efe3c80 -1 mon.a@-1(???).log v0 log_external_backlog rewinding external_log_to from 123 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:49:01.610 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6807 ms) 2024-04-19T00:49:01.610 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-19T00:49:01.610 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (20422 ms) 2024-04-19T00:49:01.610 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-19T00:49:01.610 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (10839 ms) 2024-04-19T00:49:01.610 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6954 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7041 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6976 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7008 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6978 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7082 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6974 ms) 2024-04-19T00:49:01.611 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6986 ms) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7078 ms) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6976 ms) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6931 ms) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7021 ms) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (10978 ms) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (133051 ms total) 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: 2024-04-19T00:49:01.612 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [----------] Global test environment tear-down 2024-04-19T00:49:01.613 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (215319 ms total) 2024-04-19T00:49:01.613 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-19T00:49:09.095 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-04-19T00:49:09.097 INFO:tasks.ceph.mon.g:Sent signal 18 2024-04-19T00:49:09.097 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:49:09.097 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:49:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.229+0000 7f085f959640 10 monclient: get_monmap_and_config 2024-04-19T00:49:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.229+0000 7f085f959640 10 monclient: build_initial_monmap 2024-04-19T00:49:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.229+0000 7f085f959640 10 monclient: monmap: 2024-04-19T00:49:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.232 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:09.234168+0000 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:09.234168+0000 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:09.233 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:09.234 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 Processor -- start 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 -- start start 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient: init 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient: _add_conns ranks=[5,4,0,7,1,10,6,9,3,8,2] 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0858138a50 0x7f0858138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): picked mon.noname-h con 0x7f0858138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08580a5bb0 0x7f0858139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): picked mon.noname-c con 0x7f08580a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0858137960 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08580a0d40 con 0x7f0858137960 2024-04-19T00:49:09.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f08580a1020 con 0x7f08580a5bb0 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f08581398b0 con 0x7f0858138a50 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085f959640 10 monclient(hunting): authenticate will time out at 1219.361368s 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08580a5bb0 0x7f0858139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085def6640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0858137960 0x7f0858137dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38166/0 (socket says 172.21.15.43:38166) 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085def6640 1 -- 172.21.15.43:0/881025811 learned_addr learned my addr 172.21.15.43:0/881025811 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): get_auth_request con 0x7f08580a5bb0 auth_method 0 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085d6f5640 1 --2- 172.21.15.43:0/881025811 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0858138a50 0x7f0858138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient(hunting): handle_auth_done global_id 22909 payload 274 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f084ffff640 1 -- 172.21.15.43:0/881025811 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1479416703 0 0) 0x7f08580a0d40 con 0x7f0858137960 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 -- 172.21.15.43:0/881025811 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0858138a50 msgr2=0x7f0858138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 --2- 172.21.15.43:0/881025811 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0858138a50 0x7f0858138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 -- 172.21.15.43:0/881025811 >> v1:172.21.15.43:6789/0 conn(0x7f0858137960 legacy=0x7f0858137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient: _finish_hunting 0 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 monclient: found mon.noname-c 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 -- 172.21.15.43:0/881025811 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08580a0a60 con 0x7f08580a5bb0 2024-04-19T00:49:09.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f084ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:49:09.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.233+0000 7f085cef4640 1 --2- 172.21.15.43:0/881025811 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08580a5bb0 0x7f0858139370 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f08480097f0 tx=0x7f084801c340 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d0678713380f2ac0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 1 -- 172.21.15.43:0/881025811 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08480b0070 con 0x7f08580a5bb0 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: dump: 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.245 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: _finish_auth 0 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 1 -- 172.21.15.43:0/881025811 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08480aa810 con 0x7f08580a5bb0 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 5 monclient: authenticate success, global_id 22909 2024-04-19T00:49:09.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 1 -- 172.21.15.43:0/881025811 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08480aa3a0 con 0x7f08580a5bb0 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 4 set_mon_vals no callback set 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084ffff640 10 monclient: dump: 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.247 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: shutdown 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:09.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/881025811 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08580a5bb0 msgr2=0x7f0858139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 --2- 172.21.15.43:0/881025811 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08580a5bb0 0x7f0858139370 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f08480097f0 tx=0x7f084801c340 comp rx=0 tx=0).stop 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/881025811 shutdown_connections 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 --2- 172.21.15.43:0/881025811 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08580a5bb0 0x7f0858139370 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 --2- 172.21.15.43:0/881025811 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0858138a50 0x7f0858138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/881025811 >> 172.21.15.43:0/881025811 conn(0x7f08580a7cb0 msgr2=0x7f08580a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/881025811 shutdown_connections 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/881025811 wait complete. 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: build_initial_monmap 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: monmap: 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:09.249 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 librados: starting msgr at 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 librados: starting objecter 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 Processor -- start 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- start start 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 librados: setting wanted keys 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 librados: calling monclient init 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: init 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: _add_conns ranks=[6,4,0,1,3,5,8,2,7] 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f08580a5bb0 0x7f0858136030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): picked mon.noname-d con 0x7f08580a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 0x7f0858136570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): picked mon.noname-c con 0x7f0858137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0858138a50 0x7f085813c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): picked mon.noname-a con 0x7f0858138a50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0858143bb0 con 0x7f0858138a50 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0858143a30 con 0x7f0858137960 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0858143d30 con 0x7f08580a5bb0 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient(hunting): authenticate will time out at 1219.366370s 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 0x7f0858136570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f08580a5bb0 0x7f0858136030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 0x7f0858136570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:51396/0 (socket says 172.21.15.43:51396) 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 -- 172.21.15.43:0/486925234 learned_addr learned my addr 172.21.15.43:0/486925234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:09.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): get_auth_request con 0x7f0858137960 auth_method 0 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): get_auth_request con 0x7f08580a5bb0 auth_method 0 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0858138a50 0x7f085813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): get_auth_request con 0x7f0858138a50 auth_method 0 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085d6f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085def6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient(hunting): handle_auth_done global_id 22918 payload 824 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 -- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f08580a5bb0 msgr2=0x7f0858136030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f08580a5bb0 0x7f0858136030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 -- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0858138a50 msgr2=0x7f085813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0858138a50 0x7f085813c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient: _finish_hunting 0 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 monclient: found mon.noname-c 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 -- 172.21.15.43:0/486925234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08581402a0 con 0x7f0858137960 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085cef4640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 0x7f0858136570 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0848009710 tx=0x7f0848019140 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 client.?.objecter ms_handle_connect 0x7f0858137960 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08480b0040 con 0x7f0858137960 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: dump: 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.255 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: _finish_auth 0 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08480b9070 con 0x7f0858137960 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 5 monclient: authenticate success, global_id 22918 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: _renew_subs 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f085809fc90 con 0x7f0858137960 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08480044b0 con 0x7f0858137960 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:09.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084dffb640 10 monclient: dump: 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.258 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f084f7fe640 4 set_mon_vals no callback set 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 client.22918.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: _renew_subs 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f085813dfb0 con 0x7f0858137960 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.237+0000 7f085f959640 1 librados: init done 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085f959640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085f959640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085f959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f085813e860 con 0x7f0858137960 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0848004040 con 0x7f0858137960 2024-04-19T00:49:09.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f083c038e10 0x7f083c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(212..212 src has 1..212) ==== 6494+0+0 (secure 0 0 0) 0x7f08480e3190 con 0x7f0858137960 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 10 client.22918.objecter ms_dispatch 0x7f0858000ba0 osd_map(212..212 src has 1..212) 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 3 client.22918.objecter handle_osd_map got epochs [212,212] > 0 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 3 client.22918.objecter handle_osd_map decoding full epoch 212 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 20 client.22918.objecter dump_active .. 0 homeless 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085d6f5640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f083c038e10 0x7f083c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085d6f5640 10 monclient: get_auth_request con 0x7f083c038e10 auth_method 0 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f085d6f5640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f083c038e10 0x7f083c03b2d0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f085400f360 tx=0x7f0854007710 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.241+0000 7f084dffb640 10 client.22918.objecter ms_handle_connect 0x7f083c038e10 2024-04-19T00:49:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.245+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f08480160b0 con 0x7f0858137960 2024-04-19T00:49:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.245+0000 7f084dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:09.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.245+0000 7f084dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.465+0000 7f085f959640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:49:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.465+0000 7f085f959640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:49:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.465+0000 7f085f959640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.465+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f085813fac0 con 0x7f0858137960 2024-04-19T00:49:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f084dffb640 1 -- 172.21.15.43:0/486925234 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f08480aadc0 con 0x7f0858137960 2024-04-19T00:49:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f084dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:49:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f084dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:09.469 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":34,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:49:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 10 librados: watch_flush enter 2024-04-19T00:49:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 10 librados: watch_flush exit 2024-04-19T00:49:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 20 client.22918.objecter shutdown clearing up homeless session... 2024-04-19T00:49:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 10 client.22918.objecter successfully canceled tick 2024-04-19T00:49:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f083c038e10 msgr2=0x7f083c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f083c038e10 0x7f083c03b2d0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f085400f360 tx=0x7f0854007710 comp rx=0 tx=0).stop 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 10 monclient: shutdown 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 msgr2=0x7f0858136570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 0x7f0858136570 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f0848009710 tx=0x7f0848019140 comp rx=0 tx=0).stop 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 shutdown_connections 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f083c038e10 0x7f083c03b2d0 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0858138a50 0x7f085813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0858137960 0x7f0858136570 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 --2- 172.21.15.43:0/486925234 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f08580a5bb0 0x7f0858136030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 >> 172.21.15.43:0/486925234 conn(0x7f08580a7cb0 msgr2=0x7f08580a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 shutdown_connections 2024-04-19T00:49:09.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 -- 172.21.15.43:0/486925234 wait complete. 2024-04-19T00:49:09.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.469+0000 7f085f959640 1 librados: shutdown 2024-04-19T00:49:09.488 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:49:09.488 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:49:09.488 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:49:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient: get_monmap_and_config 2024-04-19T00:49:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient: build_initial_monmap 2024-04-19T00:49:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient: monmap: 2024-04-19T00:49:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:09.617 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:09.619112+0000 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:09.619112+0000 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:09.618 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:09.619 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:09.619 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 Processor -- start 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 -- start start 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient: init 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient: _add_conns ranks=[6,10,2,8,9,1,0,4,3,5,7] 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 0x7f8c98138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): picked mon.noname-d con 0x7f8c98138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:09.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c980a5bb0 0x7f8c98139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): picked mon.noname-f con 0x7f8c980a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c98137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8c980a1020 con 0x7f8c980a5bb0 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8c981398b0 con 0x7f8c98137960 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8c980a0780 con 0x7f8c98138a50 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8ca049d640 10 monclient(hunting): authenticate will time out at 1219.746401s 2024-04-19T00:49:09.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 0x7f8c98138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 0x7f8c98138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40656/0 (socket says 172.21.15.43:40656) 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 1 -- 172.21.15.43:0/2154710175 learned_addr learned my addr 172.21.15.43:0/2154710175 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98137de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:38504/0 (socket says 172.21.15.43:38504) 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): get_auth_request con 0x7f8c98138a50 auth_method 0 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): get_auth_request con 0x7f8c98137960 auth_method 0 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9da38640 1 --2- 172.21.15.43:0/2154710175 >> v2:172.21.15.161:3300/0 conn(0x7f8c980a5bb0 0x7f8c98139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9da38640 10 monclient(hunting): get_auth_request con 0x7f8c980a5bb0 auth_method 0 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9da38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9da38640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9da38640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient(hunting): handle_auth_done global_id 23024 payload 274 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 -- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 msgr2=0x7f8c98138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:09.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 --2- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 0x7f8c98138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 -- 172.21.15.43:0/2154710175 >> v2:172.21.15.161:3300/0 conn(0x7f8c980a5bb0 msgr2=0x7f8c98139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 --2- 172.21.15.43:0/2154710175 >> v2:172.21.15.161:3300/0 conn(0x7f8c980a5bb0 0x7f8c98139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient: _finish_hunting 0 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 monclient: found mon.noname-b 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9ea3a640 1 -- 172.21.15.43:0/2154710175 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c980a0d40 con 0x7f8c98137960 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.617+0000 7f8c9e239640 1 --2- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 0x7f8c98138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9ea3a640 1 --2- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98137de0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009d90 tx=0x7f8c9001c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7405f46895cbdef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 1 -- 172.21.15.43:0/2154710175 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c900b2070 con 0x7f8c98137960 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: dump: 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.634 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: _finish_auth 0 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: _check_auth_tickets 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 5 monclient: authenticate success, global_id 23024 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 1 -- 172.21.15.43:0/2154710175 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c900a5ab0 con 0x7f8c98137960 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 4 set_mon_vals no callback set 2024-04-19T00:49:09.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 1 -- 172.21.15.43:0/2154710175 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c900a5ce0 con 0x7f8c98137960 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9d237640 10 monclient: dump: 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.636 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: shutdown 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 msgr2=0x7f8c98137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98137de0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f8c90009d90 tx=0x7f8c9001c800 comp rx=0 tx=0).stop 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- 172.21.15.43:0/2154710175 shutdown_connections 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98137de0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- 172.21.15.43:0/2154710175 >> v2:172.21.15.161:3300/0 conn(0x7f8c980a5bb0 0x7f8c98139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- 172.21.15.43:0/2154710175 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8c98138a50 0x7f8c98138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- 172.21.15.43:0/2154710175 >> 172.21.15.43:0/2154710175 conn(0x7f8c980a7cb0 msgr2=0x7f8c980a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- 172.21.15.43:0/2154710175 shutdown_connections 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- 172.21.15.43:0/2154710175 wait complete. 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: build_initial_monmap 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: monmap: 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:09.638 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 librados: starting msgr at 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 librados: starting objecter 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 Processor -- start 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- start start 2024-04-19T00:49:09.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 librados: setting wanted keys 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 librados: calling monclient init 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: init 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient: _add_conns ranks=[5,2,3,8,1,0,6,7,4] 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c980a5bb0 0x7f8c981d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): picked mon.noname-h con 0x7f8c980a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98136a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c98137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c98138a50 0x7f8c98136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): picked mon.noname-g con 0x7f8c98138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8c980a2100 con 0x7f8c98137960 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8c980a1f80 con 0x7f8c98138a50 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8c98142810 con 0x7f8c980a5bb0 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8ca049d640 10 monclient(hunting): authenticate will time out at 1219.751756s 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98136a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98136a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:38506/0 (socket says 172.21.15.43:38506) 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 1 -- 172.21.15.43:0/3139058904 learned_addr learned my addr 172.21.15.43:0/3139058904 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 10 monclient(hunting): get_auth_request con 0x7f8c98137960 auth_method 0 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.621+0000 7f8c9da38640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c980a5bb0 0x7f8c981d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): get_auth_request con 0x7f8c980a5bb0 auth_method 0 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c98138a50 0x7f8c98136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 10 monclient(hunting): get_auth_request con 0x7f8c98138a50 auth_method 0 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 10 monclient(hunting): handle_auth_done global_id 23033 payload 824 2024-04-19T00:49:09.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c980a5bb0 msgr2=0x7f8c981d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8c980a5bb0 0x7f8c981d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c98138a50 msgr2=0x7f8c98136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8c98138a50 0x7f8c98136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 10 monclient: _finish_hunting 0 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 monclient: found mon.noname-b 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c9813e730 con 0x7f8c98137960 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9da38640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98136a00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800d800 tx=0x7f8c8800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 client.?.objecter ms_handle_connect 0x7f8c98137960 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c8800bb30 con 0x7f8c98137960 2024-04-19T00:49:09.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: dump: 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.644 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: _finish_auth 0 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 5 monclient: authenticate success, global_id 23033 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8800bcf0 con 0x7f8c98137960 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c880148a0 con 0x7f8c98137960 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:09.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ca36640 4 set_mon_vals no callback set 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: dump: 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.646 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 monclient: _renew_subs 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c9809fc90 con 0x7f8c98137960 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 client.23033.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 monclient: _renew_subs 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c981223a0 con 0x7f8c98137960 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 1 librados: init done 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:09.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 0x7f8c9813ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9806d140 con 0x7f8c9813ca60 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 0x7f8c9813ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8c8801b030 con 0x7f8c98137960 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient: get_auth_request con 0x7f8c9813ca60 auth_method 0 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8c7c038e10 0x7f8c7c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 10 monclient(hunting): handle_auth_done global_id 54126 payload 824 2024-04-19T00:49:09.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c98121800 con 0x7f8c9813ca60 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(212..212 src has 1..212) ==== 6494+0+0 (secure 0 0 0) 0x7f8c88019020 con 0x7f8c98137960 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 client.23033.objecter ms_dispatch 0x7f8c98000ba0 osd_map(212..212 src has 1..212) 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 3 client.23033.objecter handle_osd_map got epochs [212,212] > 0 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 3 client.23033.objecter handle_osd_map decoding full epoch 212 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 20 client.23033.objecter dump_active .. 0 homeless 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 client.23033.objecter ms_handle_connect 0x7f8c9813ca60 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9e239640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 0x7f8c9813ce40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 client.23033.objecter resend_mon_ops 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8c7c038e10 0x7f8c7c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 10 monclient: get_auth_request con 0x7f8c7c038e10 auth_method 0 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c94015ae0 con 0x7f8c9813ca60 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:09.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: dump: 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.650 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8c94015670 con 0x7f8c9813ca60 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 msgr2=0x7f8c9813ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 0x7f8c9813ce40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94011980 comp rx=0 tx=0).stop 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 msgr2=0x7f8c9813ce40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813ca60 0x7f8c9813ce40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c8effd640 10 client.23033.objecter ms_handle_connect 0x7f8c7c038e10 2024-04-19T00:49:09.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.625+0000 7f8c9ea3a640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8c7c038e10 0x7f8c7c03b2d0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f8c900af000 tx=0x7f8c900a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 0x7f8c9809c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8c98121ec0 con 0x7f8c9813d380 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 0x7f8c9809c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient: get_auth_request con 0x7f8c9813d380 auth_method 0 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 10 monclient(hunting): handle_auth_done global_id 54135 payload 824 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 1 -- 172.21.15.43:0/3139058904 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8c98076190 con 0x7f8c9813d380 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9e239640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 0x7f8c9809c330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 client.23033.objecter ms_handle_connect 0x7f8c9813d380 2024-04-19T00:49:09.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 client.23033.objecter resend_mon_ops 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8c94006780 con 0x7f8c9813d380 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 monclient: dump: 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.655 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f8c94006310 con 0x7f8c9813d380 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 msgr2=0x7f8c9809c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 0x7f8c9809c330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94004740 comp rx=0 tx=0).stop 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 msgr2=0x7f8c9809c330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:09.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c8effd640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8c9813d380 0x7f8c9809c330 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c940114c0 tx=0x7f8c94004740 comp rx=0 tx=0).stop 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15451, 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:09.657 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:09.658 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:09.659 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:09.661 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:09.662 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:09.664 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:09.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:09.668 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:09.671 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:09.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.673 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.676 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:09.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:09.678 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:09.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:09.680 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 10 librados: watch_flush enter 2024-04-19T00:49:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 10 librados: watch_flush exit 2024-04-19T00:49:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 20 client.23033.objecter shutdown clearing up homeless session... 2024-04-19T00:49:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 10 client.23033.objecter successfully canceled tick 2024-04-19T00:49:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8c7c038e10 msgr2=0x7f8c7c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8c7c038e10 0x7f8c7c03b2d0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f8c900af000 tx=0x7f8c900a7070 comp rx=0 tx=0).stop 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 10 monclient: shutdown 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 msgr2=0x7f8c98136a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 --2- 172.21.15.43:0/3139058904 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8c98137960 0x7f8c98136a00 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800d800 tx=0x7f8c8800dcc0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9ea3a640 1 -- 172.21.15.43:0/3139058904 reap_dead start 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8c9ea3a640 1 -- 172.21.15.43:0/3139058904 reap_dead start 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 shutdown_connections 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 >> 172.21.15.43:0/3139058904 conn(0x7f8c980a7cb0 msgr2=0x7f8c9806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 shutdown_connections 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 -- 172.21.15.43:0/3139058904 wait complete. 2024-04-19T00:49:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.633+0000 7f8ca049d640 1 librados: shutdown 2024-04-19T00:49:09.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:09.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:49:09.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.801+0000 7f7ece8a7640 10 monclient: get_monmap_and_config 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.801+0000 7f7ece8a7640 10 monclient: build_initial_monmap 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.801+0000 7f7ece8a7640 10 monclient: monmap: 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:09.805822+0000 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:09.805822+0000 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:09.804 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:09.805 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 Processor -- start 2024-04-19T00:49:09.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 -- start start 2024-04-19T00:49:09.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient: init 2024-04-19T00:49:09.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient: _add_conns ranks=[2,6,9,0,4,1,10,8,3,5,7] 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 0x7f7ec8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec8138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): picked mon.noname-d con 0x7f7ec80a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 0x7f7ec8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec8137960 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f7ec80a1020 con 0x7f7ec8137960 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec81398b0 con 0x7f7ec8138a50 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7ec80a0780 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:09.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ece8a7640 10 monclient(hunting): authenticate will time out at 1219.932865s 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 0x7f7ec8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 0x7f7ec8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 0x7f7ec8137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53412/0 (socket says 172.21.15.43:53412) 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 1 -- 172.21.15.43:0/89764709 learned_addr learned my addr 172.21.15.43:0/89764709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 0x7f7ec8138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:38508/0 (socket says 172.21.15.43:38508) 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40672/0 (socket says 172.21.15.43:40672) 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): get_auth_request con 0x7f7ec8137960 auth_method 0 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): get_auth_request con 0x7f7ec80a5bb0 auth_method 0 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): get_auth_request con 0x7f7ec8138a50 auth_method 0 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_done global_id 23042 payload 274 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 -- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 msgr2=0x7f7ec8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 --2- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 -- 172.21.15.43:0/89764709 >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 msgr2=0x7f7ec8137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 --2- 172.21.15.43:0/89764709 >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 0x7f7ec8137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 monclient: found mon.noname-b 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 -- 172.21.15.43:0/89764709 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec80a0d40 con 0x7f7ec8138a50 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ecce44640 1 --2- 172.21.15.43:0/89764709 >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 0x7f7ec8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:09.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:09.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.805+0000 7f7ec7fff640 1 --2- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 0x7f7ec8138e30 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0097c0 tx=0x7f7ebc01c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b92ccca3f3adde8a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 1 -- 172.21.15.43:0/89764709 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ebc0b0070 con 0x7f7ec8138a50 2024-04-19T00:49:09.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: dump: 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.818 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 1 -- 172.21.15.43:0/89764709 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc0aa810 con 0x7f7ec8138a50 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 5 monclient: authenticate success, global_id 23042 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:09.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 1 -- 172.21.15.43:0/89764709 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ebc0a9300 con 0x7f7ec8138a50 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 4 set_mon_vals no callback set 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec6ffd640 10 monclient: dump: 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.820 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: shutdown 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 msgr2=0x7f7ec8138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 0x7f7ec8138e30 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0097c0 tx=0x7f7ebc01c330 comp rx=0 tx=0).stop 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/89764709 shutdown_connections 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/89764709 >> v2:172.21.15.43:3300/0 conn(0x7f7ec8137960 0x7f7ec8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/89764709 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8138a50 0x7f7ec8138e30 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/89764709 >> 172.21.15.43:0/89764709 conn(0x7f7ec80a7cb0 msgr2=0x7f7ec80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/89764709 shutdown_connections 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/89764709 wait complete. 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: build_initial_monmap 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: monmap: 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:09.822 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 librados: starting msgr at 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 librados: starting objecter 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 Processor -- start 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- start start 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 librados: setting wanted keys 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 librados: calling monclient init 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: init 2024-04-19T00:49:09.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: _add_conns ranks=[6,8,1,0,2,3,5,7,4] 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec81d0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): picked mon.noname-d con 0x7f7ec80a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7ec8137960 0x7f7ec8136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): picked mon.noname-e con 0x7f7ec8137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ec8138a50 0x7f7ec813c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): picked mon.noname-f con 0x7f7ec8138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec80a2100 con 0x7f7ec8138a50 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7ec80a1f80 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7ec8142810 con 0x7f7ec8137960 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:09.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient(hunting): authenticate will time out at 1219.938489s 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7ec8137960 0x7f7ec8136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec81d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7ec8137960 0x7f7ec8136950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56914/0 (socket says 172.21.15.43:56914) 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 1 -- 172.21.15.43:0/2812420140 learned_addr learned my addr 172.21.15.43:0/2812420140 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): get_auth_request con 0x7f7ec8137960 auth_method 0 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): get_auth_request con 0x7f7ec80a5bb0 auth_method 0 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient(hunting): handle_auth_done global_id 23064 payload 824 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ecce44640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ec8138a50 0x7f7ec813c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7ec8137960 msgr2=0x7f7ec8136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7ec8137960 0x7f7ec8136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ec8138a50 msgr2=0x7f7ec813c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ec8138a50 0x7f7ec813c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 monclient: found mon.noname-d 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:09.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec81222f0 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec7fff640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec81d0910 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0ab000 tx=0x7f7ebc0190d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ec80a5bb0 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ebc0b0050 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: dump: 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.827 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: _finish_auth 0 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ebc0b1040 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ebc0044b0 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 5 monclient: authenticate success, global_id 23064 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: _renew_subs 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec809fc90 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec4ff9640 10 monclient: dump: 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:09.829 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ec67fc640 4 set_mon_vals no callback set 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 client.23064.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: _renew_subs 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec813e390 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.809+0000 7f7ece8a7640 1 librados: init done 2024-04-19T00:49:09.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ece8a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec806d140 con 0x7f7ec8136e90 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7ebc0be030 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec77fe640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 msgr2=0x7f7ec813cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec77fe640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 msgr2=0x7f7ec813cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e00 0x7f7eb403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(213..213 src has 1..213) ==== 6606+0+0 (secure 0 0 0) 0x7f7ebc0bc020 con 0x7f7ec80a5bb0 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 10 client.23064.objecter ms_dispatch 0x7f7ec8000ba0 osd_map(213..213 src has 1..213) 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 3 client.23064.objecter handle_osd_map got epochs [213,213] > 0 2024-04-19T00:49:09.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 3 client.23064.objecter handle_osd_map decoding full epoch 213 2024-04-19T00:49:09.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 20 client.23064.objecter dump_active .. 0 homeless 2024-04-19T00:49:09.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ecce44640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e00 0x7f7eb403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:09.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ecce44640 10 monclient: get_auth_request con 0x7f7eb4038e00 auth_method 0 2024-04-19T00:49:09.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ecce44640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e00 0x7f7eb403b2c0 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009c90 tx=0x7f7ec000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:09.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:09.813+0000 7f7ec4ff9640 10 client.23064.objecter ms_handle_connect 0x7f7eb4038e00 2024-04-19T00:49:10.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient: get_auth_request con 0x7f7ec8136e90 auth_method 0 2024-04-19T00:49:10.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.017 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_done global_id 23051 payload 824 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ec813ec70 con 0x7f7ec8136e90 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f7ec813ea00 tx=0x7f7eb8013e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 client.23064.objecter ms_handle_connect 0x7f7ec8136e90 2024-04-19T00:49:10.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 client.23064.objecter resend_mon_ops 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7eb8004030 con 0x7f7ec8136e90 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 monclient: dump: 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.019 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7eb8011bb0 con 0x7f7ec8136e90 2024-04-19T00:49:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 msgr2=0x7f7ec813cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f7ec813ea00 tx=0x7f7eb8013e30 comp rx=0 tx=0).stop 2024-04-19T00:49:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 msgr2=0x7f7ec813cc00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.017+0000 7f7ec4ff9640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec8136e90 0x7f7ec813cc00 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ece8a7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:10.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 0x7f7ec809c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ece8a7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec813d550 con 0x7f7ec813d140 2024-04-19T00:49:10.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 0x7f7ec809c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient: get_auth_request con 0x7f7ec813d140 auth_method 0 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 10 monclient(hunting): handle_auth_done global_id 23060 payload 824 2024-04-19T00:49:10.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 1 -- 172.21.15.43:0/2812420140 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ec8076190 con 0x7f7ec813d140 2024-04-19T00:49:10.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec77fe640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 0x7f7ec809c2e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7ec813ea00 tx=0x7f7eb8004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 client.23064.objecter ms_handle_connect 0x7f7ec813d140 2024-04-19T00:49:10.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 client.23064.objecter resend_mon_ops 2024-04-19T00:49:10.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7eb80168d0 con 0x7f7ec813d140 2024-04-19T00:49:10.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 monclient: dump: 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.029 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7f7eb80293f0 con 0x7f7ec813d140 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 msgr2=0x7f7ec809c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 0x7f7ec809c2e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7ec813ea00 tx=0x7f7eb8004740 comp rx=0 tx=0).stop 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 msgr2=0x7f7ec809c2e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.025+0000 7f7ec4ff9640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ec813d140 0x7f7ec809c2e0 secure :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7ec813ea00 tx=0x7f7eb8004740 comp rx=0 tx=0).stop 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 293524, 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:10.031 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.032 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.033 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:10.034 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:10.035 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.036 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.038 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:10.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.044 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.045 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.046 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.047 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.049 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.050 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.051 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.052 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:10.053 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 10 librados: watch_flush enter 2024-04-19T00:49:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 10 librados: watch_flush exit 2024-04-19T00:49:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 20 client.23064.objecter shutdown clearing up homeless session... 2024-04-19T00:49:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 10 client.23064.objecter successfully canceled tick 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e00 msgr2=0x7f7eb403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e00 0x7f7eb403b2c0 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009c90 tx=0x7f7ec000b040 comp rx=0 tx=0).stop 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 10 monclient: shutdown 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 msgr2=0x7f7ec81d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 --2- 172.21.15.43:0/2812420140 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7ec80a5bb0 0x7f7ec81d0910 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc0ab000 tx=0x7f7ebc0190d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ecce44640 1 -- 172.21.15.43:0/2812420140 reap_dead start 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ecce44640 1 -- 172.21.15.43:0/2812420140 reap_dead start 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 shutdown_connections 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 >> 172.21.15.43:0/2812420140 conn(0x7f7ec80a7cb0 msgr2=0x7f7ec806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 shutdown_connections 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 -- 172.21.15.43:0/2812420140 wait complete. 2024-04-19T00:49:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.029+0000 7f7ece8a7640 1 librados: shutdown 2024-04-19T00:49:10.056 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:10.056 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:49:10.056 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:49:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.185+0000 7f86c49b1640 10 monclient: get_monmap_and_config 2024-04-19T00:49:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.185+0000 7f86c49b1640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.185+0000 7f86c49b1640 10 monclient: monmap: 2024-04-19T00:49:10.188 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:10.190371+0000 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:10.190371+0000 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.189 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:10.190 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:10.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 Processor -- start 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 -- start start 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient: init 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient: _add_conns ranks=[0,7,6,10,5,2,9,8,3,4,1] 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86c00a1fc0 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f86c00a1280 0x7f86c00a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): picked mon.noname-i con 0x7f86c00a1280 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f86c013eb30 0x7f86c0140f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): picked mon.noname-d con 0x7f86c013eb30 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:10.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86c00a05e0 con 0x7f86c00a1fc0 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f86c006c1a0 con 0x7f86c013eb30 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f86c00a08c0 con 0x7f86c00a1280 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86c49b1640 10 monclient(hunting): authenticate will time out at 1220.317627s 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f86c013eb30 0x7f86c0140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86be59c640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f86c00a1fc0 0x7f86c00a3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38178/0 (socket says 172.21.15.43:38178) 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86be59c640 1 -- 172.21.15.43:0/3974978392 learned_addr learned my addr 172.21.15.43:0/3974978392 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): get_auth_request con 0x7f86c013eb30 auth_method 0 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bdd9b640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f86c00a1280 0x7f86c00a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient(hunting): handle_auth_done global_id 23073 payload 274 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 1 -- 172.21.15.43:0/3974978392 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f86c00a1280 msgr2=0x7f86c00a4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f86c00a1280 0x7f86c00a4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 1 -- 172.21.15.43:0/3974978392 >> v1:172.21.15.43:6789/0 conn(0x7f86c00a1fc0 legacy=0x7f86c00a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 1 monclient: found mon.noname-d 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:10.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bd59a640 1 -- 172.21.15.43:0/3974978392 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 696471409 0 0) 0x7f86c00a05e0 con 0x7f86c00a1fc0 2024-04-19T00:49:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bed9d640 1 -- 172.21.15.43:0/3974978392 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c00a0020 con 0x7f86c013eb30 2024-04-19T00:49:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bd59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:49:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bdd9b640 10 monclient: get_auth_request con 0x7f86c00a1280 auth_method 0 2024-04-19T00:49:10.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.189+0000 7f86bdd9b640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f86c00a1280 0x7f86c00a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bed9d640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f86c013eb30 0x7f86c0140f20 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f86b80097f0 tx=0x7f86b801c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=843427cce77379c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 1 -- 172.21.15.43:0/3974978392 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86b80b0070 con 0x7f86c013eb30 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: dump: 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.202 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 1 -- 172.21.15.43:0/3974978392 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86b80aa810 con 0x7f86c013eb30 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 5 monclient: authenticate success, global_id 23073 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 1 -- 172.21.15.43:0/3974978392 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86b80a9300 con 0x7f86c013eb30 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bd59a640 10 monclient: dump: 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.204 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 4 set_mon_vals no callback set 2024-04-19T00:49:10.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: shutdown 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- 172.21.15.43:0/3974978392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f86c013eb30 msgr2=0x7f86c0140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f86c013eb30 0x7f86c0140f20 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f86b80097f0 tx=0x7f86b801c3a0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- 172.21.15.43:0/3974978392 shutdown_connections 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f86c013eb30 0x7f86c0140f20 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 --2- 172.21.15.43:0/3974978392 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f86c00a1280 0x7f86c00a4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- 172.21.15.43:0/3974978392 >> 172.21.15.43:0/3974978392 conn(0x7f86c00a85c0 msgr2=0x7f86c00a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- 172.21.15.43:0/3974978392 shutdown_connections 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- 172.21.15.43:0/3974978392 wait complete. 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: monmap: 2024-04-19T00:49:10.206 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.207 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 librados: starting msgr at 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 librados: starting objecter 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 Processor -- start 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- start start 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 librados: setting wanted keys 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 librados: calling monclient init 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: init 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: _add_conns ranks=[0,2,1,3,6,4,8,5,7] 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86c00a1280 0x7f86c013d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): picked mon.noname-a con 0x7f86c00a1280 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f86c00a1fc0 0x7f86c01d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): picked mon.noname-b con 0x7f86c00a1fc0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86c013eb30 0x7f86c01d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): picked mon.noname-f con 0x7f86c013eb30 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f86c00a4590 con 0x7f86c00a1280 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f86c00a4710 con 0x7f86c013eb30 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f86c0143c80 con 0x7f86c00a1fc0 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient(hunting): authenticate will time out at 1220.322896s 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86c00a1280 0x7f86c013d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f86c00a1fc0 0x7f86c01d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86c00a1280 0x7f86c013d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53420/0 (socket says 172.21.15.43:53420) 2024-04-19T00:49:10.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 -- 172.21.15.43:0/2730257580 learned_addr learned my addr 172.21.15.43:0/2730257580 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): get_auth_request con 0x7f86c00a1280 auth_method 0 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): get_auth_request con 0x7f86c00a1fc0 auth_method 0 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bed9d640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86c013eb30 0x7f86c01d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bed9d640 10 monclient(hunting): get_auth_request con 0x7f86c013eb30 auth_method 0 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient(hunting): handle_auth_done global_id 54162 payload 824 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f86c00a1fc0 msgr2=0x7f86c01d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f86c00a1fc0 0x7f86c01d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86c013eb30 msgr2=0x7f86c01d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86c013eb30 0x7f86c01d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 monclient: found mon.noname-a 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c00732f0 con 0x7f86c00a1280 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86be59c640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86c00a1280 0x7f86c013d910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f86b400c450 tx=0x7f86b400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 client.?.objecter ms_handle_connect 0x7f86c00a1280 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86b4018070 con 0x7f86c00a1280 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 monclient: dump: 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.212 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 5 monclient: authenticate success, global_id 54162 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86b40047f0 con 0x7f86c00a1280 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: _renew_subs 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c013d160 con 0x7f86c00a1280 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.193+0000 7f86aeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86b4004e50 con 0x7f86c00a1280 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bcd99640 4 set_mon_vals no callback set 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 10 client.54162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: dump: 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.214 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.215 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 10 monclient: _renew_subs 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c01d1610 con 0x7f86c00a1280 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 1 librados: init done 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 0x7f86c009be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f86c00a1c00 con 0x7f86c009ba70 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f86b40049e0 con 0x7f86c00a1280 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 0x7f86c009be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient: get_auth_request con 0x7f86c009ba70 auth_method 0 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f86a0038fe0 0x7f86a003b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 6606+0+0 (secure 0 0 0) 0x7f86b4048730 con 0x7f86c00a1280 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 client.54162.objecter ms_dispatch 0x7f86c0000ba0 osd_map(213..213 src has 1..213) 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 3 client.54162.objecter handle_osd_map got epochs [213,213] > 0 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_done global_id 22927 payload 824 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 3 client.54162.objecter handle_osd_map decoding full epoch 213 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86c009f430 con 0x7f86c009ba70 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 20 client.54162.objecter dump_active .. 0 homeless 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 client.54162.objecter ms_handle_connect 0x7f86c009ba70 2024-04-19T00:49:10.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 client.54162.objecter resend_mon_ops 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bdd9b640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 0x7f86c009be50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f86c00a8fd0 tx=0x7f86b000c8b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86b000bb50 con 0x7f86c009ba70 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bed9d640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f86a0038fe0 0x7f86a003b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: dump: 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.218 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bed9d640 10 monclient: get_auth_request con 0x7f86a0038fe0 auth_method 0 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f86b000b6e0 con 0x7f86c009ba70 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 msgr2=0x7f86c009be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 0x7f86c009be50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f86c00a8fd0 tx=0x7f86b000c8b0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 msgr2=0x7f86c009be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c009ba70 0x7f86c009be50 secure :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f86c00a8fd0 tx=0x7f86b000c8b0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86aeffd640 10 client.54162.objecter ms_handle_connect 0x7f86a0038fe0 2024-04-19T00:49:10.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.197+0000 7f86bed9d640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f86a0038fe0 0x7f86a003b4a0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f86b80ad000 tx=0x7f86b80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 0x7f86c00714a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f86c006d140 con 0x7f86c013ef10 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 0x7f86c00714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient: get_auth_request con 0x7f86c013ef10 auth_method 0 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 10 monclient(hunting): handle_auth_done global_id 22936 payload 824 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 1 -- 172.21.15.43:0/2730257580 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86c013cd30 con 0x7f86c013ef10 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bdd9b640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 0x7f86c00714a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f86c00a8fd0 tx=0x7f86b0004a50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 client.54162.objecter ms_handle_connect 0x7f86c013ef10 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 client.54162.objecter resend_mon_ops 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f86b001ddc0 con 0x7f86c013ef10 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 monclient: dump: 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.223 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f86b00278e0 con 0x7f86c013ef10 2024-04-19T00:49:10.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 msgr2=0x7f86c00714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 0x7f86c00714a0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f86c00a8fd0 tx=0x7f86b0004a50 comp rx=0 tx=0).stop 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 msgr2=0x7f86c00714a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86aeffd640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f86c013ef10 0x7f86c00714a0 secure :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f86c00a8fd0 tx=0x7f86b0004a50 comp rx=0 tx=0).stop 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:10.225 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 293724, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:49:10.226 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.227 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.228 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:10.229 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.230 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.232 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.237 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:10.239 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:10.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.242 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.248 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:10.248 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 10 librados: watch_flush enter 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 10 librados: watch_flush exit 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 20 client.54162.objecter shutdown clearing up homeless session... 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 10 client.54162.objecter successfully canceled tick 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f86a0038fe0 msgr2=0x7f86a003b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f86a0038fe0 0x7f86a003b4a0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7f86b80ad000 tx=0x7f86b80a6000 comp rx=0 tx=0).stop 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 10 monclient: shutdown 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86c00a1280 msgr2=0x7f86c013d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 --2- 172.21.15.43:0/2730257580 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f86c00a1280 0x7f86c013d910 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f86b400c450 tx=0x7f86b400c910 comp rx=0 tx=0).stop 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bed9d640 1 -- 172.21.15.43:0/2730257580 reap_dead start 2024-04-19T00:49:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86bed9d640 1 -- 172.21.15.43:0/2730257580 reap_dead start 2024-04-19T00:49:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 shutdown_connections 2024-04-19T00:49:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 >> 172.21.15.43:0/2730257580 conn(0x7f86c00a85c0 msgr2=0x7f86c013fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 shutdown_connections 2024-04-19T00:49:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 -- 172.21.15.43:0/2730257580 wait complete. 2024-04-19T00:49:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.205+0000 7f86c49b1640 1 librados: shutdown 2024-04-19T00:49:10.251 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:10.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:49:10.251 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:49:10.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient: get_monmap_and_config 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient: monmap: 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:10.370768+0000 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:10.370768+0000 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.369 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:10.370 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 Processor -- start 2024-04-19T00:49:10.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 -- start start 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient: init 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient: _add_conns ranks=[8,6,10,5,1,3,4,2,9,7,0] 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a80a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): picked mon.noname-e con 0x7f18a80a1fc0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 0x7f18a80a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): picked mon.noname-d con 0x7f18a80a1280 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18a80a43c0 0x7f18a813fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): picked mon.noname-f con 0x7f18a80a43c0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f18a806c1a0 con 0x7f18a80a43c0 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f18a80a08c0 con 0x7f18a80a1280 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f18a80a0300 con 0x7f18a80a1fc0 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18aff9d640 10 monclient(hunting): authenticate will time out at 1220.497926s 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 0x7f18a80a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 0x7f18a80a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40694/0 (socket says 172.21.15.43:40694) 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 1 -- 172.21.15.43:0/627994048 learned_addr learned my addr 172.21.15.43:0/627994048 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): get_auth_request con 0x7f18a80a1280 auth_method 0 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a80a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): get_auth_request con 0x7f18a80a1fc0 auth_method 0 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ae53a640 1 --2- 172.21.15.43:0/627994048 >> v2:172.21.15.161:3300/0 conn(0x7f18a80a43c0 0x7f18a813fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ae53a640 10 monclient(hunting): get_auth_request con 0x7f18a80a43c0 auth_method 0 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ae53a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ae53a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ae53a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient(hunting): handle_auth_done global_id 22742 payload 274 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 -- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 msgr2=0x7f18a80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 0x7f18a80a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 -- 172.21.15.43:0/627994048 >> v2:172.21.15.161:3300/0 conn(0x7f18a80a43c0 msgr2=0x7f18a813fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 --2- 172.21.15.43:0/627994048 >> v2:172.21.15.161:3300/0 conn(0x7f18a80a43c0 0x7f18a813fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 monclient: found mon.noname-e 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18add39640 1 -- 172.21.15.43:0/627994048 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a80a05e0 con 0x7f18a80a1fc0 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ad538640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 0x7f18a80a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:10.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.369+0000 7f18ae53a640 1 --2- 172.21.15.43:0/627994048 >> v2:172.21.15.161:3300/0 conn(0x7f18a80a43c0 0x7f18a813fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18add39640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a80a3940 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f18980097c0 tx=0x7f189801c330 comp rx=0 tx=0).ready entity=mon.8 client_cookie=205f04cdfaed3067 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 1 -- 172.21.15.43:0/627994048 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18980b0070 con 0x7f18a80a1fc0 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: dump: 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.383 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.384 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 1 -- 172.21.15.43:0/627994048 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18980a4e50 con 0x7f18a80a1fc0 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 1 -- 172.21.15.43:0/627994048 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18980aac80 con 0x7f18a80a1fc0 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 5 monclient: authenticate success, global_id 22742 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 4 set_mon_vals no callback set 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18acd37640 10 monclient: dump: 2024-04-19T00:49:10.385 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.386 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: shutdown 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 msgr2=0x7f18a80a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a80a3940 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f18980097c0 tx=0x7f189801c330 comp rx=0 tx=0).stop 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- 172.21.15.43:0/627994048 shutdown_connections 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- 172.21.15.43:0/627994048 >> v2:172.21.15.161:3300/0 conn(0x7f18a80a43c0 0x7f18a813fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a80a1280 0x7f18a80a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- 172.21.15.43:0/627994048 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a80a3940 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- 172.21.15.43:0/627994048 >> 172.21.15.43:0/627994048 conn(0x7f18a80a85c0 msgr2=0x7f18a80a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- 172.21.15.43:0/627994048 shutdown_connections 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- 172.21.15.43:0/627994048 wait complete. 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: monmap: 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.388 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 librados: starting msgr at 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 librados: starting objecter 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 Processor -- start 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- start start 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 librados: setting wanted keys 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 librados: calling monclient init 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: init 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient: _add_conns ranks=[1,8,7,5,0,6,2,4,3] 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18a80a1280 0x7f18a813a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): picked mon.noname-f con 0x7f18a80a1280 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:10.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a8122990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): picked mon.noname-e con 0x7f18a80a1fc0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18a80a43c0 0x7f18a8122ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): picked mon.noname-i con 0x7f18a80a43c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f18a80a19f0 con 0x7f18a80a1280 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f18a80a47d0 con 0x7f18a80a43c0 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f18a80a1b70 con 0x7f18a80a1fc0 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18aff9d640 10 monclient(hunting): authenticate will time out at 1220.503880s 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.373+0000 7f18ad538640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a8122990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a8122990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56936/0 (socket says 172.21.15.43:56936) 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 -- 172.21.15.43:0/3000310946 learned_addr learned my addr 172.21.15.43:0/3000310946 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): get_auth_request con 0x7f18a80a1fc0 auth_method 0 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18a80a1280 0x7f18a813a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ae53a640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18a80a43c0 0x7f18a8122ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient(hunting): handle_auth_done global_id 22751 payload 824 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18a80a43c0 msgr2=0x7f18a8122ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18a80a43c0 0x7f18a8122ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18a80a1280 msgr2=0x7f18a813a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18a80a1280 0x7f18a813a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 monclient: found mon.noname-e 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a8126290 con 0x7f18a80a1fc0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient: get_auth_request con 0x7f18a80a1280 auth_method 0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18a80a1280 0x7f18a813a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ad538640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a8122990 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f189c00ef30 tx=0x7f189c009f90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 client.?.objecter ms_handle_connect 0x7f18a80a1fc0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f189c00bbe0 con 0x7f18a80a1fc0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: dump: 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.393 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f189c019070 con 0x7f18a80a1fc0 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 5 monclient: authenticate success, global_id 22751 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 monclient: _renew_subs 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a809f430 con 0x7f18a80a1fc0 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f189c00bda0 con 0x7f18a80a1fc0 2024-04-19T00:49:10.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 client.22751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: dump: 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.395 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a77fe640 4 set_mon_vals no callback set 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 monclient: _renew_subs 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a8124010 con 0x7f18a80a1fc0 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 1 librados: init done 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 0x7f18a81237f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f18a806d120 con 0x7f18a8123410 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 0x7f18a81237f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient: get_auth_request con 0x7f18a8123410 auth_method 0 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f189c013070 con 0x7f18a80a1fc0 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 10 monclient(hunting): handle_auth_done global_id 23091 payload 824 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18a8123d30 con 0x7f18a8123410 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1894038e10 0x7f189403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(213..213 src has 1..213) ==== 6606+0+0 (secure 0 0 0) 0x7f189c0495b0 con 0x7f18a80a1fc0 2024-04-19T00:49:10.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 client.22751.objecter ms_dispatch 0x7f18a8000ba0 osd_map(213..213 src has 1..213) 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 3 client.22751.objecter handle_osd_map got epochs [213,213] > 0 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 3 client.22751.objecter handle_osd_map decoding full epoch 213 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18add39640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 0x7f18a81237f0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f18980ad000 tx=0x7f18980190d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 20 client.22751.objecter dump_active .. 0 homeless 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 client.22751.objecter ms_handle_connect 0x7f18a8123410 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 client.22751.objecter resend_mon_ops 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18980b9020 con 0x7f18a8123410 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: dump: 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.398 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ae53a640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1894038e10 0x7f189403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ae53a640 10 monclient: get_auth_request con 0x7f1894038e10 auth_method 0 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18980a4e50 con 0x7f18a8123410 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 msgr2=0x7f18a81237f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 0x7f18a81237f0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f18980ad000 tx=0x7f18980190d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 msgr2=0x7f18a81237f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8123410 0x7f18a81237f0 secure :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f18980ad000 tx=0x7f18980190d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18a67fc640 10 client.22751.objecter ms_handle_connect 0x7f1894038e10 2024-04-19T00:49:10.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.377+0000 7f18ae53a640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1894038e10 0x7f189403b2d0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f18a8125220 tx=0x7f18a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 0x7f18a8072440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f18a81247f0 con 0x7f18a8072030 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 0x7f18a8072440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient: get_auth_request con 0x7f18a8072030 auth_method 0 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 10 monclient(hunting): handle_auth_done global_id 23100 payload 824 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 1 -- 172.21.15.43:0/3000310946 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18a8076500 con 0x7f18a8072030 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 client.22751.objecter ms_handle_connect 0x7f18a8072030 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 client.22751.objecter resend_mon_ops 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18add39640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 0x7f18a8072440 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f189801cbd0 tx=0x7f18980a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18980b9040 con 0x7f18a8072030 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 monclient: dump: 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.404 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f1898003bd0 con 0x7f18a8072030 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 msgr2=0x7f18a8072440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 0x7f18a8072440 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f189801cbd0 tx=0x7f18980a8000 comp rx=0 tx=0).stop 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 msgr2=0x7f18a8072440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18a67fc640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18a8072030 0x7f18a8072440 secure :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f189801cbd0 tx=0x7f18980a8000 comp rx=0 tx=0).stop 2024-04-19T00:49:10.405 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 293900, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:10.406 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.409 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.410 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:10.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:10.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.424 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.426 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.427 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:10.428 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:10.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 10 librados: watch_flush enter 2024-04-19T00:49:10.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 10 librados: watch_flush exit 2024-04-19T00:49:10.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 20 client.22751.objecter shutdown clearing up homeless session... 2024-04-19T00:49:10.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 10 client.22751.objecter successfully canceled tick 2024-04-19T00:49:10.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1894038e10 msgr2=0x7f189403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1894038e10 0x7f189403b2d0 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f18a8125220 tx=0x7f18a0008040 comp rx=0 tx=0).stop 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 10 monclient: shutdown 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 msgr2=0x7f18a8122990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 --2- 172.21.15.43:0/3000310946 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18a80a1fc0 0x7f18a8122990 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f189c00ef30 tx=0x7f189c009f90 comp rx=0 tx=0).stop 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18ae53a640 1 -- 172.21.15.43:0/3000310946 reap_dead start 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18ae53a640 1 -- 172.21.15.43:0/3000310946 reap_dead start 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 shutdown_connections 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 >> 172.21.15.43:0/3000310946 conn(0x7f18a80a85c0 msgr2=0x7f18a80a75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 shutdown_connections 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 -- 172.21.15.43:0/3000310946 wait complete. 2024-04-19T00:49:10.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.385+0000 7f18aff9d640 1 librados: shutdown 2024-04-19T00:49:10.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:10.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:49:10.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:49:10.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient: get_monmap_and_config 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient: monmap: 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:10.550796+0000 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:10.550796+0000 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.549 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:10.550 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:10.550 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 Processor -- start 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 -- start start 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient: init 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient: _add_conns ranks=[1,9,2,10,4,7,6,5,3,0,8] 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0b780a4b10 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0b780a4ef0 0x7f0b780a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): picked mon.noname-a con 0x7f0b780a4ef0 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0b7813e170 0x7f0b78140560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): picked mon.noname-b con 0x7f0b7813e170 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:10.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b781398f0 con 0x7f0b780a4b10 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f0b7806e750 con 0x7f0b780a4ef0 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7806f250 con 0x7f0b7813e170 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b7e0c7640 10 monclient(hunting): authenticate will time out at 1220.678008s 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0b780a4ef0 0x7f0b780a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0b7813e170 0x7f0b78140560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0b780a4ef0 0x7f0b780a5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53424/0 (socket says 172.21.15.43:53424) 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 -- 172.21.15.43:0/2142208042 learned_addr learned my addr 172.21.15.43:0/2142208042 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request con 0x7f0b780a4ef0 auth_method 0 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): get_auth_request con 0x7f0b7813e170 auth_method 0 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_done global_id 54171 payload 274 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 -- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0b7813e170 msgr2=0x7f0b78140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0b7813e170 0x7f0b78140560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 -- 172.21.15.43:0/2142208042 >> v1:172.21.15.161:6789/0 conn(0x7f0b780a4b10 legacy=0x7f0b781391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 monclient: found mon.noname-a 2024-04-19T00:49:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.549+0000 7f0b76ffd640 1 -- 172.21.15.43:0/2142208042 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7806f530 con 0x7f0b780a4ef0 2024-04-19T00:49:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4ef0 0x7f0b780a5370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009870 tx=0x7f0b6801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4696336f43f8f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 1 -- 172.21.15.43:0/2142208042 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b680b0070 con 0x7f0b780a4ef0 2024-04-19T00:49:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: dump: 2024-04-19T00:49:10.562 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.563 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 1 -- 172.21.15.43:0/2142208042 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b680aa810 con 0x7f0b780a4ef0 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 5 monclient: authenticate success, global_id 54171 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 4 set_mon_vals no callback set 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 1 -- 172.21.15.43:0/2142208042 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b680a9300 con 0x7f0b780a4ef0 2024-04-19T00:49:10.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b767fc640 10 monclient: dump: 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.565 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: shutdown 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4ef0 msgr2=0x7f0b780a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4ef0 0x7f0b780a5370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009870 tx=0x7f0b6801c420 comp rx=0 tx=0).stop 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/2142208042 shutdown_connections 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0b7813e170 0x7f0b78140560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/2142208042 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4ef0 0x7f0b780a5370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/2142208042 >> 172.21.15.43:0/2142208042 conn(0x7f0b780a98c0 msgr2=0x7f0b780a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/2142208042 shutdown_connections 2024-04-19T00:49:10.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/2142208042 wait complete. 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: monmap: 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.567 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 librados: starting msgr at 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 librados: starting objecter 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 Processor -- start 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- start start 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 librados: setting wanted keys 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 librados: calling monclient init 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: init 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient: _add_conns ranks=[0,3,8,1,4,7,2,5,6] 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4b10 0x7f0b781d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): picked mon.noname-a con 0x7f0b780a4b10 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b780a4ef0 0x7f0b78136bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): picked mon.noname-g con 0x7f0b780a4ef0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813e170 0x7f0b7813bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): picked mon.noname-e con 0x7f0b7813e170 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0b78143650 con 0x7f0b780a4b10 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0b781434d0 con 0x7f0b780a4ef0 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0b781437d0 con 0x7f0b7813e170 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b7e0c7640 10 monclient(hunting): authenticate will time out at 1220.683294s 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4b10 0x7f0b781d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4b10 0x7f0b781d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:53440/0 (socket says 172.21.15.43:53440) 2024-04-19T00:49:10.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 -- 172.21.15.43:0/586242915 learned_addr learned my addr 172.21.15.43:0/586242915 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813e170 0x7f0b7813bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): get_auth_request con 0x7f0b780a4b10 auth_method 0 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): get_auth_request con 0x7f0b7813e170 auth_method 0 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b780a4ef0 0x7f0b78136bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request con 0x7f0b780a4ef0 auth_method 0 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient(hunting): handle_auth_done global_id 54180 payload 824 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813e170 msgr2=0x7f0b7813bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813e170 0x7f0b7813bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b780a4ef0 msgr2=0x7f0b78136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0b780a4ef0 0x7f0b78136bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 monclient: found mon.noname-a 2024-04-19T00:49:10.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.553+0000 7f0b777fe640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b78121560 con 0x7f0b780a4b10 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b777fe640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4b10 0x7f0b781d08c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c530 tx=0x7f0b6c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 client.?.objecter ms_handle_connect 0x7f0b780a4b10 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b6c017070 con 0x7f0b780a4b10 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: dump: 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.572 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b6c003600 con 0x7f0b780a4b10 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b6c003c60 con 0x7f0b780a4b10 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: dump: 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.574 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b75ffb640 4 set_mon_vals no callback set 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 5 monclient: authenticate success, global_id 54180 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 monclient: _renew_subs 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b7806e920 con 0x7f0b780a4b10 2024-04-19T00:49:10.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 client.54180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 monclient: _renew_subs 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7813d6c0 con 0x7f0b780a4b10 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 1 librados: init done 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 0x7f0b7813c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0b7806d400 con 0x7f0b781370f0 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0b6c0037f0 con 0x7f0b780a4b10 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 0x7f0b7813c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient: get_auth_request con 0x7f0b781370f0 auth_method 0 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b64038e10 0x7f0b6403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_done global_id 22769 payload 824 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0b7813df50 con 0x7f0b781370f0 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(213..213 src has 1..213) ==== 6606+0+0 (secure 0 0 0) 0x7f0b6c048730 con 0x7f0b780a4b10 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 client.54180.objecter ms_dispatch 0x7f0b78000ba0 osd_map(213..213 src has 1..213) 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 3 client.54180.objecter handle_osd_map got epochs [213,213] > 0 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 3 client.54180.objecter handle_osd_map decoding full epoch 213 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 20 client.54180.objecter dump_active .. 0 homeless 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 client.54180.objecter ms_handle_connect 0x7f0b781370f0 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 client.54180.objecter resend_mon_ops 2024-04-19T00:49:10.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 0x7f0b7813c160 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f0b780a6cc0 tx=0x7f0b680191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b77fff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b64038e10 0x7f0b6403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b77fff640 10 monclient: get_auth_request con 0x7f0b64038e10 auth_method 0 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b680b9030 con 0x7f0b781370f0 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: dump: 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.578 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0b680b1040 con 0x7f0b781370f0 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 msgr2=0x7f0b7813c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 0x7f0b7813c160 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f0b780a6cc0 tx=0x7f0b680191c0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 msgr2=0x7f0b7813c160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b781370f0 0x7f0b7813c160 secure :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f0b780a6cc0 tx=0x7f0b680191c0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b77fff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b64038e10 0x7f0b6403b2d0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f0b78121a30 tx=0x7f0b7000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.557+0000 7f0b5ffff640 10 client.54180.objecter ms_handle_connect 0x7f0b64038e10 2024-04-19T00:49:10.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:10.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 0x7f0b780a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0b7806d140 con 0x7f0b7813c6a0 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 0x7f0b780a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient: get_auth_request con 0x7f0b7813c6a0 auth_method 0 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 10 monclient(hunting): handle_auth_done global_id 22778 payload 824 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 1 -- 172.21.15.43:0/586242915 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0b7813cab0 con 0x7f0b7813c6a0 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 client.54180.objecter ms_handle_connect 0x7f0b7813c6a0 2024-04-19T00:49:10.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 client.54180.objecter resend_mon_ops 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b76ffd640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 0x7f0b780a0500 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0b6801aad0 tx=0x7f0b680a8000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b680b9040 con 0x7f0b7813c6a0 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 monclient: dump: 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.583 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f0b68003b00 con 0x7f0b7813c6a0 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 msgr2=0x7f0b780a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 0x7f0b780a0500 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0b6801aad0 tx=0x7f0b680a8000 comp rx=0 tx=0).stop 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 msgr2=0x7f0b780a0500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b5ffff640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0b7813c6a0 0x7f0b780a0500 secure :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f0b6801aad0 tx=0x7f0b680a8000 comp rx=0 tx=0).stop 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 294076, 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:10.585 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.586 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.587 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:10.588 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.589 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:10.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:10.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:10.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:10.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.597 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.600 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:10.603 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.605 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:10.607 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:10.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 10 librados: watch_flush enter 2024-04-19T00:49:10.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 10 librados: watch_flush exit 2024-04-19T00:49:10.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 20 client.54180.objecter shutdown clearing up homeless session... 2024-04-19T00:49:10.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 10 client.54180.objecter successfully canceled tick 2024-04-19T00:49:10.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b64038e10 msgr2=0x7f0b6403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b64038e10 0x7f0b6403b2d0 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7f0b78121a30 tx=0x7f0b7000b040 comp rx=0 tx=0).stop 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 10 monclient: shutdown 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4b10 msgr2=0x7f0b781d08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 --2- 172.21.15.43:0/586242915 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0b780a4b10 0x7f0b781d08c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c00c530 tx=0x7f0b6c00c9f0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b77fff640 1 -- 172.21.15.43:0/586242915 reap_dead start 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b77fff640 1 -- 172.21.15.43:0/586242915 reap_dead start 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 shutdown_connections 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 >> 172.21.15.43:0/586242915 conn(0x7f0b780a98c0 msgr2=0x7f0b78072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 shutdown_connections 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 -- 172.21.15.43:0/586242915 wait complete. 2024-04-19T00:49:10.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.565+0000 7f0b7e0c7640 1 librados: shutdown 2024-04-19T00:49:10.611 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:10.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:49:10.611 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:49:10.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.729+0000 7fa3a5061640 10 monclient: get_monmap_and_config 2024-04-19T00:49:10.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.729+0000 7fa3a5061640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.729+0000 7fa3a5061640 10 monclient: monmap: 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:10.733343+0000 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:10.733343+0000 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.732 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.733 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.733 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:10.733 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:10.733 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.729+0000 7fa3a5061640 1 Processor -- start 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.729+0000 7fa3a5061640 1 -- start start 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.729+0000 7fa3a5061640 10 monclient: init 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient: _add_conns ranks=[6,4,7,1,10,9,8,2,3,5,0] 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 0x7fa3a0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): picked mon.noname-d con 0x7fa3a0138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): picked mon.noname-c con 0x7fa3a00a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): picked mon.noname-i con 0x7fa3a0137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:10.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa3a01398b0 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa3a00a0a60 con 0x7fa3a0138a50 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa3a00a0780 con 0x7fa3a0137960 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient(hunting): authenticate will time out at 1220.860341s 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 0x7fa3a0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a0139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 0x7fa3a0138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40714/0 (socket says 172.21.15.43:40714) 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a0139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:51410/0 (socket says 172.21.15.43:51410) 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 -- 172.21.15.43:0/3073955180 learned_addr learned my addr 172.21.15.43:0/3073955180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): get_auth_request con 0x7fa3a0138a50 auth_method 0 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): get_auth_request con 0x7fa3a00a5bb0 auth_method 0 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39f59e640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39f59e640 10 monclient(hunting): get_auth_request con 0x7fa3a0137960 auth_method 0 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_done global_id 23109 payload 274 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 -- 172.21.15.43:0/3073955180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 msgr2=0x7fa3a0137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 -- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 msgr2=0x7fa3a0139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a0139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 monclient: found mon.noname-d 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 -- 172.21.15.43:0/3073955180 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a00a0d40 con 0x7fa3a0138a50 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39e59c640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39ed9d640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 0x7fa3a0138e30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa3900098a0 tx=0x7fa39001c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=3f27b6ca14053a1e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 1 -- 172.21.15.43:0/3073955180 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3900b0070 con 0x7fa3a0138a50 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: dump: 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.747 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 5 monclient: authenticate success, global_id 23109 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 1 -- 172.21.15.43:0/3073955180 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3900aa810 con 0x7fa3a0138a50 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:10.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 4 set_mon_vals no callback set 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 1 -- 172.21.15.43:0/3073955180 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3900a9300 con 0x7fa3a0138a50 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa39dd9b640 10 monclient: dump: 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.749 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 10 monclient: shutdown 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 msgr2=0x7fa3a0138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 0x7fa3a0138e30 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fa3900098a0 tx=0x7fa39001c440 comp rx=0 tx=0).stop 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- 172.21.15.43:0/3073955180 shutdown_connections 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 --2- 172.21.15.43:0/3073955180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3a0138a50 0x7fa3a0138e30 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- 172.21.15.43:0/3073955180 >> 172.21.15.43:0/3073955180 conn(0x7fa3a00ab1b0 msgr2=0x7fa3a00ab5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- 172.21.15.43:0/3073955180 shutdown_connections 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.733+0000 7fa3a5061640 1 -- 172.21.15.43:0/3073955180 wait complete. 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: monmap: 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.751 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 librados: starting msgr at 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 librados: starting objecter 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 Processor -- start 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- start start 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 librados: setting wanted keys 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 librados: calling monclient init 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: init 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: _add_conns ranks=[5,7,1,3,2,0,4,6,8] 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a01d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): picked mon.noname-h con 0x7fa3a00a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0136a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): picked mon.noname-i con 0x7fa3a0137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0138a50 0x7fa3a013c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): picked mon.noname-f con 0x7fa3a0138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a00a2150 con 0x7fa3a0138a50 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa3a00a1fd0 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa3a0142810 con 0x7fa3a0137960 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): authenticate will time out at 1220.865550s 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a01d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a01d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:36526/0 (socket says 172.21.15.43:36526) 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 -- 172.21.15.43:0/2546741265 learned_addr learned my addr 172.21.15.43:0/2546741265 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): get_auth_request con 0x7fa3a00a5bb0 auth_method 0 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0136a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0138a50 0x7fa3a013c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): get_auth_request con 0x7fa3a0138a50 auth_method 0 2024-04-19T00:49:10.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): get_auth_request con 0x7fa3a0137960 auth_method 0 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient(hunting): handle_auth_done global_id 44645 payload 824 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 msgr2=0x7fa3a0136a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3a0137960 0x7fa3a0136a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0138a50 msgr2=0x7fa3a013c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0138a50 0x7fa3a013c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 monclient: found mon.noname-h 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a01223f0 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39ed9d640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a01d09b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3900097c0 tx=0x7fa3900191e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 client.?.objecter ms_handle_connect 0x7fa3a00a5bb0 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3900bb070 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: dump: 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.757 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3900044b0 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3900b0040 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 5 monclient: authenticate success, global_id 44645 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: _renew_subs 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:10.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a009ee60 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa39d59a640 4 set_mon_vals no callback set 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3877fe640 10 monclient: dump: 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.759 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 client.44645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: _renew_subs 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a0121b50 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 librados: init done 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 0x7fa3a013cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.737+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a006d140 con 0x7fa3a0136f90 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 0x7fa3a013cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient: get_auth_request con 0x7fa3a0136f90 auth_method 0 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa3900aacf0 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa38c038e00 0x7fa38c03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(214..214 src has 1..214) ==== 7029+0+0 (secure 0 0 0) 0x7fa3900e2de0 con 0x7fa3a00a5bb0 2024-04-19T00:49:10.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 client.44645.objecter ms_dispatch 0x7fa3a0000ba0 osd_map(214..214 src has 1..214) 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 3 client.44645.objecter handle_osd_map got epochs [214,214] > 0 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 3 client.44645.objecter handle_osd_map decoding full epoch 214 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 20 client.44645.objecter dump_active .. 0 homeless 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 10 monclient(hunting): handle_auth_done global_id 23860 payload 824 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3a0121f40 con 0x7fa3a0136f90 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39f59e640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa38c038e00 0x7fa38c03b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39f59e640 10 monclient: get_auth_request con 0x7fa38c038e00 auth_method 0 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 client.44645.objecter ms_handle_connect 0x7fa3a0136f90 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 client.44645.objecter resend_mon_ops 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39e59c640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 0x7fa3a013cd90 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00a70a0 tx=0x7fa394011e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa394014440 con 0x7fa3a0136f90 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:49:10.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 monclient: dump: 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.763 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 client.44645.objecter ms_handle_connect 0x7fa38c038e00 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa39f59e640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa38c038e00 0x7fa38c03b2c0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00ac190 tx=0x7fa398009820 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa39400fb30 con 0x7fa3a0136f90 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 msgr2=0x7fa3a013cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 0x7fa3a013cd90 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00a70a0 tx=0x7fa394011e30 comp rx=0 tx=0).stop 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 msgr2=0x7fa3a013cd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.741+0000 7fa3877fe640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a0136f90 0x7fa3a013cd90 secure :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00a70a0 tx=0x7fa394011e30 comp rx=0 tx=0).stop 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 0x7fa3a013d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a0121800 con 0x7fa3a013ef40 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 0x7fa3a013d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient: get_auth_request con 0x7fa3a013ef40 auth_method 0 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 10 monclient(hunting): handle_auth_done global_id 23869 payload 824 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 1 -- 172.21.15.43:0/2546741265 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa3a01d1830 con 0x7fa3a013ef40 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 client.44645.objecter ms_handle_connect 0x7fa3a013ef40 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 client.44645.objecter resend_mon_ops 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39e59c640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 0x7fa3a013d2d0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00a70a0 tx=0x7fa3940046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3940148b0 con 0x7fa3a013ef40 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:49:10.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 monclient: dump: 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.768 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fa39401d400 con 0x7fa3a013ef40 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 msgr2=0x7fa3a013d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 0x7fa3a013d2d0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00a70a0 tx=0x7fa3940046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 msgr2=0x7fa3a013d2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3877fe640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3a013ef40 0x7fa3a013d2d0 secure :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00a70a0 tx=0x7fa3940046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 294218, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:10.770 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.771 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.772 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:10.773 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.774 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:10.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.781 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:10.783 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:10.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:10.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:10.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:10.792 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 10 librados: watch_flush enter 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 10 librados: watch_flush exit 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 20 client.44645.objecter shutdown clearing up homeless session... 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 10 client.44645.objecter successfully canceled tick 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa38c038e00 msgr2=0x7fa38c03b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa38c038e00 0x7fa38c03b2c0 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00ac190 tx=0x7fa398009820 comp rx=0 tx=0).stop 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 10 monclient: shutdown 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa3a00a5bb0 msgr2=0x7fa3a01d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 --2- 172.21.15.43:0/2546741265 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa3a00a5bb0 0x7fa3a01d09b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3900097c0 tx=0x7fa3900191e0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39f59e640 1 -- 172.21.15.43:0/2546741265 reap_dead start 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa39f59e640 1 -- 172.21.15.43:0/2546741265 reap_dead start 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 shutdown_connections 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 >> 172.21.15.43:0/2546741265 conn(0x7fa3a00ab1b0 msgr2=0x7fa3a00a60c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 shutdown_connections 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 -- 172.21.15.43:0/2546741265 wait complete. 2024-04-19T00:49:10.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.749+0000 7fa3a5061640 1 librados: shutdown 2024-04-19T00:49:10.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:10.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:49:10.796 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:49:10.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient: get_monmap_and_config 2024-04-19T00:49:10.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient: monmap: 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:10.922621+0000 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:10.922621+0000 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.921 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.922 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.922 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:10.922 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:10.922 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 Processor -- start 2024-04-19T00:49:10.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 -- start start 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient: init 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient: _add_conns ranks=[7,2,4,10,1,0,6,3,5,8,9] 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff9ec138a50 0x7ff9ec138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): picked mon.noname-i con 0x7ff9ec138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): picked mon.noname-b con 0x7ff9ec0a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff9ec137960 0x7ff9ec137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): picked mon.noname-c con 0x7ff9ec137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec0a1020 con 0x7ff9ec0a5bb0 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff9ec1398b0 con 0x7ff9ec137960 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff9ec0a0780 con 0x7ff9ec138a50 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f37ea640 10 monclient(hunting): authenticate will time out at 1221.049673s 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57150/0 (socket says 172.21.15.43:57150) 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 -- 172.21.15.43:0/364646589 learned_addr learned my addr 172.21.15.43:0/364646589 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): get_auth_request con 0x7ff9ec0a5bb0 auth_method 0 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff9ec137960 0x7ff9ec137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): get_auth_request con 0x7ff9ec137960 auth_method 0 2024-04-19T00:49:10.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1586640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff9ec138a50 0x7ff9ec138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1586640 10 monclient(hunting): get_auth_request con 0x7ff9ec138a50 auth_method 0 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1586640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1586640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f1586640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_done global_id 23087 payload 274 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 -- 172.21.15.43:0/364646589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff9ec138a50 msgr2=0x7ff9ec138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff9ec138a50 0x7ff9ec138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 -- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff9ec137960 msgr2=0x7ff9ec137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff9ec137960 0x7ff9ec137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 monclient: found mon.noname-b 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:10.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.921+0000 7ff9f0d85640 1 -- 172.21.15.43:0/364646589 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec0a0d40 con 0x7ff9ec0a5bb0 2024-04-19T00:49:10.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec139370 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098a0 tx=0x7ff9dc01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=431e402de1972118 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 1 -- 172.21.15.43:0/364646589 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9dc0b0070 con 0x7ff9ec0a5bb0 2024-04-19T00:49:10.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: dump: 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.936 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 1 -- 172.21.15.43:0/364646589 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9dc0044a0 con 0x7ff9ec0a5bb0 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 5 monclient: authenticate success, global_id 23087 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 1 -- 172.21.15.43:0/364646589 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9dc003a90 con 0x7ff9ec0a5bb0 2024-04-19T00:49:10.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 4 set_mon_vals no callback set 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e3fff640 10 monclient: dump: 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.938 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: shutdown 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 msgr2=0x7ff9ec139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec139370 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098a0 tx=0x7ff9dc01c410 comp rx=0 tx=0).stop 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/364646589 shutdown_connections 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec139370 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff9ec137960 0x7ff9ec137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/364646589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff9ec138a50 0x7ff9ec138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/364646589 >> 172.21.15.43:0/364646589 conn(0x7ff9ec0a7cb0 msgr2=0x7ff9ec0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/364646589 shutdown_connections 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/364646589 wait complete. 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: build_initial_monmap 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: monmap: 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:10.940 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 librados: starting msgr at 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 librados: starting objecter 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 Processor -- start 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- start start 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 librados: setting wanted keys 2024-04-19T00:49:10.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 librados: calling monclient init 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: init 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: _add_conns ranks=[3,6,8,5,2,4,0,1,7] 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): picked mon.noname-g con 0x7ff9ec0a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff9ec137960 0x7ff9ec13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): picked mon.noname-d con 0x7ff9ec137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9ec138a50 0x7ff9ec13cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): picked mon.noname-e con 0x7ff9ec138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec0a2100 con 0x7ff9ec0a5bb0 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ff9ec0a1f80 con 0x7ff9ec137960 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ff9ec142810 con 0x7ff9ec138a50 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient(hunting): authenticate will time out at 1221.054874s 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff9ec137960 0x7ff9ec13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9ec138a50 0x7ff9ec13cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff9ec137960 0x7ff9ec13c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33778/0 (socket says 172.21.15.43:33778) 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 -- 172.21.15.43:0/1696920865 learned_addr learned my addr 172.21.15.43:0/1696920865 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): get_auth_request con 0x7ff9ec137960 auth_method 0 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): get_auth_request con 0x7ff9ec138a50 auth_method 0 2024-04-19T00:49:10.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1d87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1586640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1586640 10 monclient(hunting): get_auth_request con 0x7ff9ec0a5bb0 auth_method 0 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1586640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1586640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f1586640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient(hunting): handle_auth_done global_id 23118 payload 824 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9ec138a50 msgr2=0x7ff9ec13cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9ec138a50 0x7ff9ec13cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a5bb0 msgr2=0x7ff9ec072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a5bb0 0x7ff9ec072e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient: _finish_hunting 0 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 monclient: found mon.noname-d 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9ec13ea20 con 0x7ff9ec137960 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f0d85640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff9ec137960 0x7ff9ec13c7a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0ae040 tx=0x7ff9dc0191b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 client.?.objecter ms_handle_connect 0x7ff9ec137960 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:10.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9dc0b0040 con 0x7ff9ec137960 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: dump: 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.946 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9dc003e20 con 0x7ff9ec137960 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 5 monclient: authenticate success, global_id 23118 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: _renew_subs 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:49:10.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9ec06fa10 con 0x7ff9ec137960 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9dc0ab8b0 con 0x7ff9ec137960 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e1ffb640 10 monclient: dump: 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 client.23118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: _renew_subs 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9e37fe640 4 set_mon_vals no callback set 2024-04-19T00:49:10.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9ec070290 con 0x7ff9ec137960 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.925+0000 7ff9f37ea640 1 librados: init done 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f37ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 0x7ff9ec0a3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec06d140 con 0x7ff9ec13d220 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff9dc0ab440 con 0x7ff9ec137960 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff9d0038e10 0x7ff9d003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(214..214 src has 1..214) ==== 7029+0+0 (secure 0 0 0) 0x7ff9dc0e2c70 con 0x7ff9ec137960 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 client.23118.objecter ms_dispatch 0x7ff9ec000ba0 osd_map(214..214 src has 1..214) 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 3 client.23118.objecter handle_osd_map got epochs [214,214] > 0 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 3 client.23118.objecter handle_osd_map decoding full epoch 214 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 0x7ff9ec0a3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 20 client.23118.objecter dump_active .. 0 homeless 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient: get_auth_request con 0x7ff9ec13d220 auth_method 0 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1d87640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff9d0038e10 0x7ff9d003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1d87640 10 monclient: get_auth_request con 0x7ff9d0038e10 auth_method 0 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 10 monclient(hunting): handle_auth_done global_id 18777 payload 824 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9ec13d600 con 0x7ff9ec13d220 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 client.23118.objecter ms_handle_connect 0x7ff9d0038e10 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1d87640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff9d0038e10 0x7ff9d003b2d0 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec13de00 tx=0x7ff9e400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 client.23118.objecter ms_handle_connect 0x7ff9ec13d220 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 client.23118.objecter resend_mon_ops 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9f1586640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 0x7ff9ec0a3060 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800f440 tx=0x7ff9e800f900 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9e8019030 con 0x7ff9ec13d220 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 monclient: dump: 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.952 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff9e8014670 con 0x7ff9ec13d220 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 msgr2=0x7ff9ec0a3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 0x7ff9ec0a3060 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800f440 tx=0x7ff9e800f900 comp rx=0 tx=0).stop 2024-04-19T00:49:10.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 msgr2=0x7ff9ec0a3060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.929+0000 7ff9e1ffb640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec13d220 0x7ff9ec0a3060 secure :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800f440 tx=0x7ff9e800f900 comp rx=0 tx=0).stop 2024-04-19T00:49:10.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f37ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:10.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 0x7ff9ec0a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:10.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f37ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:10.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff9ec13e2d0 con 0x7ff9ec0a35a0 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 0x7ff9ec0a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient: get_auth_request con 0x7ff9ec0a35a0 auth_method 0 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 10 monclient(hunting): handle_auth_done global_id 18786 payload 824 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 1 -- 172.21.15.43:0/1696920865 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff9ec076190 con 0x7ff9ec0a35a0 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9f1586640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 0x7ff9ec0a3980 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800f440 tx=0x7ff9e80046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9e1ffb640 10 client.23118.objecter ms_handle_connect 0x7ff9ec0a35a0 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9e1ffb640 10 client.23118.objecter resend_mon_ops 2024-04-19T00:49:10.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9e801f240 con 0x7ff9ec0a35a0 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9e1ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.937+0000 7ff9e1ffb640 10 monclient: dump: 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:10.957 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7ff9e801f240 con 0x7ff9ec0a35a0 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 msgr2=0x7ff9ec0a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 0x7ff9ec0a3980 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800f440 tx=0x7ff9e80046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 msgr2=0x7ff9ec0a3980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:10.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9e1ffb640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff9ec0a35a0 0x7ff9ec0a3980 secure :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800f440 tx=0x7ff9e80046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 294422, 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:10.959 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.960 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.961 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.962 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:10.963 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:10.964 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.966 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.968 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.970 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.971 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.972 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.973 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.974 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.975 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.976 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:10.978 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.980 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:10.981 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:10.982 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:10.982 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:10.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:10.982 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:10.982 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 10 librados: watch_flush enter 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 10 librados: watch_flush exit 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 20 client.23118.objecter shutdown clearing up homeless session... 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 10 client.23118.objecter successfully canceled tick 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff9d0038e10 msgr2=0x7ff9d003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff9d0038e10 0x7ff9d003b2d0 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec13de00 tx=0x7ff9e400b040 comp rx=0 tx=0).stop 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 10 monclient: shutdown 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff9ec137960 msgr2=0x7ff9ec13c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 --2- 172.21.15.43:0/1696920865 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff9ec137960 0x7ff9ec13c7a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0ae040 tx=0x7ff9dc0191b0 comp rx=0 tx=0).stop 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f1d87640 1 -- 172.21.15.43:0/1696920865 reap_dead start 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f1d87640 1 -- 172.21.15.43:0/1696920865 reap_dead start 2024-04-19T00:49:10.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 shutdown_connections 2024-04-19T00:49:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 >> 172.21.15.43:0/1696920865 conn(0x7ff9ec0a7cb0 msgr2=0x7ff9ec0a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 shutdown_connections 2024-04-19T00:49:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 -- 172.21.15.43:0/1696920865 wait complete. 2024-04-19T00:49:10.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:10.941+0000 7ff9f37ea640 1 librados: shutdown 2024-04-19T00:49:10.985 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:10.985 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:49:10.985 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:49:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient: get_monmap_and_config 2024-04-19T00:49:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient: build_initial_monmap 2024-04-19T00:49:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient: monmap: 2024-04-19T00:49:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:11.107474+0000 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:11.107474+0000 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:11.106 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:11.107 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:11.107 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:11.107 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:11.107 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 Processor -- start 2024-04-19T00:49:11.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 -- start start 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient: init 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient: _add_conns ranks=[7,0,9,5,3,1,2,10,6,8,4] 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f42ac138cd0 0x7f42ac1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): picked mon.noname-i con 0x7f42ac138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42ac0a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f42ac1395f0 0x7f42ac135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): picked mon.noname-a con 0x7f42ac1395f0 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42ac0a3dc0 con 0x7f42ac0a5b80 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f42ac0a4410 con 0x7f42ac1395f0 2024-04-19T00:49:11.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f42ac0a40a0 con 0x7f42ac138cd0 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42b20b6640 10 monclient(hunting): authenticate will time out at 1221.234639s 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aaffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f42ac0a5b80 0x7f42ac1358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57092/0 (socket says 172.21.15.43:57092) 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f42ac1395f0 0x7f42ac135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aaffd640 1 -- 172.21.15.43:0/220218739 learned_addr learned my addr 172.21.15.43:0/220218739 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): get_auth_request con 0x7f42ac1395f0 auth_method 0 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42ab7fe640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f42ac138cd0 0x7f42ac1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42ab7fe640 10 monclient(hunting): get_auth_request con 0x7f42ac138cd0 auth_method 0 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aa7fc640 1 -- 172.21.15.43:0/220218739 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284705318 0 0) 0x7f42ac0a3dc0 con 0x7f42ac0a5b80 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aa7fc640 10 monclient(hunting): _negotiate 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aa7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aa7fc640 10 monclient(hunting): my global_id is 54198 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42aa7fc640 1 -- 172.21.15.43:0/220218739 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f428c003530 con 0x7f42ac0a5b80 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient(hunting): handle_auth_done global_id 54189 payload 274 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 1 -- 172.21.15.43:0/220218739 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f42ac138cd0 msgr2=0x7f42ac1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:11.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f42ac138cd0 0x7f42ac1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 1 -- 172.21.15.43:0/220218739 >> v1:172.21.15.43:6789/0 conn(0x7f42ac0a5b80 legacy=0x7f42ac1358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 1 monclient: found mon.noname-a 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42abfff640 1 -- 172.21.15.43:0/220218739 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac0a2ae0 con 0x7f42ac1395f0 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.105+0000 7f42ab7fe640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f42ac138cd0 0x7f42ac1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42abfff640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac1395f0 0x7f42ac135fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42a40097c0 tx=0x7f42a401c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4babc0f6c39639e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 1 -- 172.21.15.43:0/220218739 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f42a40b1070 con 0x7f42ac1395f0 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: dump: 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.114 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.115 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 1 -- 172.21.15.43:0/220218739 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ac136710 con 0x7f42ac1395f0 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 5 monclient: authenticate success, global_id 54189 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 1 -- 172.21.15.43:0/220218739 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f42ac0a20a0 con 0x7f42ac1395f0 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 4 set_mon_vals no callback set 2024-04-19T00:49:11.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42aa7fc640 10 monclient: dump: 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.117 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: shutdown 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- 172.21.15.43:0/220218739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac1395f0 msgr2=0x7f42ac135fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac1395f0 0x7f42ac135fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42a40097c0 tx=0x7f42a401c040 comp rx=0 tx=0).stop 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- 172.21.15.43:0/220218739 shutdown_connections 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac1395f0 0x7f42ac135fc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 --2- 172.21.15.43:0/220218739 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f42ac138cd0 0x7f42ac1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- 172.21.15.43:0/220218739 >> 172.21.15.43:0/220218739 conn(0x7f42ac0ab2f0 msgr2=0x7f42ac0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:11.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- 172.21.15.43:0/220218739 shutdown_connections 2024-04-19T00:49:11.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- 172.21.15.43:0/220218739 wait complete. 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: build_initial_monmap 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: monmap: 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:11.126 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 librados: starting msgr at 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 librados: starting objecter 2024-04-19T00:49:11.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 Processor -- start 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- start start 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 librados: setting wanted keys 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 librados: calling monclient init 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: init 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient: _add_conns ranks=[0,3,1,6,7,2,5,8,4] 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 0x7f42ac131b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): picked mon.noname-a con 0x7f42ac0a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42ac138cd0 0x7f42ac132040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): picked mon.noname-g con 0x7f42ac138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42ac1395f0 0x7f42ac09f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): picked mon.noname-f con 0x7f42ac1395f0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:11.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f42ac144660 con 0x7f42ac0a5b80 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42ac1444e0 con 0x7f42ac1395f0 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f42ac1447e0 con 0x7f42ac138cd0 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42b20b6640 10 monclient(hunting): authenticate will time out at 1221.239653s 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 0x7f42ac131b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 0x7f42ac131b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35462/0 (socket says 172.21.15.43:35462) 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 1 -- 172.21.15.43:0/934222027 learned_addr learned my addr 172.21.15.43:0/934222027 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): get_auth_request con 0x7f42ac0a5b80 auth_method 0 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.109+0000 7f42ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42ac138cd0 0x7f42ac132040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42ac1395f0 0x7f42ac09f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 10 monclient(hunting): get_auth_request con 0x7f42ac1395f0 auth_method 0 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 10 monclient(hunting): handle_auth_done global_id 54207 payload 824 2024-04-19T00:49:11.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42ac138cd0 msgr2=0x7f42ac132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42ac138cd0 0x7f42ac132040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42ac1395f0 msgr2=0x7f42ac09f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42ac1395f0 0x7f42ac09f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 monclient: found mon.noname-a 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42ac0a3590 con 0x7f42ac0a5b80 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient: get_auth_request con 0x7f42ac138cd0 auth_method 0 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f42ac138cd0 0x7f42ac132040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42ab7fe640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 0x7f42ac131b00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4294009fc0 tx=0x7f429400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 client.?.objecter ms_handle_connect 0x7f42ac0a5b80 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f429401a070 con 0x7f42ac0a5b80 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: dump: 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.132 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: _finish_auth 0 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4294003e10 con 0x7f42ac0a5b80 2024-04-19T00:49:11.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 5 monclient: authenticate success, global_id 54207 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 monclient: _renew_subs 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42ac0a3040 con 0x7f42ac0a5b80 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f42940044a0 con 0x7f42ac0a5b80 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: dump: 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.134 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a9ffb640 4 set_mon_vals no callback set 2024-04-19T00:49:11.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 client.54207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 monclient: _renew_subs 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42ac13b4b0 con 0x7f42ac0a5b80 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 1 librados: init done 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4294004640 con 0x7f42ac0a5b80 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 0x7f42ac02c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f42ac06d140 con 0x7f42ac09ffa0 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4290038e10 0x7f429003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 7029+0+0 (secure 0 0 0) 0x7f4294049960 con 0x7f42ac0a5b80 2024-04-19T00:49:11.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 client.54207.objecter ms_dispatch 0x7f42ac000ba0 osd_map(214..214 src has 1..214) 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 3 client.54207.objecter handle_osd_map got epochs [214,214] > 0 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 3 client.54207.objecter handle_osd_map decoding full epoch 214 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 0x7f42ac02c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient: get_auth_request con 0x7f42ac09ffa0 auth_method 0 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 20 client.54207.objecter dump_active .. 0 homeless 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4290038e10 0x7f429003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 10 monclient: get_auth_request con 0x7f4290038e10 auth_method 0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 10 monclient(hunting): handle_auth_done global_id 44654 payload 824 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42ac136710 con 0x7f42ac09ffa0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 client.54207.objecter ms_handle_connect 0x7f42ac09ffa0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 client.54207.objecter resend_mon_ops 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42aaffd640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 0x7f42ac02c450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42ac0a0a00 tx=0x7f429c00db80 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 client.54207.objecter ms_handle_connect 0x7f4290038e10 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42abfff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4290038e10 0x7f429003b2d0 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f42a40ae000 tx=0x7f42a40a6070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f429c005d30 con 0x7f42ac09ffa0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: dump: 2024-04-19T00:49:11.138 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.139 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f429c017070 con 0x7f42ac09ffa0 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 msgr2=0x7f42ac02c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 0x7f42ac02c450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42ac0a0a00 tx=0x7f429c00db80 comp rx=0 tx=0).stop 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 msgr2=0x7f42ac02c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.113+0000 7f42a3fff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac09ffa0 0x7f42ac02c450 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f42ac0a0a00 tx=0x7f429c00db80 comp rx=0 tx=0).stop 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42b20b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42b20b6640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 0x7f42ac02c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42b20b6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f42ac068240 con 0x7f42ac0a0380 2024-04-19T00:49:11.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 0x7f42ac02c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient: get_auth_request con 0x7f42ac0a0380 auth_method 0 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 10 monclient(hunting): handle_auth_done global_id 44663 payload 824 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 1 -- 172.21.15.43:0/934222027 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42ac0693b0 con 0x7f42ac0a0380 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42aaffd640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 0x7f42ac02c990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f429c001230 tx=0x7f429c005800 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42a3fff640 10 client.54207.objecter ms_handle_connect 0x7f42ac0a0380 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42a3fff640 10 client.54207.objecter resend_mon_ops 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f429c01ce10 con 0x7f42ac0a0380 2024-04-19T00:49:11.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42a3fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.121+0000 7f42a3fff640 10 monclient: dump: 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.142 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f429c007280 con 0x7f42ac0a0380 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 msgr2=0x7f42ac02c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 0x7f42ac02c990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f429c001230 tx=0x7f429c005800 comp rx=0 tx=0).stop 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 msgr2=0x7f42ac02c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:11.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42a3fff640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f42ac0a0380 0x7f42ac02c990 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f429c001230 tx=0x7f429c005800 comp rx=0 tx=0).stop 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 150852, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:11.144 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:11.145 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:11.146 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:11.147 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:11.148 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.149 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:11.153 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.157 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.158 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.161 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.162 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.164 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:11.165 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.166 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:11.167 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 10 librados: watch_flush enter 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 10 librados: watch_flush exit 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 20 client.54207.objecter shutdown clearing up homeless session... 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 10 client.54207.objecter successfully canceled tick 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4290038e10 msgr2=0x7f429003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4290038e10 0x7f429003b2d0 secure :-1 s=READY pgs=3413 cs=0 l=1 rev1=1 crypto rx=0x7f42a40ae000 tx=0x7f42a40a6070 comp rx=0 tx=0).stop 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 10 monclient: shutdown 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:11.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 msgr2=0x7f42ac131b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 0x7f42ac131b00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4294009fc0 tx=0x7f429400d560 comp rx=0 tx=0).stop 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42abfff640 1 -- 172.21.15.43:0/934222027 reap_dead start 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 shutdown_connections 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 --2- 172.21.15.43:0/934222027 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f42ac0a5b80 0x7f42ac131b00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 >> 172.21.15.43:0/934222027 conn(0x7f42ac0ab2f0 msgr2=0x7f42ac0a94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 shutdown_connections 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 -- 172.21.15.43:0/934222027 wait complete. 2024-04-19T00:49:11.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.125+0000 7f42b20b6640 1 librados: shutdown 2024-04-19T00:49:11.170 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:11.171 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:49:11.171 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:49:11.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient: get_monmap_and_config 2024-04-19T00:49:11.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient: build_initial_monmap 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient: monmap: 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:11.291395+0000 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:11.291395+0000 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:11.290 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:11.291 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:11.291 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:11.291 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:11.291 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:11.291 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 Processor -- start 2024-04-19T00:49:11.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 -- start start 2024-04-19T00:49:11.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient: init 2024-04-19T00:49:11.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:11.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient: _add_conns ranks=[4,10,0,3,8,5,7,2,9,1,6] 2024-04-19T00:49:11.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 0x7eff481390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): picked mon.noname-c con 0x7eff48138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff480a5b80 0x7eff481395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): picked mon.noname-f con 0x7eff480a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff481358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff480a3dc0 con 0x7eff481358b0 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7eff480a4410 con 0x7eff480a5b80 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7eff48139b30 con 0x7eff48138cd0 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4fbdb640 10 monclient(hunting): authenticate will time out at 1221.418563s 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 0x7eff481390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 0x7eff481390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58678/0 (socket says 172.21.15.43:58678) 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 -- 172.21.15.43:0/2038969709 learned_addr learned my addr 172.21.15.43:0/2038969709 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): get_auth_request con 0x7eff48138cd0 auth_method 0 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4e178640 1 --1- 172.21.15.43:0/2038969709 >> v1:172.21.15.43:6789/0 conn(0x7eff481358b0 0x7eff48135d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57100/0 (socket says 172.21.15.43:57100) 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d176640 1 --2- 172.21.15.43:0/2038969709 >> v2:172.21.15.161:3300/0 conn(0x7eff480a5b80 0x7eff481395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d176640 10 monclient(hunting): get_auth_request con 0x7eff480a5b80 auth_method 0 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d176640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d176640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient(hunting): handle_auth_done global_id 22963 payload 274 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 -- 172.21.15.43:0/2038969709 >> v2:172.21.15.161:3300/0 conn(0x7eff480a5b80 msgr2=0x7eff481395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 --2- 172.21.15.43:0/2038969709 >> v2:172.21.15.161:3300/0 conn(0x7eff480a5b80 0x7eff481395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 -- 172.21.15.43:0/2038969709 >> v1:172.21.15.43:6789/0 conn(0x7eff481358b0 legacy=0x7eff48135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient: _finish_hunting 0 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 monclient: found mon.noname-c 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4c975640 1 -- 172.21.15.43:0/2038969709 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029902962 0 0) 0x7eff480a3dc0 con 0x7eff481358b0 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4d977640 1 -- 172.21.15.43:0/2038969709 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff480a40a0 con 0x7eff48138cd0 2024-04-19T00:49:11.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.289+0000 7eff4c975640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:49:11.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4d977640 1 --2- 172.21.15.43:0/2038969709 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 0x7eff481390b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7eff40009e50 tx=0x7eff4001c890 comp rx=0 tx=0).ready entity=mon.4 client_cookie=daf91708c2cdb4b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 1 -- 172.21.15.43:0/2038969709 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff400090d0 con 0x7eff48138cd0 2024-04-19T00:49:11.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: dump: 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.304 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: _finish_auth 0 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: _check_auth_tickets 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 1 -- 172.21.15.43:0/2038969709 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff400a5ad0 con 0x7eff48138cd0 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 5 monclient: authenticate success, global_id 22963 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 4 set_mon_vals no callback set 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 1 -- 172.21.15.43:0/2038969709 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff400a5c80 con 0x7eff48138cd0 2024-04-19T00:49:11.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4c975640 10 monclient: dump: 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.306 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: shutdown 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 -- 172.21.15.43:0/2038969709 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 msgr2=0x7eff481390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/2038969709 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 0x7eff481390b0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7eff40009e50 tx=0x7eff4001c890 comp rx=0 tx=0).stop 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 -- 172.21.15.43:0/2038969709 shutdown_connections 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/2038969709 >> v2:172.21.15.161:3300/0 conn(0x7eff480a5b80 0x7eff481395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/2038969709 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff48138cd0 0x7eff481390b0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 -- 172.21.15.43:0/2038969709 >> 172.21.15.43:0/2038969709 conn(0x7eff480ab2f0 msgr2=0x7eff480ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:11.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 -- 172.21.15.43:0/2038969709 shutdown_connections 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 -- 172.21.15.43:0/2038969709 wait complete. 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: build_initial_monmap 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: monmap: 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:11.308 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 librados: starting msgr at 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 librados: starting objecter 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 Processor -- start 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 -- start start 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 librados: setting wanted keys 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 librados: calling monclient init 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: init 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:11.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient: _add_conns ranks=[7,6,3,5,1,8,4,2,0] 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff480a5b80 0x7eff481d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.293+0000 7eff4fbdb640 10 monclient(hunting): picked mon.noname-i con 0x7eff480a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff481358b0 0x7eff481d1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): picked mon.noname-d con 0x7eff481358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff48138cd0 0x7eff481322b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): picked mon.noname-g con 0x7eff48138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7eff48143ab0 con 0x7eff48138cd0 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7eff480a20a0 con 0x7eff481358b0 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7eff48143c30 con 0x7eff480a5b80 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:11.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): authenticate will time out at 1221.424048s 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff481358b0 0x7eff481d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff481358b0 0x7eff481d1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33788/0 (socket says 172.21.15.43:33788) 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 -- 172.21.15.43:0/1391659981 learned_addr learned my addr 172.21.15.43:0/1391659981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): get_auth_request con 0x7eff481358b0 auth_method 0 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff480a5b80 0x7eff481d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4e178640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff48138cd0 0x7eff481322b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient(hunting): handle_auth_done global_id 23127 payload 824 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff480a5b80 msgr2=0x7eff481d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff480a5b80 0x7eff481d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff48138cd0 msgr2=0x7eff481322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff48138cd0 0x7eff481322b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient: _finish_hunting 0 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 monclient: found mon.noname-d 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4809bb10 con 0x7eff481358b0 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient: get_auth_request con 0x7eff480a5b80 auth_method 0 2024-04-19T00:49:11.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff480a5b80 0x7eff481d0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4e178640 10 monclient: get_auth_request con 0x7eff48138cd0 auth_method 0 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4e178640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff48138cd0 0x7eff481322b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d176640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff481358b0 0x7eff481d1350 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00d7d0 tx=0x7eff3c00dc90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 client.?.objecter ms_handle_connect 0x7eff481358b0 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff3c00bb30 con 0x7eff481358b0 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: dump: 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.313 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:11.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3c00bcf0 con 0x7eff481358b0 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 5 monclient: authenticate success, global_id 23127 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient: _renew_subs 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff480a3590 con 0x7eff481358b0 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff3c0148a0 con 0x7eff481358b0 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 client.23127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:11.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff47fff640 4 set_mon_vals no callback set 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: dump: 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.316 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient: _renew_subs 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7eff48135630 con 0x7eff481358b0 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 librados: init done 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7eff3c01b030 con 0x7eff481358b0 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 0x7eff481327f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7eff48133c80 con 0x7eff4809c4b0 2024-04-19T00:49:11.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff34038e10 0x7eff3403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(214..214 src has 1..214) ==== 7029+0+0 (secure 0 0 0) 0x7eff3c019020 con 0x7eff481358b0 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 client.23127.objecter ms_dispatch 0x7eff48000ba0 osd_map(214..214 src has 1..214) 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 3 client.23127.objecter handle_osd_map got epochs [214,214] > 0 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 3 client.23127.objecter handle_osd_map decoding full epoch 214 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 20 client.23127.objecter dump_active .. 0 homeless 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 0x7eff481327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient: get_auth_request con 0x7eff4809c4b0 auth_method 0 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4e178640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff34038e10 0x7eff3403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4e178640 10 monclient: get_auth_request con 0x7eff34038e10 auth_method 0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 10 monclient(hunting): handle_auth_done global_id 19321 payload 824 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff480693b0 con 0x7eff4809c4b0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 client.23127.objecter ms_handle_connect 0x7eff4809c4b0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4d977640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 0x7eff481327f0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7eff480a3520 tx=0x7eff40019690 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 client.23127.objecter resend_mon_ops 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 client.23127.objecter ms_handle_connect 0x7eff34038e10 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff4e178640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff34038e10 0x7eff3403b2d0 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7eff38000f70 tx=0x7eff38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff400bb070 con 0x7eff4809c4b0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.297+0000 7eff467fc640 10 monclient: dump: 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.319 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7eff40003fa0 con 0x7eff4809c4b0 2024-04-19T00:49:11.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:11.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 msgr2=0x7eff481327f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 0x7eff481327f0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7eff480a3520 tx=0x7eff40019690 comp rx=0 tx=0).stop 2024-04-19T00:49:11.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 msgr2=0x7eff481327f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:11.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.301+0000 7eff467fc640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff4809c4b0 0x7eff481327f0 secure :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7eff480a3520 tx=0x7eff40019690 comp rx=0 tx=0).stop 2024-04-19T00:49:11.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4fbdb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 0x7eff48133140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4fbdb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7eff481340f0 con 0x7eff48132d30 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 0x7eff48133140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient: get_auth_request con 0x7eff48132d30 auth_method 0 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 10 monclient(hunting): handle_auth_done global_id 19330 payload 824 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 1 -- 172.21.15.43:0/1391659981 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff48133680 con 0x7eff48132d30 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff4d977640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 0x7eff48133140 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7eff4001afa0 tx=0x7eff400a7000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff467fc640 10 client.23127.objecter ms_handle_connect 0x7eff48132d30 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.305+0000 7eff467fc640 10 client.23127.objecter resend_mon_ops 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7eff400a5c30 con 0x7eff48132d30 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 10 monclient: dump: 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.324 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7eff400b7420 con 0x7eff48132d30 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 msgr2=0x7eff48133140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 0x7eff48133140 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7eff4001afa0 tx=0x7eff400a7000 comp rx=0 tx=0).stop 2024-04-19T00:49:11.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 msgr2=0x7eff48133140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff467fc640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff48132d30 0x7eff48133140 secure :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7eff4001afa0 tx=0x7eff400a7000 comp rx=0 tx=0).stop 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 294766, 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:11.326 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:11.327 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:11.329 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:11.330 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:11.333 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:11.333 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:11.333 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:11.333 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:11.334 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:11.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:11.339 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.341 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:11.344 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:11.345 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.347 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:11.348 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.349 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:49:11.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:11.351 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:11.351 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:11.351 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:11.351 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:11.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 10 librados: watch_flush enter 2024-04-19T00:49:11.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 10 librados: watch_flush exit 2024-04-19T00:49:11.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 20 client.23127.objecter shutdown clearing up homeless session... 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 10 client.23127.objecter successfully canceled tick 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff34038e10 msgr2=0x7eff3403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff34038e10 0x7eff3403b2d0 secure :-1 s=READY pgs=3414 cs=0 l=1 rev1=1 crypto rx=0x7eff38000f70 tx=0x7eff38009040 comp rx=0 tx=0).stop 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 10 monclient: shutdown 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff481358b0 msgr2=0x7eff481d1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 --2- 172.21.15.43:0/1391659981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff481358b0 0x7eff481d1350 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00d7d0 tx=0x7eff3c00dc90 comp rx=0 tx=0).stop 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4e178640 1 -- 172.21.15.43:0/1391659981 reap_dead start 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4e178640 1 -- 172.21.15.43:0/1391659981 reap_dead start 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 shutdown_connections 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 >> 172.21.15.43:0/1391659981 conn(0x7eff480ab2f0 msgr2=0x7eff480a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 shutdown_connections 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 -- 172.21.15.43:0/1391659981 wait complete. 2024-04-19T00:49:11.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.309+0000 7eff4fbdb640 1 librados: shutdown 2024-04-19T00:49:11.354 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:11.354 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T00:49:11.354 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient: get_monmap_and_config 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient: build_initial_monmap 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient: monmap: 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:11.472734+0000 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:11.472734+0000 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:11.471 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:11.472 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 Processor -- start 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 -- start start 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient: init 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient: _add_conns ranks=[4,9,7,5,10,8,0,6,1,3,2] 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 0x7ffb84138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb84138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffb840a5bb0 0x7ffb84139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb840a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb84137960 0x7ffb84137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): picked mon.noname-i con 0x7ffb84137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7ffb840a1020 con 0x7ffb840a5bb0 2024-04-19T00:49:11.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ffb841398b0 con 0x7ffb84138a50 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ffb840a0780 con 0x7ffb84137960 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.469+0000 7ffb8accb640 10 monclient(hunting): authenticate will time out at 1221.599855s 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 0x7ffb84138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffb840a5bb0 0x7ffb84139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 0x7ffb84138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58680/0 (socket says 172.21.15.43:58680) 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ffb840a5bb0 0x7ffb84139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35466/0 (socket says 172.21.15.43:35466) 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 1 -- 172.21.15.43:0/278316824 learned_addr learned my addr 172.21.15.43:0/278316824 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): get_auth_request con 0x7ffb84138a50 auth_method 0 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): get_auth_request con 0x7ffb840a5bb0 auth_method 0 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb89268640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb84137960 0x7ffb84137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb89268640 10 monclient(hunting): get_auth_request con 0x7ffb84137960 auth_method 0 2024-04-19T00:49:11.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb89268640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb89268640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb89268640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient(hunting): handle_auth_done global_id 54225 payload 274 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 -- 172.21.15.43:0/278316824 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb84137960 msgr2=0x7ffb84137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb84137960 0x7ffb84137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 -- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 msgr2=0x7ffb84138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 0x7ffb84138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 monclient: found mon.noname-a 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 -- 172.21.15.43:0/278316824 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb840a0d40 con 0x7ffb840a5bb0 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb88a67640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 0x7ffb84138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:11.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb83fff640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb840a5bb0 0x7ffb84139370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009d90 tx=0x7ffb6c01c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72860ca3b33a162 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 1 -- 172.21.15.43:0/278316824 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb6c0b2070 con 0x7ffb840a5bb0 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: dump: 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.481 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 5 monclient: authenticate success, global_id 54225 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 1 -- 172.21.15.43:0/278316824 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb6c0a5ab0 con 0x7ffb840a5bb0 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:11.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 4 set_mon_vals no callback set 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 1 -- 172.21.15.43:0/278316824 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb6c0a5ce0 con 0x7ffb840a5bb0 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb837fe640 10 monclient: dump: 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.483 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 10 monclient: shutdown 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 -- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb840a5bb0 msgr2=0x7ffb84139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb840a5bb0 0x7ffb84139370 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffb6c009d90 tx=0x7ffb6c01c800 comp rx=0 tx=0).stop 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 -- 172.21.15.43:0/278316824 shutdown_connections 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ffb840a5bb0 0x7ffb84139370 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb84137960 0x7ffb84137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 --2- 172.21.15.43:0/278316824 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb84138a50 0x7ffb84138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 -- 172.21.15.43:0/278316824 >> 172.21.15.43:0/278316824 conn(0x7ffb840a7cb0 msgr2=0x7ffb840a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 -- 172.21.15.43:0/278316824 shutdown_connections 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.473+0000 7ffb8accb640 1 -- 172.21.15.43:0/278316824 wait complete. 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: build_initial_monmap 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: monmap: 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:11.485 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 librados: starting msgr at 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 librados: starting objecter 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 Processor -- start 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- start start 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 librados: setting wanted keys 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 librados: calling monclient init 2024-04-19T00:49:11.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: init 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _add_conns ranks=[7,3,2,8,5,1,0,6,4] 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb840a5bb0 0x7ffb84135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): picked mon.noname-i con 0x7ffb840a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb84137960 0x7ffb84135c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): picked mon.noname-g con 0x7ffb84137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 0x7ffb84136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb84138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffb840a2100 con 0x7ffb84138a50 2024-04-19T00:49:11.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ffb840a1f80 con 0x7ffb84137960 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ffb84142810 con 0x7ffb840a5bb0 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient(hunting): authenticate will time out at 1221.605041s 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 0x7ffb84136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 0x7ffb84136180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57154/0 (socket says 172.21.15.43:57154) 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 -- 172.21.15.43:0/1964443169 learned_addr learned my addr 172.21.15.43:0/1964443169 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): get_auth_request con 0x7ffb84138a50 auth_method 0 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb840a5bb0 0x7ffb84135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb83fff640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb84137960 0x7ffb84135c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 10 monclient(hunting): get_auth_request con 0x7ffb840a5bb0 auth_method 0 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb83fff640 10 monclient(hunting): get_auth_request con 0x7ffb84137960 auth_method 0 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb83fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient(hunting): handle_auth_done global_id 23096 payload 824 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 -- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb840a5bb0 msgr2=0x7ffb84135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb840a5bb0 0x7ffb84135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 -- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb84137960 msgr2=0x7ffb84135c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb84137960 0x7ffb84135c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient: _finish_hunting 0 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 monclient: found mon.noname-b 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 -- 172.21.15.43:0/1964443169 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb8409fc90 con 0x7ffb84138a50 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb840a5bb0 0x7ffb84135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb83fff640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb84137960 0x7ffb84135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb89268640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 0x7ffb84136180 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00d830 tx=0x7ffb7c00dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 client.?.objecter ms_handle_connect 0x7ffb84138a50 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:11.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb7c00bad0 con 0x7ffb84138a50 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: dump: 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.495 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: _finish_auth 0 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: _check_auth_tickets 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 5 monclient: authenticate success, global_id 23096 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb7c019070 con 0x7ffb84138a50 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffb7c00bc90 con 0x7ffb84138a50 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _renew_subs 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb840a2ff0 con 0x7ffb84138a50 2024-04-19T00:49:11.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb82ffd640 4 set_mon_vals no callback set 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 client.23096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 monclient: dump: 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:11.497 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _renew_subs 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb8409dd80 con 0x7ffb84138a50 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 librados: init done 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb840008e0 con 0x7ffb84138a50 2024-04-19T00:49:11.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ffb7c00b660 con 0x7ffb84138a50 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffb64038e10 0x7ffb6403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 7029+0+0 (secure 0 0 0) 0x7ffb7c00f040 con 0x7ffb84138a50 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 10 client.23096.objecter ms_dispatch 0x7ffb84000ba0 osd_map(214..214 src has 1..214) 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 3 client.23096.objecter handle_osd_map got epochs [214,214] > 0 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 3 client.23096.objecter handle_osd_map decoding full epoch 214 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb817fa640 20 client.23096.objecter dump_active .. 0 homeless 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffb64038e10 0x7ffb6403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.477+0000 7ffb88a67640 10 monclient: get_auth_request con 0x7ffb64038e10 auth_method 0 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.481+0000 7ffb88a67640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffb64038e10 0x7ffb6403b2d0 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009a80 tx=0x7ffb74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.481+0000 7ffb817fa640 10 client.23096.objecter ms_handle_connect 0x7ffb64038e10 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.485+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffb8409fc90 con 0x7ffb84138a50 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.485+0000 7ffb817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:11.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.485+0000 7ffb817fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:11.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.725+0000 7ffb8accb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T00:49:11.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.725+0000 7ffb8accb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T00:49:11.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.725+0000 7ffb8accb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:11.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.725+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ffb840a2640 con 0x7ffb84138a50 2024-04-19T00:49:11.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb817fa640 1 -- 172.21.15.43:0/1964443169 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ffb7c00be30 con 0x7ffb84138a50 2024-04-19T00:49:11.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T00:49:11.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb817fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=3186400397,config=2586702516,health=2943877026,kv=4196696685,logm=1079764170}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=11,kv=12,logm=65} crc {auth=2094719637,config=2586702516,health=1961110894,kv=4196696685,logm=398772087}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=12,config=2,health=12,kv=12,logm=62} crc {auth=3770130240,config=2586702516,health=2943877026,kv=4196696685,logm=146306103}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.732 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=429587843,config=2586702516,health=2943877026,kv=4196696685,logm=3604224500}) 2024-04-19T00:49:11.733 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.733 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 10 librados: watch_flush enter 2024-04-19T00:49:11.733 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=12,config=2,health=11,kv=12,logm=63} crc {auth=3770130240,config=2586702516,health=57432911,kv=4196696685,logm=3367626713}) 2024-04-19T00:49:11.733 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.733 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.733 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=2013857640,config=2586702516,health=1734290584,kv=8226665,logm=3339679439}) 2024-04-19T00:49:11.734 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.734 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 10 librados: watch_flush exit 2024-04-19T00:49:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 20 client.23096.objecter shutdown clearing up homeless session... 2024-04-19T00:49:11.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 10 client.23096.objecter successfully canceled tick 2024-04-19T00:49:11.734 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=3873444991,config=2586702516,health=2943877026,kv=2161468089,logm=1292147438}) 2024-04-19T00:49:11.735 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.735 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=1114681874,config=2586702516,health=2804724091,kv=4196696685,logm=1115245127}) 2024-04-19T00:49:11.735 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=3907516245,config=2586702516,health=2943877026,kv=4196696685,logm=3787860830}) 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffb64038e10 msgr2=0x7ffb6403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffb64038e10 0x7ffb6403b2d0 secure :-1 s=READY pgs=3415 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009a80 tx=0x7ffb74008040 comp rx=0 tx=0).stop 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 10 monclient: shutdown 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 msgr2=0x7ffb84136180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.729+0000 7ffb8accb640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 0x7ffb84136180 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c00d830 tx=0x7ffb7c00dcf0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 shutdown_connections 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffb64038e10 0x7ffb6403b2d0 unknown :-1 s=CLOSED pgs=3415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb84138a50 0x7ffb84136180 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ffb84137960 0x7ffb84135c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 --2- 172.21.15.43:0/1964443169 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffb840a5bb0 0x7ffb84135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 >> 172.21.15.43:0/1964443169 conn(0x7ffb840a7cb0 msgr2=0x7ffb8406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:11.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 shutdown_connections 2024-04-19T00:49:11.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 -- 172.21.15.43:0/1964443169 wait complete. 2024-04-19T00:49:11.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ffb8accb640 1 librados: shutdown 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3661692311}) 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128256232}) 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.738 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2396715412}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2027483281}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3859337345}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2320070868}) 2024-04-19T00:49:11.739 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.741 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.741 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1433899790}) 2024-04-19T00:49:11.741 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.741 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188067916}) 2024-04-19T00:49:11.741 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.733+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2719639452}) 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1681530098}) 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273083599}) 2024-04-19T00:49:11.743 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2016931947}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=183146590}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=686913116}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3527052848}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.744 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2452286048}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807145496}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.737+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3442820286}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409271504}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266709270}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3268079125}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.745 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=573592546}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3915309712}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=122995300}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=81353118}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420885011}) 2024-04-19T00:49:11.746 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.741+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1959655072}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033641747}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=201215074}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3393102991}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2758276367}) 2024-04-19T00:49:11.749 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1511835118}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=94207331}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3117273211}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143030324}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1884545174}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.750 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844601122}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2424809201}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4097565428}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2905142446}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4274984818}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.751 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4184507129}) 2024-04-19T00:49:11.752 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.752 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.752 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2603614158}) 2024-04-19T00:49:11.752 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.752 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204176827}) 2024-04-19T00:49:11.752 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.745+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1733789869}) 2024-04-19T00:49:11.753 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.753 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.753 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405058425}) 2024-04-19T00:49:11.753 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979713141}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=247908684}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1660338902}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=356210848}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.754 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2995078031}) 2024-04-19T00:49:11.755 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.755 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.755 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2176054091}) 2024-04-19T00:49:11.755 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.755 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=670745795}) 2024-04-19T00:49:11.755 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.749+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1501535724}) 2024-04-19T00:49:11.756 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T00:49:11.762 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.762 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991426493}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=201378574}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1918192615}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3391004269}) 2024-04-19T00:49:11.763 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3157595212}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3379481674}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=443400832}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=389144491}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2657510033}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.764 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821945530}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668312552}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638758841}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1870663213}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3693801146}) 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.765 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3215826290}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2886549267}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308736100}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.753+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1041789396}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582036048}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2134933218}) 2024-04-19T00:49:11.766 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1895983394}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4134297791}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=481469164}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2830697769}) 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.767 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=467550459}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133020628}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.757+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3513861812}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347373529}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355754975}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=268536331}) 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.768 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2705111498}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1134581894}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=781506293}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2620676301}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598711908}) 2024-04-19T00:49:11.769 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1230990061}) 2024-04-19T00:49:11.782 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.782 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.782 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385013432}) 2024-04-19T00:49:11.782 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389914121}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3364641186}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2352015397}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1473043548}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.761+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.783 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1773499734}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=79725833}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=321722700}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2274350162}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799739718}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668065259}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.784 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=926808208}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1144291064}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3463015870}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2414432521}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3313350401}) 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.785 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3018811144}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.765+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2437961113}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269819635}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1649610374}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2726240267}) 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.786 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3470240164}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1368357867}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=342821048}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2743322344}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487315705}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.769+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1834690577}) 2024-04-19T00:49:11.787 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479449793}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801604766}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2412131304}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2415531591}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.788 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=363018256}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1222589013}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=863233025}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549962808}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.773+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=307767068}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667175786}) 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.789 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2774957569}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2245117178}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1500274697}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=808708886}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.790 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=151232220}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=170725861}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574396633}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.777+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1328082430}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383529147}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039513612}) 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2028169178}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=172766541}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=61411579}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1040141116}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=170343424}) 2024-04-19T00:49:11.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=529255198}) 2024-04-19T00:49:11.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=513399242}) 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175861072}) 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1717277491}) 2024-04-19T00:49:11.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1940910653}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.781+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4048271316}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=785124840}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3609555982}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3478061609}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728089082}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2143446246}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075692949}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4139607511}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776395193}) 2024-04-19T00:49:11.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1691872468}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3188427823}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4090220301}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3308339828}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582111493}) 2024-04-19T00:49:11.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.785+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4203116028}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764990841}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036645251}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2204587643}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1503825478}) 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1032382876}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3693045510}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3422587027}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90193595}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.789+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3311722252}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131732670}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070750915}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2032393584}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2595678782}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1330660068}) 2024-04-19T00:49:11.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3199407677}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=904652855}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=242961403}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1939569310}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612420055}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1211145530}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2324025104}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185367937}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2506401739}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1473247929}) 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3594049167}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=731011055}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.793+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4042423240}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1647283050}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627620329}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=335678355}) 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3936827039}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3591369371}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2708092721}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3240430014}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2116026203}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.797+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3914876470}) 2024-04-19T00:49:11.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585766072}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3954393158}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3177167116}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2610109876}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=541393752}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1011985124}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=85477363}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850882572}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.801+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4037129188}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041987542}) 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=659001285}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1588808076}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2115712036}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3156678325}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=81971892}) 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1153927226}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590549152}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.805+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4186473809}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975170332}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865839093}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2179417047}) 2024-04-19T00:49:11.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2313771023}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1719783519}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1237991855}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1007181466}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756279114}) 2024-04-19T00:49:11.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329694739}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243531672}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2139311308}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4175782709}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=985689259}) 2024-04-19T00:49:11.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2974150050}) 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2691834112}) 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1856024072}) 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734020197}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.809+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=62292279}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1592740212}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854321821}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3677891704}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4128275253}) 2024-04-19T00:49:11.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2135761845}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3427861511}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1765812189}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594078068}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1273468167}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101766074}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234935435}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=79513633}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3062102342}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2773287925}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4191922770}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3385735946}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987897294}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.813+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4246572044}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094627196}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749281999}) 2024-04-19T00:49:11.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1755267163}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1204865902}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=847818256}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=858161582}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4134065204}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=514545981}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.817+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4041852502}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311527995}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3329083146}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3474538366}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4039822410}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1870246957}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2467271330}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2478608614}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467385924}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358688274}) 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600137847}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3741606305}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3792635995}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4232807251}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1436793204}) 2024-04-19T00:49:11.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1790496072}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1363415255}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635800936}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.821+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4125579883}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504112884}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64382666}) 2024-04-19T00:49:11.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1881482788}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=889913426}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1522975206}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=856556813}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3766944086}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515381794}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.825+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3141068881}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=41,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11} crc {logm=4022699404,mdsmap=2840952586,mgr=4237663773,mgr_metadata=2750671369,mgrstat=2474479547,mon_config_key=3723730930,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=3622204543}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=42,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=4081035442,mdsmap=2840952586,mgr=3781018680,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3556731808,mon_config_key=4212284759,monmap=2862618235,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=3180140693}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=44,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3842617212,mdsmap=3842495371,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1376198579,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=42,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=8} crc {logm=1292545603,mdsmap=3140514009,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3556731808,mon_config_key=4212284759,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=1465129730,osd_snap=485678012}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=41,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=962573768,mdsmap=2840952586,mgr=2791694888,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3556731808,mon_config_key=3030485707,monmap=2862618235,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=1521239276}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=66554745,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=468131075,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=44,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=3639364330,mdsmap=3842495371,mgr=4016963310,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3556731808,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=1738499245}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=42,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=389286084,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=376853755,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2868658283}) 2024-04-19T00:49:11.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=44,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=7} crc {logm=3639364330,mdsmap=3140514009,mgr=4015822221,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3556731808,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=1465129730,osd_snap=2721938684}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=10588912,osdmap=3748980757}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=3781650831}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1545651343,osdmap=1322538995}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=3696730160,osdmap=1291215724}) 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=10588912,osdmap=2638237205}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=2678732019,osdmap=1389184576}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=10588912,osdmap=2015694090}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=1749533659,osdmap=1519704096}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.829+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=2221561027,osdmap=4038977151}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3188488470}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1155114314}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=865045448}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=4039440697}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4099940460}) 2024-04-19T00:49:11.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1236535552}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3265423412}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4086788235}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.833+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1119429188}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1008116471}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4205083832}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3475419582}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3019080127}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1772122014}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=211301985}) 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1197428426}) 2024-04-19T00:49:11.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=604887648}) 2024-04-19T00:49:11.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.837+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3616852255}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=42} crc {osdmap=2811386734}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=41} crc {osdmap=4199111741}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=38} crc {osdmap=2304129794}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=41} crc {osdmap=1603980606}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=41} crc {osdmap=173885932}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=40} crc {osdmap=2648006344}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=41} crc {osdmap=1156902977}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=1874981211}) 2024-04-19T00:49:11.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:49:11.853+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=38} crc {osdmap=1120450461}) 2024-04-19T00:49:12.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:49:12.757 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient: get_monmap_and_config 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient: build_initial_monmap 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient: monmap: 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:12.907771+0000 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:12.907771+0000 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:12.906 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:12.907 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 Processor -- start 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 -- start start 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient: init 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient: _add_conns ranks=[0,9,10,5,1,4,2,8,3,7,6] 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc094138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc0940a5b80 0x7fc09413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): picked mon.noname-a con 0x7fc0940a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc0941397c0 0x7fc0941407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:12.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): picked mon.noname-f con 0x7fc0941397c0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0940a3dc0 con 0x7fc094138cd0 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fc0940a4410 con 0x7fc0940a5b80 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc0940a40a0 con 0x7fc0941397c0 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc09a7d1640 10 monclient(hunting): authenticate will time out at 1223.035638s 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc08bfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc0940a5b80 0x7fc09413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc093fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fc094138cd0 0x7fc0941390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57108/0 (socket says 172.21.15.43:57108) 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc093fff640 1 -- 172.21.15.43:0/2686716811 learned_addr learned my addr 172.21.15.43:0/2686716811 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc08bfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc0940a5b80 0x7fc09413e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35482/0 (socket says 172.21.15.43:35482) 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc08bfff640 10 monclient(hunting): get_auth_request con 0x7fc0940a5b80 auth_method 0 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc08bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:12.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc08bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.905+0000 7fc08bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc098d6e640 1 --2- 172.21.15.43:0/2686716811 >> v2:172.21.15.161:3300/0 conn(0x7fc0941397c0 0x7fc0941407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc098d6e640 10 monclient(hunting): get_auth_request con 0x7fc0941397c0 auth_method 0 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc098d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc098d6e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc098d6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 10 monclient(hunting): handle_auth_done global_id 54234 payload 274 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 1 -- 172.21.15.43:0/2686716811 >> v2:172.21.15.161:3300/0 conn(0x7fc0941397c0 msgr2=0x7fc0941407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 1 --2- 172.21.15.43:0/2686716811 >> v2:172.21.15.161:3300/0 conn(0x7fc0941397c0 0x7fc0941407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:12.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 1 -- 172.21.15.43:0/2686716811 >> v1:172.21.15.43:6789/0 conn(0x7fc094138cd0 legacy=0x7fc0941390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 1 monclient: found mon.noname-a 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 1 -- 172.21.15.43:0/2686716811 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 443782444 0 0) 0x7fc0940a3dc0 con 0x7fc094138cd0 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 1 -- 172.21.15.43:0/2686716811 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940a2ae0 con 0x7fc0940a5b80 2024-04-19T00:49:12.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc08bfff640 1 --2- 172.21.15.43:0/2686716811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0940a5b80 0x7fc09413e270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0940a79a0 tx=0x7fc07c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f39a220e4a11867 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 1 -- 172.21.15.43:0/2686716811 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc07c0b0070 con 0x7fc0940a5b80 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: dump: 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:12.919 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:12.920 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 5 monclient: authenticate success, global_id 54234 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 1 -- 172.21.15.43:0/2686716811 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc07c0aa810 con 0x7fc0940a5b80 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 1 -- 172.21.15.43:0/2686716811 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc07c0a9300 con 0x7fc0940a5b80 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 4 set_mon_vals no callback set 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:12.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc0937fe640 10 monclient: dump: 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:12.922 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 10 monclient: shutdown 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 -- 172.21.15.43:0/2686716811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0940a5b80 msgr2=0x7fc09413e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/2686716811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0940a5b80 0x7fc09413e270 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc0940a79a0 tx=0x7fc07c01c340 comp rx=0 tx=0).stop 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 -- 172.21.15.43:0/2686716811 shutdown_connections 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/2686716811 >> v2:172.21.15.161:3300/0 conn(0x7fc0941397c0 0x7fc0941407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/2686716811 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc0940a5b80 0x7fc09413e270 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 -- 172.21.15.43:0/2686716811 >> 172.21.15.43:0/2686716811 conn(0x7fc0940ab2f0 msgr2=0x7fc0940ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 -- 172.21.15.43:0/2686716811 shutdown_connections 2024-04-19T00:49:12.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 1 -- 172.21.15.43:0/2686716811 wait complete. 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 10 monclient: build_initial_monmap 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.909+0000 7fc09a7d1640 10 monclient: monmap: 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:12.924 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 librados: starting msgr at 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 librados: starting objecter 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 Processor -- start 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- start start 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 librados: setting wanted keys 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 librados: calling monclient init 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: init 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _add_conns ranks=[4,3,6,7,1,2,8,0,5] 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 0x7fc0941358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): picked mon.noname-c con 0x7fc0940a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:12.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc094138cd0 0x7fc094135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): picked mon.noname-g con 0x7fc094138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc0941397c0 0x7fc094136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): picked mon.noname-d con 0x7fc0941397c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc094143730 con 0x7fc094138cd0 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc0940a20a0 con 0x7fc0940a5b80 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fc0941438b0 con 0x7fc0941397c0 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient(hunting): authenticate will time out at 1223.040803s 2024-04-19T00:49:12.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 0x7fc0941358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 0x7fc0941358b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58690/0 (socket says 172.21.15.43:58690) 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc0941397c0 0x7fc094136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 -- 172.21.15.43:0/244372096 learned_addr learned my addr 172.21.15.43:0/244372096 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): get_auth_request con 0x7fc0940a5b80 auth_method 0 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): get_auth_request con 0x7fc0941397c0 auth_method 0 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:12.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc094138cd0 0x7fc094135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 10 monclient(hunting): get_auth_request con 0x7fc094138cd0 auth_method 0 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient(hunting): handle_auth_done global_id 22981 payload 824 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 -- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc0941397c0 msgr2=0x7fc094136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc0941397c0 0x7fc094136330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 -- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc094138cd0 msgr2=0x7fc094135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:12.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc094138cd0 0x7fc094135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 monclient: found mon.noname-c 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 -- 172.21.15.43:0/244372096 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0940a3040 con 0x7fc0940a5b80 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc098d6e640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc0941397c0 0x7fc094136330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc093fff640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 0x7fc0941358b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c480 tx=0x7fc08400c940 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 client.?.objecter ms_handle_connect 0x7fc0940a5b80 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc084018070 con 0x7fc0940a5b80 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: dump: 2024-04-19T00:49:12.929 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:12.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: _finish_auth 0 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: _check_auth_tickets 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0840047f0 con 0x7fc0940a5b80 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc084004e50 con 0x7fc0940a5b80 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 5 monclient: authenticate success, global_id 22981 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _renew_subs 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:12.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc09409fea0 con 0x7fc0940a5b80 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 monclient: dump: 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:12.932 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc092ffd640 4 set_mon_vals no callback set 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 client.22981.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _renew_subs 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc09409ecd0 con 0x7fc0940a5b80 2024-04-19T00:49:12.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 librados: init done 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc0840049e0 con 0x7fc0940a5b80 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0940a1a30 con 0x7fc0940a5b80 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc074038fb0 0x7fc07403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(216..216 src has 1..216) ==== 6213+0+0 (secure 0 0 0) 0x7fc084048750 con 0x7fc0940a5b80 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 10 client.22981.objecter ms_dispatch 0x7fc094000ba0 osd_map(216..216 src has 1..216) 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 3 client.22981.objecter handle_osd_map got epochs [216,216] > 0 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 3 client.22981.objecter handle_osd_map decoding full epoch 216 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc0917fa640 20 client.22981.objecter dump_active .. 0 homeless 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc074038fb0 0x7fc07403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.913+0000 7fc08bfff640 10 monclient: get_auth_request con 0x7fc074038fb0 auth_method 0 2024-04-19T00:49:12.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.917+0000 7fc08bfff640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc074038fb0 0x7fc07403b470 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0ad000 tx=0x7fc07c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:12.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.917+0000 7fc0917fa640 10 client.22981.objecter ms_handle_connect 0x7fc074038fb0 2024-04-19T00:49:12.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.917+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc0940a3040 con 0x7fc0940a5b80 2024-04-19T00:49:12.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.917+0000 7fc0917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:12.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:12.917+0000 7fc0917fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:13.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc09a7d1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:49:13.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc09a7d1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:49:13.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc09a7d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:13.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc094068240 con 0x7fc0940a5b80 2024-04-19T00:49:13.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc0917fa640 1 -- 172.21.15.43:0/244372096 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fc08401a960 con 0x7fc0940a5b80 2024-04-19T00:49:13.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc0917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:49:13.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.145+0000 7fc0917fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:13.147 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":34,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:49:13.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 10 librados: watch_flush enter 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 10 librados: watch_flush exit 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 20 client.22981.objecter shutdown clearing up homeless session... 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 10 client.22981.objecter successfully canceled tick 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc074038fb0 msgr2=0x7fc07403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc074038fb0 0x7fc07403b470 secure :-1 s=READY pgs=3416 cs=0 l=1 rev1=1 crypto rx=0x7fc07c0ad000 tx=0x7fc07c0a6000 comp rx=0 tx=0).stop 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 10 monclient: shutdown 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 msgr2=0x7fc0941358b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 0x7fc0941358b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fc08400c480 tx=0x7fc08400c940 comp rx=0 tx=0).stop 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 shutdown_connections 2024-04-19T00:49:13.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc074038fb0 0x7fc07403b470 unknown :-1 s=CLOSED pgs=3416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc0941397c0 0x7fc094136330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc094138cd0 0x7fc094135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 --2- 172.21.15.43:0/244372096 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc0940a5b80 0x7fc0941358b0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 >> 172.21.15.43:0/244372096 conn(0x7fc0940ab2f0 msgr2=0x7fc094140580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 shutdown_connections 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 -- 172.21.15.43:0/244372096 wait complete. 2024-04-19T00:49:13.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.149+0000 7fc09a7d1640 1 librados: shutdown 2024-04-19T00:49:13.165 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:49:13.166 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:49:13.166 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:49:13.166 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient: get_monmap_and_config 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient: monmap: 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:13.286942+0000 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:13.286942+0000 2024-04-19T00:49:13.285 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:13.286 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 Processor -- start 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 -- start start 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient: init 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient: _add_conns ranks=[2,3,4,1,9,10,5,7,8,6,0] 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 0x7f9b341390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): picked mon.noname-b con 0x7f9b34138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): picked mon.noname-g con 0x7f9b340a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9b341358b0 0x7f9b34135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): picked mon.noname-c con 0x7f9b341358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9b340a4410 con 0x7f9b34138cd0 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9b34139b30 con 0x7f9b340a5b80 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9b340a3dc0 con 0x7f9b341358b0 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b390be640 10 monclient(hunting): authenticate will time out at 1223.414554s 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 0x7f9b341390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9b341358b0 0x7f9b34135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 0x7f9b341390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57158/0 (socket says 172.21.15.43:57158) 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 1 -- 172.21.15.43:0/948241369 learned_addr learned my addr 172.21.15.43:0/948241369 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request con 0x7f9b34138cd0 auth_method 0 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): get_auth_request con 0x7f9b341358b0 auth_method 0 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3259c640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient(hunting): handle_auth_done global_id 22990 payload 274 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 -- 172.21.15.43:0/948241369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 msgr2=0x7f9b341395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 -- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 msgr2=0x7f9b341390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 0x7f9b341390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 monclient: found mon.noname-c 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3359e640 1 -- 172.21.15.43:0/948241369 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b340a40a0 con 0x7f9b341358b0 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3259c640 10 monclient: get_auth_request con 0x7f9b340a5b80 auth_method 0 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b3259c640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:13.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.285+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 0x7f9b341390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:13.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3359e640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9b341358b0 0x7f9b34135d30 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009dc0 tx=0x7f9b2c01c7f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=fb007e7fcee3e49a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 1 -- 172.21.15.43:0/948241369 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b2c0a6960 con 0x7f9b341358b0 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: dump: 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.300 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 1 -- 172.21.15.43:0/948241369 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b2c0a6b20 con 0x7f9b341358b0 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 5 monclient: authenticate success, global_id 22990 2024-04-19T00:49:13.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 1 -- 172.21.15.43:0/948241369 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b2c0a6d10 con 0x7f9b341358b0 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 4 set_mon_vals no callback set 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b31d9b640 10 monclient: dump: 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.302 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: shutdown 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9b341358b0 msgr2=0x7f9b34135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9b341358b0 0x7f9b34135d30 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009dc0 tx=0x7f9b2c01c7f0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- 172.21.15.43:0/948241369 shutdown_connections 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9b341358b0 0x7f9b34135d30 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- 172.21.15.43:0/948241369 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b34138cd0 0x7f9b341390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- 172.21.15.43:0/948241369 >> 172.21.15.43:0/948241369 conn(0x7f9b340ab2f0 msgr2=0x7f9b340ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- 172.21.15.43:0/948241369 shutdown_connections 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- 172.21.15.43:0/948241369 wait complete. 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: monmap: 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:13.304 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 librados: starting msgr at 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 librados: starting objecter 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 Processor -- start 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- start start 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 librados: setting wanted keys 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 librados: calling monclient init 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: init 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient: _add_conns ranks=[3,2,7,6,1,0,5,4,8] 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): picked mon.noname-g con 0x7f9b340a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b341358b0 0x7f9b3413eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): picked mon.noname-b con 0x7f9b341358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:13.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9b34138cd0 0x7f9b3413f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): picked mon.noname-i con 0x7f9b34138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9b340a2010 con 0x7f9b341358b0 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9b340a1e90 con 0x7f9b340a5b80 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9b34142aa0 con 0x7f9b34138cd0 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b390be640 10 monclient(hunting): authenticate will time out at 1223.419766s 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b341358b0 0x7f9b3413eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b341358b0 0x7f9b3413eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57170/0 (socket says 172.21.15.43:57170) 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 1 -- 172.21.15.43:0/733272248 learned_addr learned my addr 172.21.15.43:0/733272248 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 10 monclient(hunting): get_auth_request con 0x7f9b341358b0 auth_method 0 2024-04-19T00:49:13.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.289+0000 7f9b3259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9b34138cd0 0x7f9b3413f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request con 0x7f9b340a5b80 auth_method 0 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 10 monclient(hunting): get_auth_request con 0x7f9b34138cd0 auth_method 0 2024-04-19T00:49:13.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 10 monclient(hunting): handle_auth_done global_id 23114 payload 824 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9b34138cd0 msgr2=0x7f9b3413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9b34138cd0 0x7f9b3413f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 msgr2=0x7f9b341d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 monclient: found mon.noname-b 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3413b950 con 0x7f9b341358b0 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b340a5b80 0x7f9b341d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:49:13.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3259c640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b341358b0 0x7f9b3413eea0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800c7f0 tx=0x7f9b2800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 client.?.objecter ms_handle_connect 0x7f9b341358b0 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b28015070 con 0x7f9b341358b0 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.310 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 5 monclient: authenticate success, global_id 23114 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b28007ca0 con 0x7f9b341358b0 2024-04-19T00:49:13.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 monclient: _renew_subs 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b340a3040 con 0x7f9b341358b0 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b28007e50 con 0x7f9b341358b0 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 client.23114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3159a640 4 set_mon_vals no callback set 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.312 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 monclient: _renew_subs 2024-04-19T00:49:13.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b34135690 con 0x7f9b341358b0 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 1 librados: init done 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 0x7f9b34133ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3406d140 con 0x7f9b34133700 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9b28018070 con 0x7f9b341358b0 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 0x7f9b34133ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient: get_auth_request con 0x7f9b34133700 auth_method 0 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b20038fb0 0x7f9b2003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 6213+0+0 (secure 0 0 0) 0x7f9b28048fe0 con 0x7f9b341358b0 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 client.23114.objecter ms_dispatch 0x7f9b34000ba0 osd_map(216..216 src has 1..216) 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 3 client.23114.objecter handle_osd_map got epochs [216,216] > 0 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 3 client.23114.objecter handle_osd_map decoding full epoch 216 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_done global_id 54252 payload 824 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b34068240 con 0x7f9b34133700 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 20 client.23114.objecter dump_active .. 0 homeless 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 client.23114.objecter ms_handle_connect 0x7f9b34133700 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 0x7f9b34133ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b24011550 tx=0x7f9b24011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 client.23114.objecter resend_mon_ops 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b20038fb0 0x7f9b2003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 10 monclient: get_auth_request con 0x7f9b20038fb0 auth_method 0 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b240044a0 con 0x7f9b34133700 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.316 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9b24004030 con 0x7f9b34133700 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 msgr2=0x7f9b34133ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 0x7f9b34133ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b24011550 tx=0x7f9b24011a10 comp rx=0 tx=0).stop 2024-04-19T00:49:13.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 msgr2=0x7f9b34133ae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34133700 0x7f9b34133ae0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9b24011550 tx=0x7f9b24011a10 comp rx=0 tx=0).stop 2024-04-19T00:49:13.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b1b7fe640 10 client.23114.objecter ms_handle_connect 0x7f9b20038fb0 2024-04-19T00:49:13.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.293+0000 7f9b3359e640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b20038fb0 0x7f9b2003b470 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0af000 tx=0x7f9b2c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 0x7f9b34137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b341d13e0 con 0x7f9b34134020 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 0x7f9b34137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient: get_auth_request con 0x7f9b34134020 auth_method 0 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 10 monclient(hunting): handle_auth_done global_id 54261 payload 824 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 1 -- 172.21.15.43:0/733272248 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9b340693b0 con 0x7f9b34134020 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b32d9d640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 0x7f9b34137a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9b24000c10 tx=0x7f9b240046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 client.23114.objecter ms_handle_connect 0x7f9b34134020 2024-04-19T00:49:13.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 client.23114.objecter resend_mon_ops 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b24003d60 con 0x7f9b34134020 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 monclient: dump: 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.321 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f9b240038f0 con 0x7f9b34134020 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 msgr2=0x7f9b34137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 0x7f9b34137a90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9b24000c10 tx=0x7f9b240046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 msgr2=0x7f9b34137a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b1b7fe640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b34134020 0x7f9b34137a90 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9b24000c10 tx=0x7f9b240046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19119, 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:13.323 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.324 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.325 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:13.326 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.327 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:13.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:13.330 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:13.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.336 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.338 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:13.340 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.343 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:13.344 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 10 librados: watch_flush enter 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 10 librados: watch_flush exit 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 20 client.23114.objecter shutdown clearing up homeless session... 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 10 client.23114.objecter successfully canceled tick 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b20038fb0 msgr2=0x7f9b2003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b20038fb0 0x7f9b2003b470 secure :-1 s=READY pgs=3417 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c0af000 tx=0x7f9b2c0a7000 comp rx=0 tx=0).stop 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 10 monclient: shutdown 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b341358b0 msgr2=0x7f9b3413eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 --2- 172.21.15.43:0/733272248 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b341358b0 0x7f9b3413eea0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800c7f0 tx=0x7f9b2800ccb0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b3359e640 1 -- 172.21.15.43:0/733272248 reap_dead start 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b3359e640 1 -- 172.21.15.43:0/733272248 reap_dead start 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 shutdown_connections 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 >> 172.21.15.43:0/733272248 conn(0x7f9b340ab2f0 msgr2=0x7f9b340a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 shutdown_connections 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 -- 172.21.15.43:0/733272248 wait complete. 2024-04-19T00:49:13.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.301+0000 7f9b390be640 1 librados: shutdown 2024-04-19T00:49:13.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:13.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:49:13.347 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:49:13.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient: get_monmap_and_config 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient: monmap: 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:13.466681+0000 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:13.466681+0000 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:13.465 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:13.466 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 Processor -- start 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 -- start start 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient: init 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient: _add_conns ranks=[4,2,7,10,6,9,0,8,1,5,3] 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 0x7fe4080a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): picked mon.noname-c con 0x7fe4080a1e00 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4080a1190 0x7fe4080a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): picked mon.noname-b con 0x7fe4080a1190 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4080a4350 0x7fe40813fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): picked mon.noname-i con 0x7fe4080a4350 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:13.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe40806c690 con 0x7fe4080a1190 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe4080735a0 con 0x7fe4080a1e00 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe40809fff0 con 0x7fe4080a4350 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40cb58640 10 monclient(hunting): authenticate will time out at 1223.593791s 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 0x7fe4080a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 0x7fe4080a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58708/0 (socket says 172.21.15.43:58708) 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 -- 172.21.15.43:0/3216467117 learned_addr learned my addr 172.21.15.43:0/3216467117 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): get_auth_request con 0x7fe4080a1e00 auth_method 0 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4080a1190 0x7fe4080a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): get_auth_request con 0x7fe4080a1190 auth_method 0 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe406d9d640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4080a4350 0x7fe40813fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe406d9d640 10 monclient(hunting): get_auth_request con 0x7fe4080a4350 auth_method 0 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe406d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe406d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe406d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient(hunting): handle_auth_done global_id 22999 payload 274 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 -- 172.21.15.43:0/3216467117 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4080a4350 msgr2=0x7fe40813fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4080a4350 0x7fe40813fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 -- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4080a1190 msgr2=0x7fe4080a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4080a1190 0x7fe4080a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 monclient: found mon.noname-c 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:13.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe40659c640 1 -- 172.21.15.43:0/3216467117 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40806c330 con 0x7fe4080a1e00 2024-04-19T00:49:13.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.465+0000 7fe405d9b640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4080a1190 0x7fe4080a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 0x7fe4080a38d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e80 tx=0x7fe3f801c910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=88cb734fa6d35b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 1 -- 172.21.15.43:0/3216467117 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f80090d0 con 0x7fe4080a1e00 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: dump: 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.480 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 5 monclient: authenticate success, global_id 22999 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 1 -- 172.21.15.43:0/3216467117 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f80a5a70 con 0x7fe4080a1e00 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 1 -- 172.21.15.43:0/3216467117 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f80a5c60 con 0x7fe4080a1e00 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 4 set_mon_vals no callback set 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40559a640 10 monclient: dump: 2024-04-19T00:49:13.482 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: shutdown 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 msgr2=0x7fe4080a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 0x7fe4080a38d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e80 tx=0x7fe3f801c910 comp rx=0 tx=0).stop 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- 172.21.15.43:0/3216467117 shutdown_connections 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4080a1190 0x7fe4080a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4080a4350 0x7fe40813fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- 172.21.15.43:0/3216467117 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1e00 0x7fe4080a38d0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- 172.21.15.43:0/3216467117 >> 172.21.15.43:0/3216467117 conn(0x7fe4080a84c0 msgr2=0x7fe4080a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- 172.21.15.43:0/3216467117 shutdown_connections 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- 172.21.15.43:0/3216467117 wait complete. 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: monmap: 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:13.484 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 librados: starting msgr at 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 librados: starting objecter 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 Processor -- start 2024-04-19T00:49:13.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- start start 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 librados: setting wanted keys 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 librados: calling monclient init 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: init 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient: _add_conns ranks=[4,8,3,0,2,1,5,6,7] 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1190 0x7fe408071c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): picked mon.noname-c con 0x7fe4080a1190 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe4080a1e00 0x7fe4081d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): picked mon.noname-e con 0x7fe4080a1e00 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4080a4350 0x7fe4081d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): picked mon.noname-g con 0x7fe4080a4350 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe408142820 con 0x7fe4080a4350 2024-04-19T00:49:13.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe4080a23f0 con 0x7fe4080a1190 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe4081429a0 con 0x7fe4080a1e00 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40cb58640 10 monclient(hunting): authenticate will time out at 1223.599733s 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1190 0x7fe408071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe4080a1e00 0x7fe4081d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe4080a1e00 0x7fe4081d0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:50948/0 (socket says 172.21.15.43:50948) 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1190 0x7fe408071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58710/0 (socket says 172.21.15.43:58710) 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 1 -- 172.21.15.43:0/4158462084 learned_addr learned my addr 172.21.15.43:0/4158462084 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 10 monclient(hunting): get_auth_request con 0x7fe4080a1e00 auth_method 0 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe405d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 10 monclient(hunting): get_auth_request con 0x7fe4080a1190 auth_method 0 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.469+0000 7fe40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4080a4350 0x7fe4081d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 10 monclient(hunting): get_auth_request con 0x7fe4080a4350 auth_method 0 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 10 monclient(hunting): handle_auth_done global_id 22796 payload 824 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1190 msgr2=0x7fe408071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1190 0x7fe408071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4080a4350 msgr2=0x7fe4081d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4080a4350 0x7fe4081d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 monclient: found mon.noname-e 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe408073270 con 0x7fe4080a1e00 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe4080a4350 0x7fe4081d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4080a1190 0x7fe408071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe405d9b640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe4080a1e00 0x7fe4081d0f70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000ebe0 tx=0x7fe3f000c520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 client.?.objecter ms_handle_connect 0x7fe4080a1e00 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f0017070 con 0x7fe4080a1e00 2024-04-19T00:49:13.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: dump: 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.490 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 5 monclient: authenticate success, global_id 22796 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f000cc30 con 0x7fe4080a1e00 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 monclient: _renew_subs 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f000cde0 con 0x7fe4080a1e00 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe40809e370 con 0x7fe4080a1e00 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3fffff640 4 set_mon_vals no callback set 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 client.22796.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: dump: 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.492 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 monclient: _renew_subs 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe40809c130 con 0x7fe4080a1e00 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 1 librados: init done 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 0x7fe408138b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe40806d140 con 0x7fe408138790 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 0x7fe408138b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe3f001a070 con 0x7fe4080a1e00 2024-04-19T00:49:13.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient: get_auth_request con 0x7fe408138790 auth_method 0 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe3e8038e10 0x7fe3e803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 10 monclient(hunting): handle_auth_done global_id 23132 payload 824 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe40809eef0 con 0x7fe408138790 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(216..216 src has 1..216) ==== 6213+0+0 (secure 0 0 0) 0x7fe3f000f040 con 0x7fe4080a1e00 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 client.22796.objecter ms_dispatch 0x7fe408000ba0 osd_map(216..216 src has 1..216) 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 3 client.22796.objecter handle_osd_map got epochs [216,216] > 0 2024-04-19T00:49:13.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 3 client.22796.objecter handle_osd_map decoding full epoch 216 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 20 client.22796.objecter dump_active .. 0 homeless 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 client.22796.objecter ms_handle_connect 0x7fe408138790 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 client.22796.objecter resend_mon_ops 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe40659c640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 0x7fe408138b70 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e50 tx=0x7fe3f80196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe3e8038e10 0x7fe3e803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 10 monclient: get_auth_request con 0x7fe3e8038e10 auth_method 0 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f80044b0 con 0x7fe408138790 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: dump: 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe3f8004040 con 0x7fe408138790 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 msgr2=0x7fe408138b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 0x7fe408138b70 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e50 tx=0x7fe3f80196b0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 msgr2=0x7fe408138b70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe408138790 0x7fe408138b70 secure :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8009e50 tx=0x7fe3f80196b0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe406d9d640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe3e8038e10 0x7fe3e803b2d0 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7fe4080a8ef0 tx=0x7fe400008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.473+0000 7fe3feffd640 10 client.22796.objecter ms_handle_connect 0x7fe3e8038e10 2024-04-19T00:49:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 0x7fe4080a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe408073050 con 0x7fe4081390b0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 0x7fe4080a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient: get_auth_request con 0x7fe4081390b0 auth_method 0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 10 monclient(hunting): handle_auth_done global_id 23141 payload 824 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 1 -- 172.21.15.43:0/4158462084 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe40806c8d0 con 0x7fe4081390b0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 client.22796.objecter ms_handle_connect 0x7fe4081390b0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 client.22796.objecter resend_mon_ops 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40659c640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 0x7fe4080a5dc0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe3f801c950 tx=0x7fe3f80a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3f80c2e60 con 0x7fe4081390b0 2024-04-19T00:49:13.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 monclient: dump: 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.500 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fe3f8004820 con 0x7fe4081390b0 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 msgr2=0x7fe4080a5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 0x7fe4080a5dc0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe3f801c950 tx=0x7fe3f80a7000 comp rx=0 tx=0).stop 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 msgr2=0x7fe4080a5dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe3feffd640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe4081390b0 0x7fe4080a5dc0 secure :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe3f801c950 tx=0x7fe3f80a7000 comp rx=0 tx=0).stop 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:13.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 296980, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:13.502 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.503 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:13.505 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.506 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.508 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:13.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:13.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.514 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:13.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.519 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:13.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:13.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:13.523 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 10 librados: watch_flush enter 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 10 librados: watch_flush exit 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 20 client.22796.objecter shutdown clearing up homeless session... 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 10 client.22796.objecter successfully canceled tick 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe3e8038e10 msgr2=0x7fe3e803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe3e8038e10 0x7fe3e803b2d0 secure :-1 s=READY pgs=3418 cs=0 l=1 rev1=1 crypto rx=0x7fe4080a8ef0 tx=0x7fe400008040 comp rx=0 tx=0).stop 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 10 monclient: shutdown 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe4080a1e00 msgr2=0x7fe4081d0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 --2- 172.21.15.43:0/4158462084 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe4080a1e00 0x7fe4081d0f70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000ebe0 tx=0x7fe3f000c520 comp rx=0 tx=0).stop 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe406d9d640 1 -- 172.21.15.43:0/4158462084 reap_dead start 2024-04-19T00:49:13.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe406d9d640 1 -- 172.21.15.43:0/4158462084 reap_dead start 2024-04-19T00:49:13.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 shutdown_connections 2024-04-19T00:49:13.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 >> 172.21.15.43:0/4158462084 conn(0x7fe4080a84c0 msgr2=0x7fe4080a7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 shutdown_connections 2024-04-19T00:49:13.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 -- 172.21.15.43:0/4158462084 wait complete. 2024-04-19T00:49:13.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.481+0000 7fe40cb58640 1 librados: shutdown 2024-04-19T00:49:13.526 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:13.526 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:49:13.526 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:49:13.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient: get_monmap_and_config 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient: monmap: 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:13.652702+0000 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:13.652702+0000 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.651 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:13.652 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:13.652 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 Processor -- start 2024-04-19T00:49:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 -- start start 2024-04-19T00:49:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient: init 2024-04-19T00:49:13.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient: _add_conns ranks=[0,5,10,6,4,1,9,7,3,2,8] 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe380138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe38013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): picked mon.noname-h con 0x7fe380139540 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3800a5bb0 0x7fe380140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): picked mon.noname-f con 0x7fe3800a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3800a0d40 con 0x7fe380138a50 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe3800a1020 con 0x7fe3800a5bb0 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe3800a0780 con 0x7fe380139540 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.649+0000 7fe387956640 10 monclient(hunting): authenticate will time out at 1223.779717s 2024-04-19T00:49:13.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe380138a50 0x7fe380138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57118/0 (socket says 172.21.15.43:57118) 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 1 -- 172.21.15.43:0/3709594074 learned_addr learned my addr 172.21.15.43:0/3709594074 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 --2- 172.21.15.43:0/3709594074 >> v2:172.21.15.161:3300/0 conn(0x7fe3800a5bb0 0x7fe380140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe38013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): get_auth_request con 0x7fe3800a5bb0 auth_method 0 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): get_auth_request con 0x7fe380139540 auth_method 0 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2625503048 0 0) 0x7fe3800a0d40 con 0x7fe380138a50 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): _negotiate 2024-04-19T00:49:13.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): my global_id is 54270 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe368003530 con 0x7fe380138a50 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1967519222 0 0) 0x7fe368003530 con 0x7fe380138a50 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 msgr2=0x7fe38013dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe38013dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 >> v2:172.21.15.161:3300/0 conn(0x7fe3800a5bb0 msgr2=0x7fe380140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 --2- 172.21.15.43:0/3709594074 >> v2:172.21.15.161:3300/0 conn(0x7fe3800a5bb0 0x7fe380140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3800a0a60 con 0x7fe380138a50 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fe370001cf0 con 0x7fe380138a50 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: dump: 2024-04-19T00:49:13.667 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.668 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 >> v1:172.21.15.43:6789/0 conn(0x7fe380138a50 legacy=0x7fe380138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient: _add_conns ranks=[3,1,5,6,8,2,4,0,7] 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe368004240 0x7fe368004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): picked mon.g con 0x7fe368004240 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe368004da0 0x7fe368005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): picked mon.f con 0x7fe368004da0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe36800bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): picked mon.h con 0x7fe380139540 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3800a1020 con 0x7fe368004da0 2024-04-19T00:49:13.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe3800a0780 con 0x7fe368004240 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe36800c280 con 0x7fe380139540 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe37ffff640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe36800bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe368004da0 0x7fe368005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): get_auth_request con 0x7fe380139540 auth_method 0 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe368004240 0x7fe368004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:13.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): get_auth_request con 0x7fe368004da0 auth_method 0 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): get_auth_request con 0x7fe368004240 auth_method 0 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient(hunting): handle_auth_done global_id 54270 payload 293 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 -- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe368004240 msgr2=0x7fe368004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe368004240 0x7fe368004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 -- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe368004da0 msgr2=0x7fe368005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe368004da0 0x7fe368005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 monclient: found mon.h 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 1 -- 172.21.15.43:0/3709594074 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe368003530 con 0x7fe380139540 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe385ef3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe387956640 5 monclient: authenticate success, global_id 54270 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe387956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.653+0000 7fe384ef1640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe368004da0 0x7fe368005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe36800bb40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe378009ae0 tx=0x7fe37801caf0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=a82b2e26b51bc451 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3780096b0 con 0x7fe380139540 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: dump: 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.673 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3780a6520 con 0x7fe380139540 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 1 -- 172.21.15.43:0/3709594074 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3780a6b40 con 0x7fe380139540 2024-04-19T00:49:13.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37ffff640 10 monclient: dump: 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.675 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 4 set_mon_vals no callback set 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: shutdown 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 msgr2=0x7fe36800bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 --2- 172.21.15.43:0/3709594074 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe380139540 0x7fe36800bb40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe378009ae0 tx=0x7fe37801caf0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 -- 172.21.15.43:0/3709594074 reap_dead start 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/3709594074 shutdown_connections 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/3709594074 >> 172.21.15.43:0/3709594074 conn(0x7fe3800a7cb0 msgr2=0x7fe3800a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/3709594074 shutdown_connections 2024-04-19T00:49:13.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/3709594074 wait complete. 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: monmap: 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 librados: starting msgr at 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 librados: starting objecter 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 Processor -- start 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- start start 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 librados: setting wanted keys 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 librados: calling monclient init 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: init 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: _add_conns ranks=[7,2,0,3,6,8,5,4,1] 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe3801d0830 0x7fe3801369b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): picked mon.noname-i con 0x7fe3801d0830 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe380136ef0 0x7fe38009c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): picked mon.noname-b con 0x7fe380136ef0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 0x7fe38009d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): picked mon.noname-a con 0x7fe38009ce60 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe380141610 con 0x7fe38009ce60 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe380141490 con 0x7fe380136ef0 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe380141790 con 0x7fe3801d0830 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient(hunting): authenticate will time out at 1223.786543s 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 0x7fe38009d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe380136ef0 0x7fe38009c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 0x7fe38009d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35508/0 (socket says 172.21.15.43:35508) 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 -- 172.21.15.43:0/351062254 learned_addr learned my addr 172.21.15.43:0/351062254 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): get_auth_request con 0x7fe38009ce60 auth_method 0 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): get_auth_request con 0x7fe380136ef0 auth_method 0 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe384ef1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe3856f2640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe3801d0830 0x7fe3801369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe3856f2640 10 monclient(hunting): get_auth_request con 0x7fe3801d0830 auth_method 0 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe3856f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe3856f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe3856f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient(hunting): handle_auth_done global_id 54279 payload 824 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe3801d0830 msgr2=0x7fe3801369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe3801d0830 0x7fe3801369b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe380136ef0 msgr2=0x7fe38009c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe380136ef0 0x7fe38009c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 monclient: found mon.noname-a 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:13.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38009ee60 con 0x7fe38009ce60 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe385ef3640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 0x7fe38009d2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3780b0040 tx=0x7fe378019890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 client.?.objecter ms_handle_connect 0x7fe38009ce60 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3780bd070 con 0x7fe38009ce60 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: dump: 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.682 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe378000e50 con 0x7fe38009ce60 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3740096b0 con 0x7fe38009ce60 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 5 monclient: authenticate success, global_id 54279 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: _renew_subs 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe380123360 con 0x7fe38009ce60 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37f7fe640 4 set_mon_vals no callback set 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe37dffb640 10 monclient: dump: 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.684 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 client.54279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: _renew_subs 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe380123080 con 0x7fe38009ce60 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.657+0000 7fe387956640 1 librados: init done 2024-04-19T00:49:13.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe387956640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe387956640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 0x7fe3800a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe380121990 con 0x7fe3800a25b0 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe3780c0060 con 0x7fe38009ce60 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 0x7fe3800a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient: get_auth_request con 0x7fe3800a25b0 auth_method 0 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe36c038e10 0x7fe36c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(217..217 src has 1..217) ==== 6253+0+0 (secure 0 0 0) 0x7fe3780b4040 con 0x7fe38009ce60 2024-04-19T00:49:13.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 client.54279.objecter ms_dispatch 0x7fe380000ba0 osd_map(217..217 src has 1..217) 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 3 client.54279.objecter handle_osd_map got epochs [217,217] > 0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 3 client.54279.objecter handle_osd_map decoding full epoch 217 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 20 client.54279.objecter dump_active .. 0 homeless 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 10 monclient(hunting): handle_auth_done global_id 23017 payload 824 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3801d0c10 con 0x7fe3800a25b0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe3856f2640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 0x7fe3800a2990 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe370001340 tx=0x7fe370011620 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 client.54279.objecter ms_handle_connect 0x7fe3800a25b0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 client.54279.objecter resend_mon_ops 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe384ef1640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe36c038e10 0x7fe36c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe384ef1640 10 monclient: get_auth_request con 0x7fe36c038e10 auth_method 0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe370019020 con 0x7fe3800a25b0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 monclient: dump: 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.687 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.688 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe370011e60 con 0x7fe3800a25b0 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 msgr2=0x7fe3800a2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 0x7fe3800a2990 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fe370001340 tx=0x7fe370011620 comp rx=0 tx=0).stop 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 msgr2=0x7fe3800a2990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a25b0 0x7fe3800a2990 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe384ef1640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe36c038e10 0x7fe36c03b2d0 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7fe38006e8c0 tx=0x7fe374009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.661+0000 7fe37dffb640 10 client.54279.objecter ms_handle_connect 0x7fe36c038e10 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe387956640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe387956640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 0x7fe3801dbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe387956640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe38009e430 con 0x7fe3800a2ed0 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 0x7fe3801dbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient: get_auth_request con 0x7fe3800a2ed0 auth_method 0 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.665+0000 7fe3856f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe3856f2640 10 monclient(hunting): handle_auth_done global_id 23026 payload 824 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe3856f2640 1 -- 172.21.15.43:0/351062254 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe3801372d0 con 0x7fe3800a2ed0 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 client.54279.objecter ms_handle_connect 0x7fe3800a2ed0 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 client.54279.objecter resend_mon_ops 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe3856f2640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 0x7fe3801dbba0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe370001340 tx=0x7fe3800a0c30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe370019040 con 0x7fe3800a2ed0 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 monclient: dump: 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.692 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe370015e40 con 0x7fe3800a2ed0 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 msgr2=0x7fe3801dbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 0x7fe3801dbba0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe370001340 tx=0x7fe3800a0c30 comp rx=0 tx=0).stop 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 msgr2=0x7fe3801dbba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe37dffb640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe3800a2ed0 0x7fe3801dbba0 secure :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe370001340 tx=0x7fe3800a0c30 comp rx=0 tx=0).stop 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 297188, 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:13.694 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.695 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.696 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:13.697 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:13.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.708 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:13.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.714 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:13.715 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 10 librados: watch_flush enter 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 10 librados: watch_flush exit 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 20 client.54279.objecter shutdown clearing up homeless session... 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 10 client.54279.objecter successfully canceled tick 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe36c038e10 msgr2=0x7fe36c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe36c038e10 0x7fe36c03b2d0 secure :-1 s=READY pgs=3419 cs=0 l=1 rev1=1 crypto rx=0x7fe38006e8c0 tx=0x7fe374009190 comp rx=0 tx=0).stop 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 10 monclient: shutdown 2024-04-19T00:49:13.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 msgr2=0x7fe38009d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe385ef3640 1 -- 172.21.15.43:0/351062254 reap_dead start 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 0x7fe38009d2c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3780b0040 tx=0x7fe378019890 comp rx=0 tx=0).stop 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 shutdown_connections 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 --2- 172.21.15.43:0/351062254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe38009ce60 0x7fe38009d2c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 >> 172.21.15.43:0/351062254 conn(0x7fe3800a7cb0 msgr2=0x7fe38013f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 shutdown_connections 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 -- 172.21.15.43:0/351062254 wait complete. 2024-04-19T00:49:13.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.669+0000 7fe387956640 1 librados: shutdown 2024-04-19T00:49:13.718 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:13.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:49:13.719 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:49:13.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.837+0000 7f0409fab640 10 monclient: get_monmap_and_config 2024-04-19T00:49:13.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.837+0000 7f0409fab640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient: monmap: 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:13.842482+0000 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:13.842482+0000 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.841 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.842 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.842 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.842 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:13.842 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:13.842 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 Processor -- start 2024-04-19T00:49:13.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 -- start start 2024-04-19T00:49:13.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient: init 2024-04-19T00:49:13.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient: _add_conns ranks=[8,6,10,0,5,7,2,1,3,9,4] 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 0x7f04041390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): picked mon.noname-e con 0x7f0404138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 0x7f04041395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): picked mon.noname-d con 0x7f04040a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f04041358b0 0x7f0404135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): picked mon.noname-f con 0x7f04041358b0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f04040a4410 con 0x7f04041358b0 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0404139b30 con 0x7f04040a5b80 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f04040a3dc0 con 0x7f0404138cd0 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0409fab640 10 monclient(hunting): authenticate will time out at 1223.969590s 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 0x7f04041395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 0x7f04041390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 0x7f04041395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33814/0 (socket says 172.21.15.43:33814) 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 0x7f04041390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:50952/0 (socket says 172.21.15.43:50952) 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 -- 172.21.15.43:0/707232708 learned_addr learned my addr 172.21.15.43:0/707232708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): get_auth_request con 0x7f04040a5b80 auth_method 0 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): get_auth_request con 0x7f0404138cd0 auth_method 0 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0403fff640 1 --2- 172.21.15.43:0/707232708 >> v2:172.21.15.161:3300/0 conn(0x7f04041358b0 0x7f0404135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient(hunting): handle_auth_done global_id 23145 payload 274 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 -- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 msgr2=0x7f04041390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 --2- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 0x7f04041390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 -- 172.21.15.43:0/707232708 >> v2:172.21.15.161:3300/0 conn(0x7f04041358b0 msgr2=0x7f0404135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 --2- 172.21.15.43:0/707232708 >> v2:172.21.15.161:3300/0 conn(0x7f04041358b0 0x7f0404135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 monclient: found mon.noname-d 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0402ffd640 1 -- 172.21.15.43:0/707232708 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04040a40a0 con 0x7f04040a5b80 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0403fff640 10 monclient: get_auth_request con 0x7f04041358b0 auth_method 0 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f0403fff640 1 --2- 172.21.15.43:0/707232708 >> v2:172.21.15.161:3300/0 conn(0x7f04041358b0 0x7f0404135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.841+0000 7f04037fe640 1 --2- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 0x7f04041390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 1 --2- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 0x7f04041395f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009870 tx=0x7f03f401c3c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=5850b6bb54d825e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 1 -- 172.21.15.43:0/707232708 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03f40b2070 con 0x7f04040a5b80 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: dump: 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.855 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.856 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 5 monclient: authenticate success, global_id 23145 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 1 -- 172.21.15.43:0/707232708 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f40a9070 con 0x7f04040a5b80 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 1 -- 172.21.15.43:0/707232708 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03f40abc40 con 0x7f04040a5b80 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 4 set_mon_vals no callback set 2024-04-19T00:49:13.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04027fc640 10 monclient: dump: 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.858 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: shutdown 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 msgr2=0x7f04041395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 0x7f04041395f0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009870 tx=0x7f03f401c3c0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/707232708 shutdown_connections 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- 172.21.15.43:0/707232708 >> v2:172.21.15.161:3300/0 conn(0x7f04041358b0 0x7f0404135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04040a5b80 0x7f04041395f0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- 172.21.15.43:0/707232708 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0404138cd0 0x7f04041390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/707232708 >> 172.21.15.43:0/707232708 conn(0x7f04040ab2f0 msgr2=0x7f04040ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/707232708 shutdown_connections 2024-04-19T00:49:13.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/707232708 wait complete. 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: build_initial_monmap 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: monmap: 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:13.860 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 librados: starting msgr at 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 librados: starting objecter 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 Processor -- start 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- start start 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 librados: setting wanted keys 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 librados: calling monclient init 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: init 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:13.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: _add_conns ranks=[4,6,1,2,0,5,7,3,8] 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 0x7f04041d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): picked mon.noname-c con 0x7f04040a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04041358b0 0x7f040413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): picked mon.noname-d con 0x7f04041358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0404138cd0 0x7f040413f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): picked mon.noname-f con 0x7f0404138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f04040a2010 con 0x7f0404138cd0 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f04040a1e90 con 0x7f04040a5b80 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0404142aa0 con 0x7f04041358b0 2024-04-19T00:49:13.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient(hunting): authenticate will time out at 1223.974732s 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 0x7f04041d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 0x7f04041d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58740/0 (socket says 172.21.15.43:58740) 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 -- 172.21.15.43:0/895885606 learned_addr learned my addr 172.21.15.43:0/895885606 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): get_auth_request con 0x7f04040a5b80 auth_method 0 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04041358b0 0x7f040413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): get_auth_request con 0x7f04041358b0 auth_method 0 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient(hunting): handle_auth_done global_id 23035 payload 824 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0403fff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0404138cd0 0x7f040413f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04041358b0 msgr2=0x7f040413ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04041358b0 0x7f040413ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0404138cd0 msgr2=0x7f040413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0404138cd0 0x7f040413f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient: _finish_hunting 0 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 monclient: found mon.noname-c 2024-04-19T00:49:13.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0404135000 con 0x7f04040a5b80 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0403fff640 10 monclient: get_auth_request con 0x7f0404138cd0 auth_method 0 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0402ffd640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f04041358b0 0x7f040413ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0403fff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0404138cd0 0x7f040413f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f04037fe640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 0x7f04041d0a30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f03f800e9b0 tx=0x7f03f800ee70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 client.?.objecter ms_handle_connect 0x7f04040a5b80 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03f8017070 con 0x7f04040a5b80 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: dump: 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: _finish_auth 0 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03f800ce70 con 0x7f04040a5b80 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 5 monclient: authenticate success, global_id 23035 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: _renew_subs 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04040a3590 con 0x7f04040a5b80 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03f80138e0 con 0x7f04040a5b80 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f03ebfff640 10 monclient: dump: 2024-04-19T00:49:13.867 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.868 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 client.23035.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: _renew_subs 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0401ffb640 4 set_mon_vals no callback set 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f040413ae80 con 0x7f04040a5b80 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.845+0000 7f0409fab640 1 librados: init done 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0409fab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0409fab640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 0x7f04041335d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f040406d140 con 0x7f0404139f90 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f03f801a070 con 0x7f04040a5b80 2024-04-19T00:49:13.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 0x7f04041335d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient: get_auth_request con 0x7f0404139f90 auth_method 0 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f03f0038e10 0x7f03f003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(217..217 src has 1..217) ==== 6253+0+0 (secure 0 0 0) 0x7f03f800f040 con 0x7f04040a5b80 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 client.23035.objecter ms_dispatch 0x7f0404000ba0 osd_map(217..217 src has 1..217) 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 3 client.23035.objecter handle_osd_map got epochs [217,217] > 0 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 3 client.23035.objecter handle_osd_map decoding full epoch 217 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 10 monclient(hunting): handle_auth_done global_id 23163 payload 824 2024-04-19T00:49:13.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f040413b2f0 con 0x7f0404139f90 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 20 client.23035.objecter dump_active .. 0 homeless 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0402ffd640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 0x7f04041335d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f04040a4380 tx=0x7f03f40191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 client.23035.objecter ms_handle_connect 0x7f0404139f90 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 client.23035.objecter resend_mon_ops 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0403fff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f03f0038e10 0x7f03f003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0403fff640 10 monclient: get_auth_request con 0x7f03f0038e10 auth_method 0 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03f40b2040 con 0x7f0404139f90 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 monclient: dump: 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.871 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03f40093d0 con 0x7f0404139f90 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:13.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 msgr2=0x7f04041335d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 0x7f04041335d0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f04040a4380 tx=0x7f03f40191c0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 msgr2=0x7f04041335d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404139f90 0x7f04041335d0 secure :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f04040a4380 tx=0x7f03f40191c0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f03ebfff640 10 client.23035.objecter ms_handle_connect 0x7f03f0038e10 2024-04-19T00:49:13.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.849+0000 7f0403fff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f03f0038e10 0x7f03f003b2d0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f03fc007f40 tx=0x7f03fc007490 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 0x7f0404133f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0404068240 con 0x7f0404133b10 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 0x7f0404133f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient: get_auth_request con 0x7f0404133b10 auth_method 0 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:13.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 10 monclient(hunting): handle_auth_done global_id 23172 payload 824 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 1 -- 172.21.15.43:0/895885606 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0404134460 con 0x7f0404133b10 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 client.23035.objecter ms_handle_connect 0x7f0404133b10 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 client.23035.objecter resend_mon_ops 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0402ffd640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 0x7f0404133f20 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009870 tx=0x7f03f40a9000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03f40b2040 con 0x7f0404133b10 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 monclient: dump: 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:13.876 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f03f40c4070 con 0x7f0404133b10 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:13.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 msgr2=0x7f0404133f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 0x7f0404133f20 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009870 tx=0x7f03f40a9000 comp rx=0 tx=0).stop 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 msgr2=0x7f0404133f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f03ebfff640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0404133b10 0x7f0404133f20 secure :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f03f4009870 tx=0x7f03f40a9000 comp rx=0 tx=0).stop 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:13.878 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 297372, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:49:13.879 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.880 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.881 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:13.882 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:13.883 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:13.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:13.888 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:13.890 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:13.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:13.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:13.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:13.899 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:13.900 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:13.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 10 librados: watch_flush enter 2024-04-19T00:49:13.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 10 librados: watch_flush exit 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 20 client.23035.objecter shutdown clearing up homeless session... 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 10 client.23035.objecter successfully canceled tick 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f03f0038e10 msgr2=0x7f03f003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f03f0038e10 0x7f03f003b2d0 secure :-1 s=READY pgs=3420 cs=0 l=1 rev1=1 crypto rx=0x7f03fc007f40 tx=0x7f03fc007490 comp rx=0 tx=0).stop 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 10 monclient: shutdown 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 msgr2=0x7f04041d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 0x7f04041d0a30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f03f800e9b0 tx=0x7f03f800ee70 comp rx=0 tx=0).stop 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0403fff640 1 -- 172.21.15.43:0/895885606 reap_dead start 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 shutdown_connections 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 --2- 172.21.15.43:0/895885606 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f04040a5b80 0x7f04041d0a30 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:13.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 >> 172.21.15.43:0/895885606 conn(0x7f04040ab2f0 msgr2=0x7f04040a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:13.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 shutdown_connections 2024-04-19T00:49:13.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 -- 172.21.15.43:0/895885606 wait complete. 2024-04-19T00:49:13.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:13.857+0000 7f0409fab640 1 librados: shutdown 2024-04-19T00:49:13.903 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:13.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:49:13.903 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient: get_monmap_and_config 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient: monmap: 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:14.022773+0000 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:14.022773+0000 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.021 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:14.022 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 Processor -- start 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 -- start start 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient: init 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient: _add_conns ranks=[10,5,0,6,1,7,9,8,2,3,4] 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a40133b20 0x7f0a4009d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): picked mon.noname-f con 0x7f0a40133b20 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 0x7f0a400a2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): picked mon.noname-h con 0x7f0a40133f00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:14.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a40134300 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a400a0f00 con 0x7f0a40134300 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0a4009cf40 con 0x7f0a40133b20 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a4009c610 con 0x7f0a40133f00 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 10 monclient(hunting): authenticate will time out at 1224.149937s 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0a40134300 0x7f0a400a2e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57124/0 (socket says 172.21.15.43:57124) 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 -- 172.21.15.43:0/849918203 learned_addr learned my addr 172.21.15.43:0/849918203 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 1 --2- 172.21.15.43:0/849918203 >> v2:172.21.15.161:3300/0 conn(0x7f0a40133b20 0x7f0a4009d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): get_auth_request con 0x7f0a40133b20 auth_method 0 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 0x7f0a400a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): get_auth_request con 0x7f0a40133f00 auth_method 0 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164476512 0 0) 0x7f0a400a0f00 con 0x7f0a40134300 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): _negotiate 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): my global_id is 54288 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a2c003530 con 0x7f0a40134300 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3864268745 0 0) 0x7f0a2c003530 con 0x7f0a40134300 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 msgr2=0x7f0a400a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 0x7f0a400a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 >> v2:172.21.15.161:3300/0 conn(0x7f0a40133b20 msgr2=0x7f0a4009d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 --2- 172.21.15.43:0/849918203 >> v2:172.21.15.161:3300/0 conn(0x7f0a40133b20 0x7f0a4009d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4009c8f0 con 0x7f0a40134300 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f0a38001cf0 con 0x7f0a40134300 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: dump: 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.031 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 >> v1:172.21.15.43:6789/0 conn(0x7f0a40134300 legacy=0x7f0a400a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient: _add_conns ranks=[5,3,0,1,8,7,2,4,6] 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 0x7f0a2c004210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): picked mon.h con 0x7f0a40133f00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a2c004810 0x7f0a2c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): picked mon.g con 0x7f0a2c004810 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a2c0053a0 0x7f0a2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): picked mon.a con 0x7f0a2c0053a0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:14.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4009cf40 con 0x7f0a2c0053a0 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4009c610 con 0x7f0a2c004810 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a2c00c2e0 con 0x7f0a40133f00 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a44f08640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 1 --2- 172.21.15.43:0/849918203 >> v2:172.21.15.161:3300/0 conn(0x7f0a40133b20 0x7f0a4009d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a2c0053a0 0x7f0a2c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): get_auth_request con 0x7f0a2c0053a0 auth_method 0 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient(hunting): handle_auth_done global_id 54288 payload 293 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 -- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 msgr2=0x7f0a2c004210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45709640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a2c004810 0x7f0a2c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a45f0a640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 0x7f0a2c004210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133f00 0x7f0a2c004210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 -- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a2c004810 msgr2=0x7f0a2c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a2c004810 0x7f0a2c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 monclient: found mon.a 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 1 -- 172.21.15.43:0/849918203 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c003530 con 0x7f0a2c0053a0 2024-04-19T00:49:14.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4670b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 5 monclient: authenticate success, global_id 54288 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.021+0000 7f0a4816e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4670b640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a2c0053a0 0x7f0a2c00b9d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800a650 tx=0x7f0a3801d0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1f321975a7c069c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3800a2a0 con 0x7f0a2c0053a0 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: dump: 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.035 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a38007540 con 0x7f0a2c0053a0 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 1 -- 172.21.15.43:0/849918203 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a380ac3d0 con 0x7f0a2c0053a0 2024-04-19T00:49:14.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 4 set_mon_vals no callback set 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a44f08640 10 monclient: dump: 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.037 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: shutdown 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- 172.21.15.43:0/849918203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a2c0053a0 msgr2=0x7f0a2c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 --2- 172.21.15.43:0/849918203 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0a2c0053a0 0x7f0a2c00b9d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a3800a650 tx=0x7f0a3801d0c0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4670b640 1 -- 172.21.15.43:0/849918203 reap_dead start 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- 172.21.15.43:0/849918203 shutdown_connections 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- 172.21.15.43:0/849918203 >> 172.21.15.43:0/849918203 conn(0x7f0a400a77d0 msgr2=0x7f0a400a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- 172.21.15.43:0/849918203 shutdown_connections 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- 172.21.15.43:0/849918203 wait complete. 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: monmap: 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.038 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.039 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 librados: starting msgr at 2024-04-19T00:49:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 librados: starting objecter 2024-04-19T00:49:14.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 Processor -- start 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- start start 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 librados: setting wanted keys 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 librados: calling monclient init 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: init 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient: _add_conns ranks=[7,3,5,0,4,8,1,2,6] 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a40132e70 0x7f0a401d0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): picked mon.noname-i con 0x7f0a40132e70 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a40133250 0x7f0a40137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): picked mon.noname-g con 0x7f0a40133250 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133630 0x7f0a40137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): picked mon.noname-h con 0x7f0a40133630 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0a400a3700 con 0x7f0a40133250 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a400a3580 con 0x7f0a40133630 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0a4013f3e0 con 0x7f0a40132e70 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.025+0000 7f0a4816e640 10 monclient(hunting): authenticate will time out at 1224.155979s 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a40133250 0x7f0a40137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a40132e70 0x7f0a401d0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133630 0x7f0a40137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a40133250 0x7f0a40137890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:44338/0 (socket says 172.21.15.43:44338) 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 -- 172.21.15.43:0/3949677805 learned_addr learned my addr 172.21.15.43:0/3949677805 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): get_auth_request con 0x7f0a40133250 auth_method 0 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): get_auth_request con 0x7f0a40132e70 auth_method 0 2024-04-19T00:49:14.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): get_auth_request con 0x7f0a40133630 auth_method 0 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient(hunting): handle_auth_done global_id 19384 payload 824 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133630 msgr2=0x7f0a40137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133630 0x7f0a40137dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a40133250 msgr2=0x7f0a40137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a40133250 0x7f0a40137890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 monclient: found mon.noname-i 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4009f460 con 0x7f0a40132e70 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a40133250 0x7f0a40137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a40133630 0x7f0a40137dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45f0a640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a40132e70 0x7f0a401d0ac0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c010870 tx=0x7f0a3c010d30 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 client.?.objecter ms_handle_connect 0x7f0a40132e70 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3c00f040 con 0x7f0a40132e70 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: dump: 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.051 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 5 monclient: authenticate success, global_id 19384 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 monclient: _renew_subs 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:49:14.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a401394c0 con 0x7f0a40132e70 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3c00e980 con 0x7f0a40132e70 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 client.19384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 monclient: _renew_subs 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a37fff640 4 set_mon_vals no callback set 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4009ed60 con 0x7f0a40132e70 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 1 librados: init done 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3c021060 con 0x7f0a40132e70 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: dump: 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.053 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.054 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 0x7f0a40071080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a400a02a0 con 0x7f0a40138310 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 0x7f0a40071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient: get_auth_request con 0x7f0a40138310 auth_method 0 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 10 monclient(hunting): handle_auth_done global_id 22814 payload 824 2024-04-19T00:49:14.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a4006d120 con 0x7f0a40138310 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.7 v2:172.21.15.161:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0a3c00eb30 con 0x7f0a40132e70 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a45709640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 0x7f0a40071080 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0a30013b20 tx=0x7f0a30009f80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a24038e10 0x7f0a2403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 client.19384.objecter ms_handle_connect 0x7f0a40138310 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 client.19384.objecter resend_mon_ops 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.7 v2:172.21.15.161:3303/0 5 ==== osd_map(217..217 src has 1..217) ==== 6253+0+0 (secure 0 0 0) 0x7f0a3c049860 con 0x7f0a40132e70 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 client.19384.objecter ms_dispatch 0x7f0a40000ba0 osd_map(217..217 src has 1..217) 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 3 client.19384.objecter handle_osd_map got epochs [217,217] > 0 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 3 client.19384.objecter handle_osd_map decoding full epoch 217 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 20 client.19384.objecter dump_active .. 0 homeless 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3001b070 con 0x7f0a40138310 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:14.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: dump: 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.057 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a30011d60 con 0x7f0a40138310 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 msgr2=0x7f0a40071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 0x7f0a40071080 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0a30013b20 tx=0x7f0a30009f80 comp rx=0 tx=0).stop 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 msgr2=0x7f0a40071080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a367fc640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a40138310 0x7f0a40071080 secure :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f0a30013b20 tx=0x7f0a30009f80 comp rx=0 tx=0).stop 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a24038e10 0x7f0a2403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.029+0000 7f0a4670b640 10 monclient: get_auth_request con 0x7f0a24038e10 auth_method 0 2024-04-19T00:49:14.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.033+0000 7f0a4670b640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a24038e10 0x7f0a2403b2d0 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f0a380ad000 tx=0x7f0a38001ff0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.033+0000 7f0a367fc640 10 client.19384.objecter ms_handle_connect 0x7f0a24038e10 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 0x7f0a400719d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a40068240 con 0x7f0a400715c0 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 0x7f0a400719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient: get_auth_request con 0x7f0a400715c0 auth_method 0 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 10 monclient(hunting): handle_auth_done global_id 22823 payload 824 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 1 -- 172.21.15.43:0/3949677805 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a401386f0 con 0x7f0a400715c0 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 client.19384.objecter ms_handle_connect 0x7f0a400715c0 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 client.19384.objecter resend_mon_ops 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a45709640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 0x7f0a400719d0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0a30013b20 tx=0x7f0a300046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a3000f040 con 0x7f0a400715c0 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 monclient: dump: 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f0a30020930 con 0x7f0a400715c0 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 msgr2=0x7f0a400719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 0x7f0a400719d0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0a30013b20 tx=0x7f0a300046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 msgr2=0x7f0a400719d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a367fc640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a400715c0 0x7f0a400719d0 secure :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f0a30013b20 tx=0x7f0a300046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 297548, 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:14.062 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.063 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.064 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.065 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.066 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.068 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.071 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:14.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.074 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:14.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.077 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.081 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:14.082 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:14.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 10 librados: watch_flush enter 2024-04-19T00:49:14.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 10 librados: watch_flush exit 2024-04-19T00:49:14.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 20 client.19384.objecter shutdown clearing up homeless session... 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 10 client.19384.objecter successfully canceled tick 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a24038e10 msgr2=0x7f0a2403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a24038e10 0x7f0a2403b2d0 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 crypto rx=0x7f0a380ad000 tx=0x7f0a38001ff0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 10 monclient: shutdown 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a40132e70 msgr2=0x7f0a401d0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 --2- 172.21.15.43:0/3949677805 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a40132e70 0x7f0a401d0ac0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c010870 tx=0x7f0a3c010d30 comp rx=0 tx=0).stop 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4670b640 1 -- 172.21.15.43:0/3949677805 reap_dead start 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4670b640 1 -- 172.21.15.43:0/3949677805 reap_dead start 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 shutdown_connections 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 >> 172.21.15.43:0/3949677805 conn(0x7f0a400a77d0 msgr2=0x7f0a400a55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 shutdown_connections 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 -- 172.21.15.43:0/3949677805 wait complete. 2024-04-19T00:49:14.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.037+0000 7f0a4816e640 1 librados: shutdown 2024-04-19T00:49:14.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:14.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:49:14.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:49:14.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient: get_monmap_and_config 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient: monmap: 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:14.203875+0000 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:14.203875+0000 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.202 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:14.203 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 Processor -- start 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 -- start start 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient: init 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient: _add_conns ranks=[9,7,10,8,5,1,2,4,3,0,6] 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fea8c0a1e00 0x7fea8c0a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): picked mon.noname-a con 0x7fea8c0a1e00 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:14.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fea8c0a1190 0x7fea8c0a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): picked mon.noname-i con 0x7fea8c0a1190 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fea8c0a4350 0x7fea8c13fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): picked mon.noname-f con 0x7fea8c0a4350 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fea8c06c690 con 0x7fea8c0a1e00 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fea8c0735a0 con 0x7fea8c0a4350 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fea8c09fff0 con 0x7fea8c0a1190 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.201+0000 7fea91881640 10 monclient(hunting): authenticate will time out at 1224.331894s 2024-04-19T00:49:14.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fea8c0a1e00 0x7fea8c0a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fea8c0a1e00 0x7fea8c0a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35528/0 (socket says 172.21.15.43:35528) 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 -- 172.21.15.43:0/1121809102 learned_addr learned my addr 172.21.15.43:0/1121809102 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): get_auth_request con 0x7fea8c0a1e00 auth_method 0 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8b7fe640 1 --2- 172.21.15.43:0/1121809102 >> v2:172.21.15.161:3300/0 conn(0x7fea8c0a4350 0x7fea8c13fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8b7fe640 10 monclient(hunting): get_auth_request con 0x7fea8c0a4350 auth_method 0 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient(hunting): handle_auth_done global_id 54297 payload 274 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 -- 172.21.15.43:0/1121809102 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fea8c0a1190 msgr2=0x7fea8c0a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 --2- 172.21.15.43:0/1121809102 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fea8c0a1190 0x7fea8c0a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 -- 172.21.15.43:0/1121809102 >> v2:172.21.15.161:3300/0 conn(0x7fea8c0a4350 msgr2=0x7fea8c13fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 --2- 172.21.15.43:0/1121809102 >> v2:172.21.15.161:3300/0 conn(0x7fea8c0a4350 0x7fea8c13fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 monclient: found mon.noname-a 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:14.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 -- 172.21.15.43:0/1121809102 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c06c330 con 0x7fea8c0a1e00 2024-04-19T00:49:14.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8affd640 1 --2- 172.21.15.43:0/1121809102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea8c0a1e00 0x7fea8c0a38d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea8c06c0b0 tx=0x7fea7c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28690df87084e015 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 1 -- 172.21.15.43:0/1121809102 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea7c0b0070 con 0x7fea8c0a1e00 2024-04-19T00:49:14.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: dump: 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.216 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 5 monclient: authenticate success, global_id 54297 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 1 -- 172.21.15.43:0/1121809102 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7c0aa810 con 0x7fea8c0a1e00 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 4 set_mon_vals no callback set 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 1 -- 172.21.15.43:0/1121809102 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea7c0a9300 con 0x7fea8c0a1e00 2024-04-19T00:49:14.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea8a7fc640 10 monclient: dump: 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.218 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 10 monclient: shutdown 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 -- 172.21.15.43:0/1121809102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea8c0a1e00 msgr2=0x7fea8c0a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 --2- 172.21.15.43:0/1121809102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea8c0a1e00 0x7fea8c0a38d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea8c06c0b0 tx=0x7fea7c01c3e0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 -- 172.21.15.43:0/1121809102 shutdown_connections 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 --2- 172.21.15.43:0/1121809102 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fea8c0a1190 0x7fea8c0a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 --2- 172.21.15.43:0/1121809102 >> v2:172.21.15.161:3300/0 conn(0x7fea8c0a4350 0x7fea8c13fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 --2- 172.21.15.43:0/1121809102 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fea8c0a1e00 0x7fea8c0a38d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 -- 172.21.15.43:0/1121809102 >> 172.21.15.43:0/1121809102 conn(0x7fea8c0a84c0 msgr2=0x7fea8c0a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 -- 172.21.15.43:0/1121809102 shutdown_connections 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.205+0000 7fea91881640 1 -- 172.21.15.43:0/1121809102 wait complete. 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: monmap: 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.220 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 librados: starting msgr at 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 librados: starting objecter 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 Processor -- start 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- start start 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 librados: setting wanted keys 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 librados: calling monclient init 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: init 2024-04-19T00:49:14.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: _add_conns ranks=[4,8,6,1,3,7,0,2,5] 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fea8c0a1190 0x7fea8c1d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): picked mon.noname-c con 0x7fea8c0a1190 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fea8c0a1e00 0x7fea8c13aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): picked mon.noname-e con 0x7fea8c0a1e00 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 0x7fea8c122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): picked mon.noname-d con 0x7fea8c0a4350 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fea8c142820 con 0x7fea8c0a1190 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fea8c0a23f0 con 0x7fea8c0a4350 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fea8c1429a0 con 0x7fea8c0a1e00 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): authenticate will time out at 1224.337108s 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fea8c0a1190 0x7fea8c1d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fea8c0a1e00 0x7fea8c13aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 0x7fea8c122920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fea8c0a1190 0x7fea8c1d0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58752/0 (socket says 172.21.15.43:58752) 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 -- 172.21.15.43:0/1436223403 learned_addr learned my addr 172.21.15.43:0/1436223403 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): get_auth_request con 0x7fea8c0a1190 auth_method 0 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): get_auth_request con 0x7fea8c0a1e00 auth_method 0 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): get_auth_request con 0x7fea8c0a4350 auth_method 0 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 msgr2=0x7fea8c122920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 msgr2=0x7fea8c122920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 0x7fea8c122920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient(hunting): handle_auth_done global_id 23044 payload 824 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 0x7fea8c122920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fea8c0a1e00 msgr2=0x7fea8c13aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fea8c0a1e00 0x7fea8c13aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 msgr2=0x7fea8c122920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fea8c0a4350 0x7fea8c122920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 monclient: found mon.noname-c 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c124170 con 0x7fea8c0a1190 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8affd640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fea8c0a1190 0x7fea8c1d0d00 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fea7c01cc80 tx=0x7fea7c019180 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 client.?.objecter ms_handle_connect 0x7fea8c0a1190 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea7c0b0040 con 0x7fea8c0a1190 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: dump: 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.226 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7c0044b0 con 0x7fea8c0a1190 2024-04-19T00:49:14.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 5 monclient: authenticate success, global_id 23044 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: _renew_subs 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8c073270 con 0x7fea8c0a1190 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea7c0aade0 con 0x7fea8c0a1190 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 monclient: dump: 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.228 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea89ffb640 4 set_mon_vals no callback set 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 client.23044.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: _renew_subs 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8c126460 con 0x7fea8c0a1190 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 librados: init done 2024-04-19T00:49:14.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fea7c004040 con 0x7fea8c0a1190 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 0x7fea8c122e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c06d140 con 0x7fea8c13b000 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fea6c038e40 0x7fea6c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(217..217 src has 1..217) ==== 6253+0+0 (secure 0 0 0) 0x7fea7c0e2ec0 con 0x7fea8c0a1190 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 10 client.23044.objecter ms_dispatch 0x7fea8c000ba0 osd_map(217..217 src has 1..217) 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 3 client.23044.objecter handle_osd_map got epochs [217,217] > 0 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 3 client.23044.objecter handle_osd_map decoding full epoch 217 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea88ff9640 20 client.23044.objecter dump_active .. 0 homeless 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 0x7fea8c122e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fea6c038e40 0x7fea6c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient: get_auth_request con 0x7fea8c13b000 auth_method 0 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea8b7fe640 10 monclient: get_auth_request con 0x7fea6c038e40 auth_method 0 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.209+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea827fc640 10 monclient(hunting): handle_auth_done global_id 23932 payload 824 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea827fc640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea8c1d19e0 con 0x7fea8c13b000 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 client.23044.objecter ms_handle_connect 0x7fea6c038e40 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea8b7fe640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fea6c038e40 0x7fea6c03b300 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7fea8c13fe50 tx=0x7fea84009780 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 client.23044.objecter ms_handle_connect 0x7fea8c13b000 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 client.23044.objecter resend_mon_ops 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea827fc640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 0x7fea8c122e60 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e770 tx=0x7fea7400ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea74017070 con 0x7fea8c13b000 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 monclient: dump: 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.232 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fea7400c7a0 con 0x7fea8c13b000 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 msgr2=0x7fea8c122e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 0x7fea8c122e60 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e770 tx=0x7fea7400ec30 comp rx=0 tx=0).stop 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 msgr2=0x7fea8c122e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.213+0000 7fea88ff9640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c13b000 0x7fea8c122e60 secure :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e770 tx=0x7fea7400ec30 comp rx=0 tx=0).stop 2024-04-19T00:49:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 0x7fea8c13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c1237b0 con 0x7fea8c1233a0 2024-04-19T00:49:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 0x7fea8c13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient: get_auth_request con 0x7fea8c1233a0 auth_method 0 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 10 monclient(hunting): handle_auth_done global_id 23941 payload 824 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 1 -- 172.21.15.43:0/1436223403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fea8c073050 con 0x7fea8c1233a0 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 client.23044.objecter ms_handle_connect 0x7fea8c1233a0 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 client.23044.objecter resend_mon_ops 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea827fc640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 0x7fea8c13bcc0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e770 tx=0x7fea740047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fea74017070 con 0x7fea8c1233a0 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 monclient: dump: 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.237 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fea74006390 con 0x7fea8c1233a0 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 msgr2=0x7fea8c13bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 0x7fea8c13bcc0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fea7400e770 tx=0x7fea740047e0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 msgr2=0x7fea8c13bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea88ff9640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fea8c1233a0 0x7fea8c13bcc0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 297690, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:14.239 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:14.242 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:14.243 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:14.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.247 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.249 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.252 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.254 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.257 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.259 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.260 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.261 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:14.262 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 10 librados: watch_flush enter 2024-04-19T00:49:14.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 10 librados: watch_flush exit 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 20 client.23044.objecter shutdown clearing up homeless session... 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 10 client.23044.objecter successfully canceled tick 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fea6c038e40 msgr2=0x7fea6c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fea6c038e40 0x7fea6c03b300 secure :-1 s=READY pgs=3422 cs=0 l=1 rev1=1 crypto rx=0x7fea8c13fe50 tx=0x7fea84009780 comp rx=0 tx=0).stop 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 10 monclient: shutdown 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fea8c0a1190 msgr2=0x7fea8c1d0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 --2- 172.21.15.43:0/1436223403 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fea8c0a1190 0x7fea8c1d0d00 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fea7c01cc80 tx=0x7fea7c019180 comp rx=0 tx=0).stop 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea8b7fe640 1 -- 172.21.15.43:0/1436223403 reap_dead start 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea8b7fe640 1 -- 172.21.15.43:0/1436223403 reap_dead start 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 shutdown_connections 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 >> 172.21.15.43:0/1436223403 conn(0x7fea8c0a84c0 msgr2=0x7fea8c0a6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 shutdown_connections 2024-04-19T00:49:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 -- 172.21.15.43:0/1436223403 wait complete. 2024-04-19T00:49:14.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.221+0000 7fea91881640 1 librados: shutdown 2024-04-19T00:49:14.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:14.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:49:14.265 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient: get_monmap_and_config 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient: monmap: 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:14.392871+0000 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:14.392871+0000 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.391 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:14.392 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 1 Processor -- start 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 1 -- start start 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient: init 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient: _add_conns ranks=[1,8,2,5,6,10,9,7,4,3,0] 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe5c0138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 0x7fe5c013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient(hunting): picked mon.noname-e con 0x7fe5c00a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 0x7fe5c01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.389+0000 7fe5c83f4640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c01397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c00a3dc0 con 0x7fe5c0138cd0 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c00a4410 con 0x7fe5c01397c0 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe5c00a40a0 con 0x7fe5c00a5b80 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 10 monclient(hunting): authenticate will time out at 1224.520064s 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 0x7fe5c013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 0x7fe5c01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 0x7fe5c01407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57224/0 (socket says 172.21.15.43:57224) 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 -- 172.21.15.43:0/2009333447 learned_addr learned my addr 172.21.15.43:0/2009333447 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 0x7fe5c013e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:50990/0 (socket says 172.21.15.43:50990) 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): get_auth_request con 0x7fe5c01397c0 auth_method 0 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): get_auth_request con 0x7fe5c00a5b80 auth_method 0 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient(hunting): handle_auth_done global_id 23159 payload 274 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 -- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 msgr2=0x7fe5c013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 --2- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 0x7fe5c013e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 -- 172.21.15.43:0/2009333447 >> v1:172.21.15.161:6789/0 conn(0x7fe5c0138cd0 legacy=0x7fe5c01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 monclient: found mon.noname-b 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 -- 172.21.15.43:0/2009333447 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c00a2ae0 con 0x7fe5c01397c0 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c598f640 1 --2- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 0x7fe5c013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c6991640 1 --2- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 0x7fe5c01407c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098a0 tx=0x7fe5bc01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b4aa431359a3f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 1 -- 172.21.15.43:0/2009333447 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5bc0b1070 con 0x7fe5c01397c0 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: dump: 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.400 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 1 -- 172.21.15.43:0/2009333447 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc0aa7b0 con 0x7fe5c01397c0 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 5 monclient: authenticate success, global_id 23159 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 1 -- 172.21.15.43:0/2009333447 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5bc0aae50 con 0x7fe5c01397c0 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 4 set_mon_vals no callback set 2024-04-19T00:49:14.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c518e640 10 monclient: dump: 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 10 monclient: shutdown 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 msgr2=0x7fe5c01407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 0x7fe5c01407c0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0098a0 tx=0x7fe5bc01c440 comp rx=0 tx=0).stop 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- 172.21.15.43:0/2009333447 shutdown_connections 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c01397c0 0x7fe5c01407c0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/2009333447 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe5c00a5b80 0x7fe5c013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- 172.21.15.43:0/2009333447 >> 172.21.15.43:0/2009333447 conn(0x7fe5c00ab2f0 msgr2=0x7fe5c00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- 172.21.15.43:0/2009333447 shutdown_connections 2024-04-19T00:49:14.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.393+0000 7fe5c83f4640 1 -- 172.21.15.43:0/2009333447 wait complete. 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: monmap: 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:14.410 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 librados: starting msgr at 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 librados: starting objecter 2024-04-19T00:49:14.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 Processor -- start 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- start start 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 librados: setting wanted keys 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 librados: calling monclient init 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: init 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: _add_conns ranks=[2,7,4,6,5,3,8,1,0] 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c00a5b80 0x7fe5c01d0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): picked mon.noname-b con 0x7fe5c00a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe5c0138cd0 0x7fe5c013bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): picked mon.noname-i con 0x7fe5c0138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe5c01397c0 0x7fe5c013c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): picked mon.noname-c con 0x7fe5c01397c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c0143730 con 0x7fe5c00a5b80 2024-04-19T00:49:14.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe5c00a20a0 con 0x7fe5c01397c0 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe5c01438b0 con 0x7fe5c0138cd0 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): authenticate will time out at 1224.525288s 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c00a5b80 0x7fe5c01d0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe5c01397c0 0x7fe5c013c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c00a5b80 0x7fe5c01d0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57240/0 (socket says 172.21.15.43:57240) 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 1 -- 172.21.15.43:0/228666383 learned_addr learned my addr 172.21.15.43:0/228666383 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): get_auth_request con 0x7fe5c00a5b80 auth_method 0 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): get_auth_request con 0x7fe5c01397c0 auth_method 0 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c598f640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe5c0138cd0 0x7fe5c013bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c598f640 10 monclient(hunting): get_auth_request con 0x7fe5c0138cd0 auth_method 0 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c598f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c598f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c598f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient(hunting): handle_auth_done global_id 23053 payload 824 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe5c0138cd0 msgr2=0x7fe5c013bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe5c0138cd0 0x7fe5c013bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c00a5b80 msgr2=0x7fe5c01d0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c00a5b80 0x7fe5c01d0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 monclient: found mon.noname-c 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c013f940 con 0x7fe5c01397c0 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6190640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe5c00a5b80 0x7fe5c01d0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c6991640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe5c01397c0 0x7fe5c013c1a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0ab000 tx=0x7fe5bc0191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fe5c01397c0 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:14.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5bc0b1040 con 0x7fe5c01397c0 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: dump: 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.416 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc0aadd0 con 0x7fe5c01397c0 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 5 monclient: authenticate success, global_id 23053 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5bc004280 con 0x7fe5c01397c0 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:14.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c498d640 4 set_mon_vals no callback set 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 10 monclient: dump: 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.418 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: _renew_subs 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c00a3590 con 0x7fe5c01397c0 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 client.23053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: _renew_subs 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c013f660 con 0x7fe5c01397c0 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 librados: init done 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 0x7fe5c002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c006d140 con 0x7fe5c01d14d0 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.397+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe5bc0aa7b0 con 0x7fe5c01397c0 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 0x7fe5c002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient: get_auth_request con 0x7fe5c01d14d0 auth_method 0 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe5a4038e40 0x7fe5a403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(217..217 src has 1..217) ==== 6253+0+0 (secure 0 0 0) 0x7fe5bc0e33f0 con 0x7fe5c01397c0 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 client.23053.objecter ms_dispatch 0x7fe5c0000ba0 osd_map(217..217 src has 1..217) 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 3 client.23053.objecter handle_osd_map got epochs [217,217] > 0 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 3 client.23053.objecter handle_osd_map decoding full epoch 217 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 20 client.23053.objecter dump_active .. 0 homeless 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c598f640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe5a4038e40 0x7fe5a403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c598f640 10 monclient: get_auth_request con 0x7fe5a4038e40 auth_method 0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 10 monclient(hunting): handle_auth_done global_id 18840 payload 824 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5c00693b0 con 0x7fe5c01d14d0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 client.23053.objecter ms_handle_connect 0x7fe5c01d14d0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 client.23053.objecter resend_mon_ops 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c6190640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 0x7fe5c002c450 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8002050 tx=0x7fe5b800d520 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 client.23053.objecter ms_handle_connect 0x7fe5a4038e40 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5c598f640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe5a4038e40 0x7fe5a403b300 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009580 tx=0x7fe5b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5b8003980 con 0x7fe5c01d14d0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 monclient: dump: 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.422 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5b8006b60 con 0x7fe5c01d14d0 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 msgr2=0x7fe5c002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 0x7fe5c002c450 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8002050 tx=0x7fe5b800d520 comp rx=0 tx=0).stop 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 msgr2=0x7fe5c002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.401+0000 7fe5b6ffd640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c01d14d0 0x7fe5c002c450 secure :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8002050 tx=0x7fe5b800d520 comp rx=0 tx=0).stop 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 0x7fe5c002c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c0068240 con 0x7fe5c013cea0 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 0x7fe5c002c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient: get_auth_request con 0x7fe5c013cea0 auth_method 0 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 10 monclient(hunting): handle_auth_done global_id 18849 payload 824 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 1 -- 172.21.15.43:0/228666383 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe5c00a01a0 con 0x7fe5c013cea0 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6190640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 0x7fe5c002c990 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8002050 tx=0x7fe5b80014b0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 client.23053.objecter ms_handle_connect 0x7fe5c013cea0 2024-04-19T00:49:14.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 client.23053.objecter resend_mon_ops 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe5b801e580 con 0x7fe5c013cea0 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 monclient: dump: 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.425 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.426 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.426 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe5b801ece0 con 0x7fe5c013cea0 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 msgr2=0x7fe5c002c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 0x7fe5c002c990 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8002050 tx=0x7fe5b80014b0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 msgr2=0x7fe5c002c990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5b6ffd640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe5c013cea0 0x7fe5c002c990 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:14.427 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 297890, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:14.428 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.429 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.430 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:14.431 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.432 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.434 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.436 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.437 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.439 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.447 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:14.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:14.449 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 10 librados: watch_flush enter 2024-04-19T00:49:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 10 librados: watch_flush exit 2024-04-19T00:49:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 20 client.23053.objecter shutdown clearing up homeless session... 2024-04-19T00:49:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 10 client.23053.objecter successfully canceled tick 2024-04-19T00:49:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe5a4038e40 msgr2=0x7fe5a403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe5a4038e40 0x7fe5a403b300 secure :-1 s=READY pgs=3423 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009580 tx=0x7fe5b0008040 comp rx=0 tx=0).stop 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 10 monclient: shutdown 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe5c01397c0 msgr2=0x7fe5c013c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 --2- 172.21.15.43:0/228666383 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe5c01397c0 0x7fe5c013c1a0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0ab000 tx=0x7fe5bc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6991640 1 -- 172.21.15.43:0/228666383 reap_dead start 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c6991640 1 -- 172.21.15.43:0/228666383 reap_dead start 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 shutdown_connections 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 >> 172.21.15.43:0/228666383 conn(0x7fe5c00ab2f0 msgr2=0x7fe5c00a9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 shutdown_connections 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 -- 172.21.15.43:0/228666383 wait complete. 2024-04-19T00:49:14.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.409+0000 7fe5c83f4640 1 librados: shutdown 2024-04-19T00:49:14.453 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:14.453 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:49:14.453 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:49:14.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.565+0000 7f2238d37640 10 monclient: get_monmap_and_config 2024-04-19T00:49:14.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.565+0000 7f2238d37640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient: monmap: 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:14.570433+0000 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:14.570433+0000 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.569 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.570 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.570 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:14.570 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:14.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 Processor -- start 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 -- start start 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient: init 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient: _add_conns ranks=[10,1,4,6,3,2,9,0,7,5,8] 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2234138cd0 0x7f22341390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): picked mon.noname-f con 0x7f2234138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f22340a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f2234135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): picked mon.noname-c con 0x7f22341395f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22340a3dc0 con 0x7f22340a5b80 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f22340a4410 con 0x7f2234138cd0 2024-04-19T00:49:14.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f22340a40a0 con 0x7f22341395f0 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2238d37640 10 monclient(hunting): authenticate will time out at 1224.697494s 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f2234135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f2234135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58770/0 (socket says 172.21.15.43:58770) 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 -- 172.21.15.43:0/2806530699 learned_addr learned my addr 172.21.15.43:0/2806530699 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): get_auth_request con 0x7f22341395f0 auth_method 0 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f223259c640 1 --2- 172.21.15.43:0/2806530699 >> v2:172.21.15.161:3300/0 conn(0x7f2234138cd0 0x7f22341390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f223259c640 10 monclient(hunting): get_auth_request con 0x7f2234138cd0 auth_method 0 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f223259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f223259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f223259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient(hunting): handle_auth_done global_id 23062 payload 274 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 -- 172.21.15.43:0/2806530699 >> v2:172.21.15.161:3300/0 conn(0x7f2234138cd0 msgr2=0x7f22341390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 --2- 172.21.15.43:0/2806530699 >> v2:172.21.15.161:3300/0 conn(0x7f2234138cd0 0x7f22341390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 -- 172.21.15.43:0/2806530699 >> v1:172.21.15.161:6789/0 conn(0x7f22340a5b80 legacy=0x7f22341358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 monclient: found mon.noname-c 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.569+0000 7f2232d9d640 1 -- 172.21.15.43:0/2806530699 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22340a2ae0 con 0x7f22341395f0 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 --2- 172.21.15.43:0/2806530699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f2234135fc0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f22340a7450 tx=0x7f222c01c8b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a63810fcbce8d62d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 1 -- 172.21.15.43:0/2806530699 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0090d0 con 0x7f22341395f0 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: dump: 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.576 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 1 -- 172.21.15.43:0/2806530699 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222c0a5b80 con 0x7f22341395f0 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 1 -- 172.21.15.43:0/2806530699 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0a5d70 con 0x7f22341395f0 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 5 monclient: authenticate success, global_id 23062 2024-04-19T00:49:14.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 4 set_mon_vals no callback set 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223159a640 10 monclient: dump: 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.578 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.579 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: shutdown 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/2806530699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 msgr2=0x7f2234135fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 --2- 172.21.15.43:0/2806530699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f2234135fc0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f22340a7450 tx=0x7f222c01c8b0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/2806530699 shutdown_connections 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 --2- 172.21.15.43:0/2806530699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f2234135fc0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 --2- 172.21.15.43:0/2806530699 >> v2:172.21.15.161:3300/0 conn(0x7f2234138cd0 0x7f22341390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/2806530699 >> 172.21.15.43:0/2806530699 conn(0x7f22340ab2f0 msgr2=0x7f22340ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/2806530699 shutdown_connections 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/2806530699 wait complete. 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: monmap: 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.588 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 librados: starting msgr at 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 librados: starting objecter 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 Processor -- start 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- start start 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 librados: setting wanted keys 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 librados: calling monclient init 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: init 2024-04-19T00:49:14.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: _add_conns ranks=[3,6,4,5,1,8,7,2,0] 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22340a5b80 0x7f22340a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): picked mon.noname-g con 0x7f22340a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2234138cd0 0x7f223412e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): picked mon.noname-d con 0x7f2234138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f223412ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): picked mon.noname-c con 0x7f22341395f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2234136500 con 0x7f22340a5b80 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f22340a20a0 con 0x7f22341395f0 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2234136680 con 0x7f2234138cd0 2024-04-19T00:49:14.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient(hunting): authenticate will time out at 1224.702441s 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f223412ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2234138cd0 0x7f223412e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f223412ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:58772/0 (socket says 172.21.15.43:58772) 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 -- 172.21.15.43:0/47396711 learned_addr learned my addr 172.21.15.43:0/47396711 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): get_auth_request con 0x7f22341395f0 auth_method 0 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): get_auth_request con 0x7f2234138cd0 auth_method 0 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223259c640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22340a5b80 0x7f22340a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223259c640 10 monclient(hunting): get_auth_request con 0x7f22340a5b80 auth_method 0 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f223259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient(hunting): handle_auth_done global_id 23071 payload 824 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2234138cd0 msgr2=0x7f223412e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2234138cd0 0x7f223412e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22340a5b80 msgr2=0x7f22340a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22340a5b80 0x7f22340a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 monclient: found mon.noname-c 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22340a0250 con 0x7f22341395f0 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2231d9b640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2234138cd0 0x7f223412e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2232d9d640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f223412ee10 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f222c009040 tx=0x7f222c019650 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 client.?.objecter ms_handle_connect 0x7f22341395f0 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c0bc020 con 0x7f22341395f0 2024-04-19T00:49:14.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: dump: 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.594 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 5 monclient: authenticate success, global_id 23071 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: _renew_subs 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f222c009210 con 0x7f22341395f0 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f223409eda0 con 0x7f22341395f0 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f222c003bb0 con 0x7f22341395f0 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2230d99640 4 set_mon_vals no callback set 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 client.23071.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2222ffd640 10 monclient: dump: 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.596 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: _renew_subs 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f223409ff70 con 0x7f22341395f0 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.573+0000 7f2238d37640 1 librados: init done 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2238d37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2238d37640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 0x7f223409c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f223406d140 con 0x7f223412f350 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f222c0bf020 con 0x7f22341395f0 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 0x7f223409c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2214038e00 0x7f221403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient: get_auth_request con 0x7f223412f350 auth_method 0 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(217..217 src has 1..217) ==== 6253+0+0 (secure 0 0 0) 0x7f222c0b3040 con 0x7f22341395f0 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 client.23071.objecter ms_dispatch 0x7f2234000ba0 osd_map(217..217 src has 1..217) 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 3 client.23071.objecter handle_osd_map got epochs [217,217] > 0 2024-04-19T00:49:14.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 3 client.23071.objecter handle_osd_map decoding full epoch 217 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 20 client.23071.objecter dump_active .. 0 homeless 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2231d9b640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2214038e00 0x7f221403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2231d9b640 10 monclient: get_auth_request con 0x7f2214038e00 auth_method 0 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 10 monclient(hunting): handle_auth_done global_id 44699 payload 824 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22340a3590 con 0x7f223412f350 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 client.23071.objecter ms_handle_connect 0x7f223412f350 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f223259c640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 0x7f223409c570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f222400f4b0 tx=0x7f222400f970 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 client.23071.objecter resend_mon_ops 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2224019020 con 0x7f223412f350 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:14.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 monclient: dump: 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.600 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2231d9b640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2214038e00 0x7f221403b2c0 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f223409dcd0 tx=0x7f2228007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 client.23071.objecter ms_handle_connect 0x7f2214038e00 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2224014670 con 0x7f223412f350 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 msgr2=0x7f223409c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 0x7f223409c570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f222400f4b0 tx=0x7f222400f970 comp rx=0 tx=0).stop 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 msgr2=0x7f223409c570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.577+0000 7f2222ffd640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223412f350 0x7f223409c570 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 0x7f223409cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f2234068240 con 0x7f223409cab0 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 0x7f223409cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient: get_auth_request con 0x7f223409cab0 auth_method 0 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 10 monclient(hunting): handle_auth_done global_id 44708 payload 824 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 1 -- 172.21.15.43:0/47396711 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f223412f730 con 0x7f223409cab0 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f223259c640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 0x7f223409cec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f222400f4b0 tx=0x7f22240046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 client.23071.objecter ms_handle_connect 0x7f223409cab0 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 client.23071.objecter resend_mon_ops 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2224019040 con 0x7f223409cab0 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 monclient: dump: 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.603 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f2224015490 con 0x7f223409cab0 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 msgr2=0x7f223409cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 0x7f223409cec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f222400f4b0 tx=0x7f22240046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 msgr2=0x7f223409cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2222ffd640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f223409cab0 0x7f223409cec0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f222400f4b0 tx=0x7f22240046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:14.604 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 154316, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:49:14.605 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.606 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:14.608 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.609 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:14.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:14.616 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.619 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.621 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.624 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:14.626 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 10 librados: watch_flush enter 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 10 librados: watch_flush exit 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 20 client.23071.objecter shutdown clearing up homeless session... 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 10 client.23071.objecter successfully canceled tick 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2214038e00 msgr2=0x7f221403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2214038e00 0x7f221403b2c0 secure :-1 s=READY pgs=3424 cs=0 l=1 rev1=1 crypto rx=0x7f223409dcd0 tx=0x7f2228007e40 comp rx=0 tx=0).stop 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 10 monclient: shutdown 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 msgr2=0x7f223412ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 --2- 172.21.15.43:0/47396711 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22341395f0 0x7f223412ee10 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f222c009040 tx=0x7f222c019650 comp rx=0 tx=0).stop 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2232d9d640 1 -- 172.21.15.43:0/47396711 reap_dead start 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2232d9d640 1 -- 172.21.15.43:0/47396711 reap_dead start 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 shutdown_connections 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 >> 172.21.15.43:0/47396711 conn(0x7f22340ab2f0 msgr2=0x7f2234131da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 shutdown_connections 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 -- 172.21.15.43:0/47396711 wait complete. 2024-04-19T00:49:14.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.585+0000 7f2238d37640 1 librados: shutdown 2024-04-19T00:49:14.629 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:14.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:49:14.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient: get_monmap_and_config 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient: monmap: 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:14.742906+0000 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:14.742906+0000 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.741 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:14.742 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 Processor -- start 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 -- start start 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient: init 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient: _add_conns ranks=[0,3,8,6,2,5,4,10,9,1,7] 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8f4138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f40a5b80 0x7fd8f413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): picked mon.noname-g con 0x7fd8f40a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f41397c0 0x7fd8f41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): picked mon.noname-e con 0x7fd8f41397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8f40a3dc0 con 0x7fd8f4138cd0 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f40a4410 con 0x7fd8f40a5b80 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd8f40a40a0 con 0x7fd8f41397c0 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8fabad640 10 monclient(hunting): authenticate will time out at 1224.870561s 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f8949640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd8f4138cd0 0x7fd8f41390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57126/0 (socket says 172.21.15.43:57126) 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f8949640 1 -- 172.21.15.43:0/3479049032 learned_addr learned my addr 172.21.15.43:0/3479049032 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f41397c0 0x7fd8f41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): get_auth_request con 0x7fd8f41397c0 auth_method 0 2024-04-19T00:49:14.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f40a5b80 0x7fd8f413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f3fff640 10 monclient(hunting): get_auth_request con 0x7fd8f40a5b80 auth_method 0 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f37fe640 1 -- 172.21.15.43:0/3479049032 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012533926 0 0) 0x7fd8f40a3dc0 con 0x7fd8f4138cd0 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f37fe640 10 monclient(hunting): _negotiate 2024-04-19T00:49:14.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f37fe640 10 monclient(hunting): my global_id is 54306 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f37fe640 1 -- 172.21.15.43:0/3479049032 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e4003530 con 0x7fd8f4138cd0 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient(hunting): handle_auth_done global_id 22868 payload 274 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 1 -- 172.21.15.43:0/3479049032 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f40a5b80 msgr2=0x7fd8f413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f40a5b80 0x7fd8f413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 1 -- 172.21.15.43:0/3479049032 >> v1:172.21.15.43:6789/0 conn(0x7fd8f4138cd0 legacy=0x7fd8f41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 1 monclient: found mon.noname-e 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:14.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.741+0000 7fd8f914a640 1 -- 172.21.15.43:0/3479049032 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f40a2ae0 con 0x7fd8f41397c0 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f914a640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f41397c0 0x7fd8f41407c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009dc0 tx=0x7fd8ec01c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4fb155d8c8323412 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 1 -- 172.21.15.43:0/3479049032 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8ec0b2070 con 0x7fd8f41397c0 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: dump: 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.755 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 5 monclient: authenticate success, global_id 22868 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 1 -- 172.21.15.43:0/3479049032 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0a5a50 con 0x7fd8f41397c0 2024-04-19T00:49:14.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 1 -- 172.21.15.43:0/3479049032 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8ec0a5c40 con 0x7fd8f41397c0 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 4 set_mon_vals no callback set 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f37fe640 10 monclient: dump: 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.757 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: shutdown 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- 172.21.15.43:0/3479049032 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f41397c0 msgr2=0x7fd8f41407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f41397c0 0x7fd8f41407c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec009dc0 tx=0x7fd8ec01c850 comp rx=0 tx=0).stop 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- 172.21.15.43:0/3479049032 shutdown_connections 2024-04-19T00:49:14.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f41397c0 0x7fd8f41407c0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 --2- 172.21.15.43:0/3479049032 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd8f40a5b80 0x7fd8f413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- 172.21.15.43:0/3479049032 >> 172.21.15.43:0/3479049032 conn(0x7fd8f40ab2f0 msgr2=0x7fd8f40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- 172.21.15.43:0/3479049032 shutdown_connections 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- 172.21.15.43:0/3479049032 wait complete. 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: monmap: 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:14.759 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 librados: starting msgr at 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 librados: starting objecter 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 Processor -- start 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- start start 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 librados: setting wanted keys 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 librados: calling monclient init 2024-04-19T00:49:14.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: init 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient: _add_conns ranks=[8,1,4,6,7,0,2,3,5] 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f40a5b80 0x7fd8f41d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): picked mon.noname-e con 0x7fd8f40a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f4138cd0 0x7fd8f41d0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): picked mon.noname-f con 0x7fd8f4138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd8f41397c0 0x7fd8f41d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): picked mon.noname-c con 0x7fd8f41397c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f4143730 con 0x7fd8f4138cd0 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd8f40a20a0 con 0x7fd8f41397c0 2024-04-19T00:49:14.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd8f41438b0 con 0x7fd8f40a5b80 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8fabad640 10 monclient(hunting): authenticate will time out at 1224.875827s 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f40a5b80 0x7fd8f41d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f40a5b80 0x7fd8f41d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:51014/0 (socket says 172.21.15.43:51014) 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 1 -- 172.21.15.43:0/1051971704 learned_addr learned my addr 172.21.15.43:0/1051971704 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 10 monclient(hunting): get_auth_request con 0x7fd8f40a5b80 auth_method 0 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.745+0000 7fd8f8949640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f914a640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd8f41397c0 0x7fd8f41d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f914a640 10 monclient(hunting): get_auth_request con 0x7fd8f41397c0 auth_method 0 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f914a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f914a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f914a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 10 monclient(hunting): handle_auth_done global_id 22877 payload 824 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f4138cd0 0x7fd8f41d0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd8f41397c0 msgr2=0x7fd8f41d12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd8f41397c0 0x7fd8f41d12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f4138cd0 msgr2=0x7fd8f41d0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f4138cd0 0x7fd8f41d0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 monclient: found mon.noname-e 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f40a0650 con 0x7fd8f40a5b80 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient: get_auth_request con 0x7fd8f4138cd0 auth_method 0 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd8f4138cd0 0x7fd8f41d0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f8949640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f40a5b80 0x7fd8f41d0830 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c500 tx=0x7fd8e800c9c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 client.?.objecter ms_handle_connect 0x7fd8f40a5b80 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8e8017070 con 0x7fd8f40a5b80 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: dump: 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.764 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8007070 con 0x7fd8f40a5b80 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8e80076d0 con 0x7fd8f40a5b80 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 monclient: dump: 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.766 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f2ffd640 4 set_mon_vals no callback set 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 5 monclient: authenticate success, global_id 22877 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 monclient: _renew_subs 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f40a0930 con 0x7fd8f40a5b80 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 client.22877.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 monclient: _renew_subs 2024-04-19T00:49:14.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f40a10b0 con 0x7fd8f40a5b80 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 1 librados: init done 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 0x7fd8f4135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd8f406d140 con 0x7fd8f41358b0 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd8e8007260 con 0x7fd8f40a5b80 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd8d8038e10 0x7fd8d803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(218..218 src has 1..218) ==== 6154+0+0 (secure 0 0 0) 0x7fd8e8048680 con 0x7fd8f40a5b80 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 10 client.22877.objecter ms_dispatch 0x7fd8f4000ba0 osd_map(218..218 src has 1..218) 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 3 client.22877.objecter handle_osd_map got epochs [218,218] > 0 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 3 client.22877.objecter handle_osd_map decoding full epoch 218 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f17fa640 20 client.22877.objecter dump_active .. 0 homeless 2024-04-19T00:49:14.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 0x7fd8f4135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient: get_auth_request con 0x7fd8f41358b0 auth_method 0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_done global_id 19402 payload 824 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.749+0000 7fd8f3fff640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8f40a3590 con 0x7fd8f41358b0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f914a640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd8d8038e10 0x7fd8d803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f914a640 10 monclient: get_auth_request con 0x7fd8d8038e10 auth_method 0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 0x7fd8f4135c90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e710 tx=0x7fd8e000ebd0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f17fa640 10 client.22877.objecter ms_handle_connect 0x7fd8f41358b0 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f17fa640 10 client.22877.objecter resend_mon_ops 2024-04-19T00:49:14.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f17fa640 10 client.22877.objecter ms_handle_connect 0x7fd8d8038e10 2024-04-19T00:49:14.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.753+0000 7fd8f914a640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd8d8038e10 0x7fd8d803b2d0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0af000 tx=0x7fd8ec0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8e0017070 con 0x7fd8f41358b0 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 10 monclient: dump: 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.772 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd8e000ed00 con 0x7fd8f41358b0 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 msgr2=0x7fd8f4135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 0x7fd8f4135c90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e710 tx=0x7fd8e000ebd0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 msgr2=0x7fd8f4135c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.757+0000 7fd8f17fa640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41358b0 0x7fd8f4135c90 secure :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e710 tx=0x7fd8e000ebd0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.761+0000 7fd8fabad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.761+0000 7fd8fabad640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 0x7fd8f413bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.761+0000 7fd8fabad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.761+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd8f4068240 con 0x7fd8f41361d0 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 0x7fd8f413bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient: get_auth_request con 0x7fd8f41361d0 auth_method 0 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 10 monclient(hunting): handle_auth_done global_id 19411 payload 824 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 1 -- 172.21.15.43:0/1051971704 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd8f40a3040 con 0x7fd8f41361d0 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f3fff640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 0x7fd8f413bdb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e710 tx=0x7fd8e0004740 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 client.22877.objecter ms_handle_connect 0x7fd8f41361d0 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 client.22877.objecter resend_mon_ops 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd8e000f040 con 0x7fd8f41361d0 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 monclient: dump: 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.775 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fd8e0013410 con 0x7fd8f41361d0 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 msgr2=0x7fd8f413bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 0x7fd8f413bdb0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e710 tx=0x7fd8e0004740 comp rx=0 tx=0).stop 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 msgr2=0x7fd8f413bdb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f17fa640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd8f41361d0 0x7fd8f413bdb0 secure :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fd8e000e710 tx=0x7fd8e0004740 comp rx=0 tx=0).stop 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:14.777 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 34, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 298222, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.778 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.779 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.780 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:14.781 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:14.782 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.784 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:14.786 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.789 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.790 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:14.791 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.794 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:14.796 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:14.797 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:14.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:14.799 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:14.800 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:14.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 10 librados: watch_flush enter 2024-04-19T00:49:14.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 10 librados: watch_flush exit 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 20 client.22877.objecter shutdown clearing up homeless session... 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 10 client.22877.objecter successfully canceled tick 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd8d8038e10 msgr2=0x7fd8d803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd8d8038e10 0x7fd8d803b2d0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0af000 tx=0x7fd8ec0a7070 comp rx=0 tx=0).stop 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 10 monclient: shutdown 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f40a5b80 msgr2=0x7fd8f41d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 --2- 172.21.15.43:0/1051971704 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd8f40a5b80 0x7fd8f41d0830 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800c500 tx=0x7fd8e800c9c0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f914a640 1 -- 172.21.15.43:0/1051971704 reap_dead start 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8f914a640 1 -- 172.21.15.43:0/1051971704 reap_dead start 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 shutdown_connections 2024-04-19T00:49:14.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 >> 172.21.15.43:0/1051971704 conn(0x7fd8f40ab2f0 msgr2=0x7fd8f4137580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 shutdown_connections 2024-04-19T00:49:14.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 -- 172.21.15.43:0/1051971704 wait complete. 2024-04-19T00:49:14.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.765+0000 7fd8fabad640 1 librados: shutdown 2024-04-19T00:49:14.803 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:49:14.803 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c', 'd', 'f', 'a'] 2024-04-19T00:49:14.803 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['e'] 2024-04-19T00:49:14.803 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-19T00:49:14.803 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-04-19T00:49:14.804 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-04-19T00:49:14.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.925+0000 7f0bab0e5640 10 monclient: get_monmap_and_config 2024-04-19T00:49:14.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.925+0000 7f0bab0e5640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.925+0000 7f0bab0e5640 10 monclient: monmap: 2024-04-19T00:49:14.927 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.927 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:14.929341+0000 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:14.929341+0000 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.928 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:14.929 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:14.929 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.925+0000 7f0bab0e5640 1 Processor -- start 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.925+0000 7f0bab0e5640 1 -- start start 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.925+0000 7f0bab0e5640 10 monclient: init 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient: _add_conns ranks=[10,2,9,7,6,5,1,3,4,0,8] 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0ba4138d40 0x7f0ba40a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): picked mon.noname-f con 0x7f0ba4138d40 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 0x7f0ba4139560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba4139120 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:14.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 0x7f0ba4135d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba41358b0 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba40a2720 con 0x7f0ba41358b0 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba40a3e30 con 0x7f0ba4138d40 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba40a2b50 con 0x7f0ba4139120 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient(hunting): authenticate will time out at 1225.056361s 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 0x7f0ba4135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 0x7f0ba4139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 0x7f0ba4135d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35542/0 (socket says 172.21.15.43:35542) 2024-04-19T00:49:14.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 0x7f0ba4139560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57256/0 (socket says 172.21.15.43:57256) 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 1 -- 172.21.15.43:0/1417002038 learned_addr learned my addr 172.21.15.43:0/1417002038 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): get_auth_request con 0x7f0ba41358b0 auth_method 0 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): get_auth_request con 0x7f0ba4139120 auth_method 0 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba8e81640 1 --2- 172.21.15.43:0/1417002038 >> v2:172.21.15.161:3300/0 conn(0x7f0ba4138d40 0x7f0ba40a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba8e81640 10 monclient(hunting): get_auth_request con 0x7f0ba4138d40 auth_method 0 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba8e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba8e81640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba8e81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_done global_id 23186 payload 274 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1417002038 >> v2:172.21.15.161:3300/0 conn(0x7f0ba4138d40 msgr2=0x7f0ba40a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 --2- 172.21.15.43:0/1417002038 >> v2:172.21.15.161:3300/0 conn(0x7f0ba4138d40 0x7f0ba40a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1417002038 >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 msgr2=0x7f0ba4135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 --2- 172.21.15.43:0/1417002038 >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 0x7f0ba4135d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 monclient: found mon.noname-b 2024-04-19T00:49:14.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1417002038 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba40a2020 con 0x7f0ba4139120 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba9682640 1 --2- 172.21.15.43:0/1417002038 >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 0x7f0ba4135d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba3fff640 1 --2- 172.21.15.43:0/1417002038 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 0x7f0ba4139560 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40ac140 tx=0x7f0b9401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52d4f42eea83fe26 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 1 -- 172.21.15.43:0/1417002038 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b940ac070 con 0x7f0ba4139120 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: dump: 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.943 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 1 -- 172.21.15.43:0/1417002038 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b940a68f0 con 0x7f0ba4139120 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 1 -- 172.21.15.43:0/1417002038 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b940a7460 con 0x7f0ba4139120 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 5 monclient: authenticate success, global_id 23186 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 4 set_mon_vals no callback set 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0ba37fe640 10 monclient: dump: 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.945 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 10 monclient: shutdown 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1417002038 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 msgr2=0x7f0ba4139560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1417002038 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 0x7f0ba4139560 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40ac140 tx=0x7f0b9401c7f0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1417002038 shutdown_connections 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1417002038 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ba4139120 0x7f0ba4139560 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1417002038 >> v2:172.21.15.43:3300/0 conn(0x7f0ba41358b0 0x7f0ba4135d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1417002038 >> v2:172.21.15.161:3300/0 conn(0x7f0ba4138d40 0x7f0ba40a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1417002038 >> 172.21.15.43:0/1417002038 conn(0x7f0ba40ab2f0 msgr2=0x7f0ba40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1417002038 shutdown_connections 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.929+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1417002038 wait complete. 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: build_initial_monmap 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: monmap: 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:14.947 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:14.948 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 librados: starting msgr at 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 librados: starting objecter 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 Processor -- start 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- start start 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 librados: setting wanted keys 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 librados: calling monclient init 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: init 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: _add_conns ranks=[1,0,5,4,8,3,7,6,2] 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ba41358b0 0x7f0ba412ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): picked mon.noname-f con 0x7f0ba41358b0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ba4138d40 0x7f0ba412bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba4138d40 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:14.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ba4139120 0x7f0ba4133550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): picked mon.noname-h con 0x7f0ba4139120 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba4142c20 con 0x7f0ba4138d40 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba4142aa0 con 0x7f0ba41358b0 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0ba4142da0 con 0x7f0ba4139120 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): authenticate will time out at 1225.061473s 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ba4138d40 0x7f0ba412bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ba4138d40 0x7f0ba412bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:35544/0 (socket says 172.21.15.43:35544) 2024-04-19T00:49:14.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1276403982 learned_addr learned my addr 172.21.15.43:0/1276403982 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): get_auth_request con 0x7f0ba4138d40 auth_method 0 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ba41358b0 0x7f0ba412ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): get_auth_request con 0x7f0ba41358b0 auth_method 0 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ba4139120 0x7f0ba4133550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 10 monclient(hunting): get_auth_request con 0x7f0ba4139120 auth_method 0 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient(hunting): handle_auth_done global_id 54324 payload 824 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ba4139120 msgr2=0x7f0ba4133550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ba4139120 0x7f0ba4133550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ba41358b0 msgr2=0x7f0ba412ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ba41358b0 0x7f0ba412ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 monclient: found mon.noname-a 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:49:14.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba409ce90 con 0x7f0ba4138d40 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba3fff640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ba4138d40 0x7f0ba412bfb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b9401aea0 tx=0x7f0b94019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 client.?.objecter ms_handle_connect 0x7f0ba4138d40 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b940ac050 con 0x7f0ba4138d40 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: dump: 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.953 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: _finish_auth 0 2024-04-19T00:49:14.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: _check_auth_tickets 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b94003e10 con 0x7f0ba4138d40 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 5 monclient: authenticate success, global_id 54324 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: _renew_subs 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba409d170 con 0x7f0ba4138d40 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b940044b0 con 0x7f0ba4138d40 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 monclient: dump: 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.955 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.956 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba2ffd640 4 set_mon_vals no callback set 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 client.54324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: _renew_subs 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba412c4f0 con 0x7f0ba4138d40 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 librados: init done 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 0x7f0ba4133a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0ba4074f70 con 0x7f0ba409bc70 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0b940af070 con 0x7f0ba4138d40 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 0x7f0ba4133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient: get_auth_request con 0x7f0ba409bc70 auth_method 0 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b88038e10 0x7f0b8803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 6154+0+0 (secure 0 0 0) 0x7f0b940b7070 con 0x7f0ba4138d40 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 client.54324.objecter ms_dispatch 0x7f0ba4000ba0 osd_map(218..218 src has 1..218) 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 3 client.54324.objecter handle_osd_map got epochs [218,218] > 0 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 3 client.54324.objecter handle_osd_map decoding full epoch 218 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 20 client.54324.objecter dump_active .. 0 homeless 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_done global_id 23098 payload 824 2024-04-19T00:49:14.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ba40693b0 con 0x7f0ba409bc70 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b88038e10 0x7f0b8803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 client.54324.objecter ms_handle_connect 0x7f0ba409bc70 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba17fa640 10 client.54324.objecter resend_mon_ops 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba8e81640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 0x7f0ba4133a90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00f440 tx=0x7f0b8c00f900 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.933+0000 7f0ba9682640 10 monclient: get_auth_request con 0x7f0b88038e10 auth_method 0 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b8c019030 con 0x7f0ba409bc70 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 10 monclient: dump: 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.959 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0b8c014670 con 0x7f0ba409bc70 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:14.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:14.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 msgr2=0x7f0ba4133a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 0x7f0ba4133a90 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00f440 tx=0x7f0b8c00f900 comp rx=0 tx=0).stop 2024-04-19T00:49:14.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 msgr2=0x7f0ba4133a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba409bc70 0x7f0ba4133a90 secure :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00f440 tx=0x7f0b8c00f900 comp rx=0 tx=0).stop 2024-04-19T00:49:14.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba17fa640 10 client.54324.objecter ms_handle_connect 0x7f0b88038e10 2024-04-19T00:49:14.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.937+0000 7f0ba9682640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b88038e10 0x7f0b8803b2d0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009630 tx=0x7f0b9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0bab0e5640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:49:14.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 0x7f0ba4133fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0bab0e5640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0ba406d140 con 0x7f0ba40a1310 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 0x7f0ba4133fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient: get_auth_request con 0x7f0ba40a1310 auth_method 0 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 10 monclient(hunting): handle_auth_done global_id 23107 payload 824 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 1 -- 172.21.15.43:0/1276403982 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f0ba409d370 con 0x7f0ba40a1310 2024-04-19T00:49:14.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba8e81640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 0x7f0ba4133fd0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00f440 tx=0x7f0b8c0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba17fa640 10 client.54324.objecter ms_handle_connect 0x7f0ba40a1310 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.941+0000 7f0ba17fa640 10 client.54324.objecter resend_mon_ops 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0b8c01f240 con 0x7f0ba40a1310 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 10 monclient: dump: 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:14.964 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:14.965 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:49:14.966 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:49:14.966 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f0b8c01f240 con 0x7f0ba40a1310 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 msgr2=0x7f0ba4133fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 0x7f0ba4133fd0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00f440 tx=0x7f0b8c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 msgr2=0x7f0ba4133fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba17fa640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0ba40a1310 0x7f0ba4133fd0 secure :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00f440 tx=0x7f0b8c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 10 librados: watch_flush enter 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 10 librados: watch_flush exit 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 20 client.54324.objecter shutdown clearing up homeless session... 2024-04-19T00:49:14.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 10 client.54324.objecter successfully canceled tick 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b88038e10 msgr2=0x7f0b8803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0b88038e10 0x7f0b8803b2d0 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c009630 tx=0x7f0b9c008040 comp rx=0 tx=0).stop 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 10 monclient: shutdown 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ba4138d40 msgr2=0x7f0ba412bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 --2- 172.21.15.43:0/1276403982 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ba4138d40 0x7f0ba412bfb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b9401aea0 tx=0x7f0b94019590 comp rx=0 tx=0).stop 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba9682640 1 -- 172.21.15.43:0/1276403982 reap_dead start 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0ba9682640 1 -- 172.21.15.43:0/1276403982 reap_dead start 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 shutdown_connections 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 >> 172.21.15.43:0/1276403982 conn(0x7f0ba40ab2f0 msgr2=0x7f0ba40a8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 shutdown_connections 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 -- 172.21.15.43:0/1276403982 wait complete. 2024-04-19T00:49:14.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:14.945+0000 7f0bab0e5640 1 librados: shutdown 2024-04-19T00:49:14.970 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T00:49:14.970 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T00:49:14.970 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:49:14.983 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T00:49:14.983 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-04-19T00:49:14.983 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d store 2024-04-19T00:49:14.983 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d sync_force --yes-i-really-mean-it 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient: get_monmap_and_config 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient: build_initial_monmap 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient: monmap: 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:15.143965+0000 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:15.143965+0000 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.142 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:15.143 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 Processor -- start 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 -- start start 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient: init 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient: _add_conns ranks=[8,6,1,4,3,2,0,7,9,5,10] 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 0x7f2b400a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): picked mon.noname-e con 0x7f2b400a1cf0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 0x7f2b400a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): picked mon.noname-d con 0x7f2b4006c350 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:15.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2b400a3f60 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b400a1ad0 con 0x7f2b400a3f60 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2b40073880 con 0x7f2b4006c350 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f2b400a0ac0 con 0x7f2b400a1cf0 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b46ab8640 10 monclient(hunting): authenticate will time out at 1225.271297s 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 0x7f2b400a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 0x7f2b400a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 0x7f2b400a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:51016/0 (socket says 172.21.15.43:51016) 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 1 -- 172.21.15.43:0/1829933522 learned_addr learned my addr 172.21.15.43:0/1829933522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): get_auth_request con 0x7f2b400a1cf0 auth_method 0 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 0x7f2b400a3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33890/0 (socket says 172.21.15.43:33890) 2024-04-19T00:49:15.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): get_auth_request con 0x7f2b4006c350 auth_method 0 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:15.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient(hunting): handle_auth_done global_id 23208 payload 274 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 -- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 msgr2=0x7f2b400a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 0x7f2b400a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 -- 172.21.15.43:0/1829933522 >> v1:172.21.15.161:6789/0 conn(0x7f2b400a3f60 legacy=0x7f2b4013fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 monclient: found mon.noname-d 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b3ffff640 1 -- 172.21.15.43:0/1829933522 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b400a0220 con 0x7f2b4006c350 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.141+0000 7f2b44854640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 0x7f2b400a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3ffff640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 0x7f2b400a3a20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f2b280097f0 tx=0x7f2b2801c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1bd29e3951d98750 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 1 -- 172.21.15.43:0/1829933522 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b280b0070 con 0x7f2b4006c350 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:15.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: dump: 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:15.152 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 5 monclient: authenticate success, global_id 23208 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 1 -- 172.21.15.43:0/1829933522 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b280aa810 con 0x7f2b4006c350 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 4 set_mon_vals no callback set 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 1 -- 172.21.15.43:0/1829933522 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b280a9300 con 0x7f2b4006c350 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:15.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b3f7fe640 10 monclient: dump: 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:15.154 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 10 monclient: shutdown 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 -- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 msgr2=0x7f2b400a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 0x7f2b400a3a20 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f2b280097f0 tx=0x7f2b2801c3a0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 -- 172.21.15.43:0/1829933522 shutdown_connections 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4006c350 0x7f2b400a3a20 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/1829933522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b400a1cf0 0x7f2b400a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 -- 172.21.15.43:0/1829933522 >> 172.21.15.43:0/1829933522 conn(0x7f2b400a5630 msgr2=0x7f2b4006d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 -- 172.21.15.43:0/1829933522 shutdown_connections 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 -- 172.21.15.43:0/1829933522 wait complete. 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 10 monclient: build_initial_monmap 2024-04-19T00:49:15.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 10 monclient: monmap: 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:15.156 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 librados: starting msgr at 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 librados: starting objecter 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 Processor -- start 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 -- start start 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 librados: setting wanted keys 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 1 librados: calling monclient init 2024-04-19T00:49:15.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.145+0000 7f2b46ab8640 10 monclient: init 2024-04-19T00:49:15.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:15.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient: _add_conns ranks=[1,7,2,8,6,3,5,0,4] 2024-04-19T00:49:15.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b4006c350 0x7f2b401d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): picked mon.noname-f con 0x7f2b4006c350 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:15.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b400a1cf0 0x7f2b401d1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): picked mon.noname-i con 0x7f2b400a1cf0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b400a3f60 0x7f2b401d1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f2b400a3f60 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2b40143cd0 con 0x7f2b4006c350 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2b40143b50 con 0x7f2b400a3f60 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f2b40143e50 con 0x7f2b400a1cf0 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): authenticate will time out at 1225.276355s 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b400a3f60 0x7f2b401d1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b400a3f60 0x7f2b401d1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57270/0 (socket says 172.21.15.43:57270) 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 -- 172.21.15.43:0/3620343961 learned_addr learned my addr 172.21.15.43:0/3620343961 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:15.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): get_auth_request con 0x7f2b400a3f60 auth_method 0 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b4006c350 0x7f2b401d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): get_auth_request con 0x7f2b4006c350 auth_method 0 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:15.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient(hunting): handle_auth_done global_id 23195 payload 824 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3ffff640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b400a1cf0 0x7f2b401d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b400a1cf0 msgr2=0x7f2b401d1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b400a1cf0 0x7f2b401d1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b4006c350 msgr2=0x7f2b401d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b4006c350 0x7f2b401d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient: _finish_hunting 0 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 monclient: found mon.noname-b 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b401409c0 con 0x7f2b400a3f60 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b45055640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b400a3f60 0x7f2b401d1850 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c390 tx=0x7f2b3800c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 client.?.objecter ms_handle_connect 0x7f2b400a3f60 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b38017070 con 0x7f2b400a3f60 2024-04-19T00:49:15.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: dump: 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:15.168 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b38003600 con 0x7f2b400a3f60 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 5 monclient: authenticate success, global_id 23195 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient: _renew_subs 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:15.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4009f510 con 0x7f2b400a3f60 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b38003c20 con 0x7f2b400a3f60 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3effd640 4 set_mon_vals no callback set 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 client.23195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: dump: 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.170 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient: _renew_subs 2024-04-19T00:49:15.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b40123fc0 con 0x7f2b400a3f60 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 librados: init done 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2b380037b0 con 0x7f2b400a3f60 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 0x7f2b4013a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2b40074f70 con 0x7f2b4013a310 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2b20038e10 0x7f2b2003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 0x7f2b4013a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(218..218 src has 1..218) ==== 6154+0+0 (secure 0 0 0) 0x7f2b38048530 con 0x7f2b400a3f60 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 client.23195.objecter ms_dispatch 0x7f2b40000ba0 osd_map(218..218 src has 1..218) 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 3 client.23195.objecter handle_osd_map got epochs [218,218] > 0 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 3 client.23195.objecter handle_osd_map decoding full epoch 218 2024-04-19T00:49:15.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient: get_auth_request con 0x7f2b4013a310 auth_method 0 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 20 client.23195.objecter dump_active .. 0 homeless 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 10 monclient(hunting): handle_auth_done global_id 23217 payload 824 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b400693b0 con 0x7f2b4013a310 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3ffff640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2b20038e10 0x7f2b2003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3ffff640 10 monclient: get_auth_request con 0x7f2b20038e10 auth_method 0 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 client.23195.objecter ms_handle_connect 0x7f2b4013a310 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 client.23195.objecter resend_mon_ops 2024-04-19T00:49:15.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b44854640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 0x7f2b4013a6f0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007d30 tx=0x7f2b30010590 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b30010ea0 con 0x7f2b4013a310 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: dump: 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:15.174 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b30015440 con 0x7f2b4013a310 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 msgr2=0x7f2b4013a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 0x7f2b4013a6f0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007d30 tx=0x7f2b30010590 comp rx=0 tx=0).stop 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 msgr2=0x7f2b4013a6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:15.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013a310 0x7f2b4013a6f0 secure :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007d30 tx=0x7f2b30010590 comp rx=0 tx=0).stop 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3d7fa640 10 client.23195.objecter ms_handle_connect 0x7f2b20038e10 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.149+0000 7f2b3ffff640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2b20038e10 0x7f2b2003b2d0 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f2b280ad000 tx=0x7f2b280a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b46ab8640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 0x7f2b4013be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b46ab8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2b4006d140 con 0x7f2b4013ac30 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 0x7f2b4013be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient: get_auth_request con 0x7f2b4013ac30 auth_method 0 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:15.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 10 monclient(hunting): handle_auth_done global_id 23226 payload 824 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 1 -- 172.21.15.43:0/3620343961 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f2b400734c0 con 0x7f2b4013ac30 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b44854640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 0x7f2b4013be10 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007d30 tx=0x7f2b30004a50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 client.23195.objecter ms_handle_connect 0x7f2b4013ac30 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 client.23195.objecter resend_mon_ops 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b300167c0 con 0x7f2b4013ac30 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 monclient: dump: 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.177 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f2b30020a10 con 0x7f2b4013ac30 2024-04-19T00:49:15.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:49:15.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:15.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 msgr2=0x7f2b4013be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 0x7f2b4013be10 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f2b30007d30 tx=0x7f2b30004a50 comp rx=0 tx=0).stop 2024-04-19T00:49:15.179 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:15.179 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:49:15.180 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:49:15.180 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:15.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 msgr2=0x7f2b4013be10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:15.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.157+0000 7f2b3d7fa640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b4013ac30 0x7f2b4013be10 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 10 librados: watch_flush enter 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 10 librados: watch_flush exit 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 20 client.23195.objecter shutdown clearing up homeless session... 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 10 client.23195.objecter successfully canceled tick 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2b20038e10 msgr2=0x7f2b2003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2b20038e10 0x7f2b2003b2d0 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f2b280ad000 tx=0x7f2b280a6000 comp rx=0 tx=0).stop 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 10 monclient: shutdown 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b400a3f60 msgr2=0x7f2b401d1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 --2- 172.21.15.43:0/3620343961 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2b400a3f60 0x7f2b401d1850 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800c390 tx=0x7f2b3800c850 comp rx=0 tx=0).stop 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b45055640 1 -- 172.21.15.43:0/3620343961 reap_dead start 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b45055640 1 -- 172.21.15.43:0/3620343961 reap_dead start 2024-04-19T00:49:15.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 shutdown_connections 2024-04-19T00:49:15.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 >> 172.21.15.43:0/3620343961 conn(0x7f2b400a5630 msgr2=0x7f2b400a7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:15.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 shutdown_connections 2024-04-19T00:49:15.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 -- 172.21.15.43:0/3620343961 wait complete. 2024-04-19T00:49:15.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:15.161+0000 7f2b46ab8640 1 librados: shutdown 2024-04-19T00:49:15.184 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T00:49:15.184 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-04-19T00:49:15.185 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:49:15.197 INFO:tasks.ceph.mon.d:Stopped 2024-04-19T00:49:15.198 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-04-19T00:49:15.198 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-04-19T00:49:15.198 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-04-19T00:49:15.199 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:49:15.229 INFO:tasks.ceph.mon.f:Stopped 2024-04-19T00:49:15.229 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-19T00:49:15.229 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T00:49:15.229 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T00:49:15.229 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:49:15.251 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T00:49:15.251 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-04-19T00:49:15.251 INFO:tasks.ceph.mon.e:Sent signal 19 2024-04-19T00:49:15.252 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:49:30.253 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-04-19T00:49:30.254 INFO:tasks.ceph.mon.e:Sent signal 18 2024-04-19T00:49:30.254 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-04-19T00:49:30.254 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:49:30.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient: get_monmap_and_config 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient: build_initial_monmap 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient: monmap: 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:30.388102+0000 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:30.388102+0000 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:30.387 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:30.388 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 Processor -- start 2024-04-19T00:49:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 -- start start 2024-04-19T00:49:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient: init 2024-04-19T00:49:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient: _add_conns ranks=[3,8,1,5,6,0,10,4,7,9,2] 2024-04-19T00:49:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): picked mon.noname-g con 0x7f8ed8138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): picked mon.noname-e con 0x7f8ed80a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8ed8137960 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed80a0d40 con 0x7f8ed8137960 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed80a1020 con 0x7f8ed8138a50 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8ed81398b0 con 0x7f8ed80a5bb0 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8edd91f640 10 monclient(hunting): authenticate will time out at 1240.515224s 2024-04-19T00:49:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed77fe640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f8ed8137960 legacy=0x7f8ed8137dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:50264/0 (socket says 172.21.15.43:50264) 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 learned_addr learned my addr 172.21.15.43:0/2509855522 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ed80a5bb0 auth_method 0 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed8138a50 auth_method 0 2024-04-19T00:49:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.385+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:30.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.161:6789/0 conn(0x7f8ed8137960 legacy=0x7f8ed8137dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ed80a5bb0 auth_method 0 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed8138a50 auth_method 0 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:30.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:49:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:30.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.589+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:30.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.161:6789/0 conn(0x7f8ed8137960 legacy=0x7f8ed8137dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:30.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ed80a5bb0 auth_method 0 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed8138a50 auth_method 0 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:30.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:30.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:49:30.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:30.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:30.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:30.993+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:31.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.161:6789/0 conn(0x7f8ed8137960 legacy=0x7f8ed8137dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ed80a5bb0 auth_method 0 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ed8138a50 auth_method 0 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:31.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:31.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:31.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:31.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:49:31.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:31.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:31.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:31.797+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:33.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): tick 2024-04-19T00:49:33.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:49:33.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 monclient(hunting): continuing hunt 2024-04-19T00:49:33.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:49:33.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 msgr2=0x7f8ed8139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:33.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ed80a5bb0 0x7f8ed8139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:33.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 msgr2=0x7f8ed8138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:33.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ed8138a50 0x7f8ed8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:33.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.161:6789/0 conn(0x7f8ed8137960 legacy=0x7f8ed8137dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient: _add_conns ranks=[5,0,2,10,7,4,6,9] 2024-04-19T00:49:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-h con 0x7f8ec4003810 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ec4006240 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:33.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec4009f10 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:49:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed80a0a60 con 0x7f8ec4006240 2024-04-19T00:49:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed80a1020 con 0x7f8ec4009f10 2024-04-19T00:49:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:33.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8ed81398b0 con 0x7f8ec4003810 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed4ff9640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.43:6789/0 conn(0x7f8ec4006240 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.385+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4009f10 auth_method 0 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:33.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ec4003810 auth_method 0 2024-04-19T00:49:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.389+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:33.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.589+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.43:6789/0 conn(0x7f8ec4006240 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4009f10 auth_method 0 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ec4003810 auth_method 0 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:33.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:33.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.593+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:33.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.993+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.43:6789/0 conn(0x7f8ec4006240 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4009f10 auth_method 0 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ec4003810 auth_method 0 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:33.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:33.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:33.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:33.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:33.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:33.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:33.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:33.997+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:34.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.797+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.43:6789/0 conn(0x7f8ec4006240 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:49:34.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:34.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4009f10 auth_method 0 2024-04-19T00:49:34.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:34.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:34.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:34.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request con 0x7f8ec4003810 auth_method 0 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:34.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:34.801+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:36.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): tick 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 monclient(hunting): continuing hunt 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8ec4003810 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ec4009f10 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.43:6789/0 conn(0x7f8ec4006240 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient: _add_conns ranks=[4,10,7,6,9] 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 reap_dead start 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec400cc30 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:36.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 0x7f8ec4011f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-f con 0x7f8ec400d610 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-i con 0x7f8ec4012540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed80a1020 con 0x7f8ec400d610 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8ed81398b0 con 0x7f8ec400cc30 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8ec4015150 con 0x7f8ec4012540 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.385+0000 7f8ed4ff9640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 reap_dead start 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:36.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 msgr2=0x7f8ec4011f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 0x7f8ec4011f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4012540 auth_method 0 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:36.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:36.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 msgr2=0x7f8ec4011f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 0x7f8ec4011f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4012540 auth_method 0 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:36.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:49:36.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:36.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:36.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.593+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 msgr2=0x7f8ec4011f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 0x7f8ec4011f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4012540 auth_method 0 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:36.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:36.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:36.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:49:36.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:36.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:36.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:36.997+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:37.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:37.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:37.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 msgr2=0x7f8ec4011f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:37.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 0x7f8ec4011f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:37.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4012540 auth_method 0 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:49:37.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:37.801+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:39.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): tick 2024-04-19T00:49:39.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:49:39.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 monclient(hunting): continuing hunt 2024-04-19T00:49:39.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:49:39.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 msgr2=0x7f8ec40149e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:39.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec4012540 0x7f8ec40149e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 msgr2=0x7f8ec4011f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.161:3300/0 conn(0x7f8ec400d610 0x7f8ec4011f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient: _add_conns ranks=[6,9] 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 0x7f8ec400c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-d con 0x7f8ec4008ee0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 0x7f8ec4005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec40092c0 addr v2:172.21.15.43:3300/0 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed80a1020 con 0x7f8ec40092c0 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:39.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8ed81398b0 con 0x7f8ec4008ee0 2024-04-19T00:49:39.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed4ff9640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:39.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:39.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:49:39.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:39.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.389+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.593+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.593+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:49:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.593+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.593+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:49:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.997+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:49:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.997+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.997+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:39.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:39.997+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:49:40.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:40.801+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:49:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:40.801+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:40.801+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:40.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:40.801+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:49:42.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): tick 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 monclient(hunting): continuing hunt 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 msgr2=0x7f8ec400c3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ec4008ee0 0x7f8ec400c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 msgr2=0x7f8ec4005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> v2:172.21.15.43:3300/0 conn(0x7f8ec40092c0 0x7f8ec4005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient: _add_conns ranks=[1,4,8,6,7,5,0,2,3,10,9] 2024-04-19T00:49:42.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 reap_dead start 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8ec4009870 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec400cc30 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ec4016360 0x7f8ec4018750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): picked mon.noname-e con 0x7f8ec4016360 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ed80a0d40 con 0x7f8ec4009870 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8ed80a1020 con 0x7f8ec400cc30 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8ed81398b0 con 0x7f8ec4016360 2024-04-19T00:49:42.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed4ff9640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed67fc640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed6ffd640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.161:6789/0 conn(0x7f8ec4009870 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ec4016360 0x7f8ec4018750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4016360 auth_method 0 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_done global_id 22895 payload 274 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 msgr2=0x7f8ec400d010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 >> v1:172.21.15.161:6789/0 conn(0x7f8ec4009870 legacy=0x7f8ec40086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 monclient: found mon.noname-e 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:42.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.389+0000 7f8ed77fe640 1 -- 172.21.15.43:0/2509855522 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed80a0a60 con 0x7f8ec4016360 2024-04-19T00:49:42.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.393+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ec4016360 0x7f8ec4018750 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80a71e0 tx=0x7f8ed001dbc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=21c346502f6de90e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.393+0000 7f8ed5ffb640 1 -- 172.21.15.43:0/2509855522 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00ad8f0 con 0x7f8ec4016360 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: dump: 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.397 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: _finish_auth 0 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 1 -- 172.21.15.43:0/2509855522 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed00adab0 con 0x7f8ec4016360 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 5 monclient: authenticate success, global_id 22895 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:42.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 1 -- 172.21.15.43:0/2509855522 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00adc60 con 0x7f8ec4016360 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 4 set_mon_vals no callback set 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8ed5ffb640 10 monclient: dump: 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.399 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.400 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 10 monclient: shutdown 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 -- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ec4016360 msgr2=0x7f8ec4018750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ec4016360 0x7f8ec4018750 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80a71e0 tx=0x7f8ed001dbc0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 -- 172.21.15.43:0/2509855522 shutdown_connections 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ec4016360 0x7f8ec4018750 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 --2- 172.21.15.43:0/2509855522 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ec400cc30 0x7f8ec400d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 -- 172.21.15.43:0/2509855522 >> 172.21.15.43:0/2509855522 conn(0x7f8ed80a7cb0 msgr2=0x7f8ed80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 -- 172.21.15.43:0/2509855522 shutdown_connections 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 -- 172.21.15.43:0/2509855522 wait complete. 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 10 monclient: build_initial_monmap 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 10 monclient: monmap: 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:42.407 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 librados: starting msgr at 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 librados: starting objecter 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.397+0000 7f8edd91f640 1 Processor -- start 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 -- start start 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 librados: setting wanted keys 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 librados: calling monclient init 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: init 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: _add_conns ranks=[1,7,3,0,4,5,6,8,2] 2024-04-19T00:49:42.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ec4009870 0x7f8ed8136cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): picked mon.noname-f con 0x7f8ec4009870 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec400cc30 0x7f8ed8137230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): picked mon.noname-i con 0x7f8ec400cc30 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 0x7f8ed81d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): picked mon.noname-g con 0x7f8ec4016360 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed81380d0 con 0x7f8ec4009870 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed8137f50 con 0x7f8ec4016360 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8ed8138250 con 0x7f8ec400cc30 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient(hunting): authenticate will time out at 1252.529030s 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed6ffd640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ec4009870 msgr2=0x7f8ed8136cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:42.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ec4009870 0x7f8ed8136cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec400cc30 0x7f8ed8137230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 0x7f8ed81d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 0x7f8ed81d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:36020/0 (socket says 172.21.15.43:36020) 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 -- 172.21.15.43:0/1686459277 learned_addr learned my addr 172.21.15.43:0/1686459277 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request con 0x7f8ec4016360 auth_method 0 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request con 0x7f8ec400cc30 auth_method 0 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient(hunting): handle_auth_done global_id 18885 payload 824 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 -- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec400cc30 msgr2=0x7f8ed8137230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec400cc30 0x7f8ed8137230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 -- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ec4009870 msgr2=0x7f8ed8136cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ec4009870 0x7f8ed8136cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 monclient: found mon.noname-g 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:49:42.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 -- 172.21.15.43:0/1686459277 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed81243d0 con 0x7f8ec4016360 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed67fc640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec400cc30 0x7f8ed8137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed77fe640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 0x7f8ed81d0830 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00af040 tx=0x7f8ed001aa00 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 client.?.objecter ms_handle_connect 0x7f8ec4016360 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00adc60 con 0x7f8ec4016360 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: dump: 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.412 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: _finish_auth 0 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:42.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed00ade20 con 0x7f8ec4016360 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ed00c0d20 con 0x7f8ec4016360 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ebffff640 10 monclient: dump: 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.414 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 5 monclient: authenticate success, global_id 18885 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: _renew_subs 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed8137770 con 0x7f8ec4016360 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8ed57fa640 4 set_mon_vals no callback set 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 client.18885.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: _renew_subs 2024-04-19T00:49:42.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed809ee60 con 0x7f8ec4016360 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.401+0000 7f8edd91f640 1 librados: init done 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8edd91f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8edd91f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8edd91f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed8123ae0 con 0x7f8ec4016360 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8ed00b4040 con 0x7f8ec4016360 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8ec403b060 0x7f8ec403d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(221..221 src has 1..221) ==== 7009+0+0 (secure 0 0 0) 0x7f8ed00b2020 con 0x7f8ec4016360 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 10 client.18885.objecter ms_dispatch 0x7f8ed8000ba0 osd_map(221..221 src has 1..221) 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 3 client.18885.objecter handle_osd_map got epochs [221,221] > 0 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 3 client.18885.objecter handle_osd_map decoding full epoch 221 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 20 client.18885.objecter dump_active .. 0 homeless 2024-04-19T00:49:42.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8ec403b060 0x7f8ec403d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ed6ffd640 10 monclient: get_auth_request con 0x7f8ec403b060 auth_method 0 2024-04-19T00:49:42.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ed6ffd640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8ec403b060 0x7f8ec403d520 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8006e90 tx=0x7f8ec8006de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.405+0000 7f8ebffff640 10 client.18885.objecter ms_handle_connect 0x7f8ec403b060 2024-04-19T00:49:42.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.417+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8ed0017ca0 con 0x7f8ec4016360 2024-04-19T00:49:42.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.417+0000 7f8ebffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:42.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.417+0000 7f8ebffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:42.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:49:42.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:49:42.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:42.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8ed8123ae0 con 0x7f8ec4016360 2024-04-19T00:49:42.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8ebffff640 1 -- 172.21.15.43:0/1686459277 <== mon.3 v2:172.21.15.161:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3168 (secure 0 0 0) 0x7f8ed00ac3e0 con 0x7f8ec4016360 2024-04-19T00:49:42.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8ebffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:49:42.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8ebffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:42.662 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":42,"quorum":[2,3,5,7,8],"quorum_names":["b","g","h","i","e"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:49:42.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 10 librados: watch_flush enter 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 10 librados: watch_flush exit 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 20 client.18885.objecter shutdown clearing up homeless session... 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.661+0000 7f8edd91f640 10 client.18885.objecter successfully canceled tick 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8ec403b060 msgr2=0x7f8ec403d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8ec403b060 0x7f8ec403d520 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8006e90 tx=0x7f8ec8006de0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 10 monclient: shutdown 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 msgr2=0x7f8ed81d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 0x7f8ed81d0830 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00af040 tx=0x7f8ed001aa00 comp rx=0 tx=0).stop 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 shutdown_connections 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8ec403b060 0x7f8ec403d520 unknown :-1 s=CLOSED pgs=3429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8ec4016360 0x7f8ed81d0830 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8ec400cc30 0x7f8ed8137230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 --2- 172.21.15.43:0/1686459277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ec4009870 0x7f8ed8136cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 >> 172.21.15.43:0/1686459277 conn(0x7f8ed80a7cb0 msgr2=0x7f8ed81c0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:42.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 shutdown_connections 2024-04-19T00:49:42.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 -- 172.21.15.43:0/1686459277 wait complete. 2024-04-19T00:49:42.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.665+0000 7f8edd91f640 1 librados: shutdown 2024-04-19T00:49:42.682 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-04-19T00:49:42.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-04-19T00:49:42.683 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:49:42.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient: get_monmap_and_config 2024-04-19T00:49:42.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient: build_initial_monmap 2024-04-19T00:49:42.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient: monmap: 2024-04-19T00:49:42.805 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:42.807406+0000 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:42.807406+0000 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:42.806 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:42.807 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:42.807 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:42.807 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:42.807 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:42.807 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 Processor -- start 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 -- start start 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient: init 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient: _add_conns ranks=[10,5,4,2,3,9,0,7,6,1,8] 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f337c138cd0 0x7f337c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): picked mon.noname-f con 0x7f337c138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 0x7f337c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): picked mon.noname-h con 0x7f337c0a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f337c1358b0 0x7f337c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): picked mon.noname-c con 0x7f337c1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f337c0a4410 con 0x7f337c138cd0 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f337c139b30 con 0x7f337c1358b0 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f337c0a3dc0 con 0x7f337c0a5b80 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3380d7f640 10 monclient(hunting): authenticate will time out at 1252.934536s 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f337ad9d640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f337c1358b0 msgr2=0x7f337c135d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f337ad9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f337c1358b0 0x7f337c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:42.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f337a59c640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7f337c138cd0 msgr2=0x7f337c1390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f337a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f337c138cd0 0x7f337c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 0x7f337c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 0x7f337c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40186/0 (socket says 172.21.15.43:40186) 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 -- 172.21.15.43:0/3762769325 learned_addr learned my addr 172.21.15.43:0/3762769325 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): get_auth_request con 0x7f337c0a5b80 auth_method 0 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient(hunting): handle_auth_done global_id 44726 payload 274 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 -- 172.21.15.43:0/3762769325 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f337c1358b0 msgr2=0x7f337c135d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:42.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 --2- 172.21.15.43:0/3762769325 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f337c1358b0 0x7f337c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 -- 172.21.15.43:0/3762769325 >> v2:172.21.15.161:3300/0 conn(0x7f337c138cd0 msgr2=0x7f337c1390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 --2- 172.21.15.43:0/3762769325 >> v2:172.21.15.161:3300/0 conn(0x7f337c138cd0 0x7f337c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient: _finish_hunting 0 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 monclient: found mon.noname-h 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:49:42.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.805+0000 7f3373fff640 1 -- 172.21.15.43:0/3762769325 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c0a40a0 con 0x7f337c0a5b80 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3373fff640 1 --2- 172.21.15.43:0/3762769325 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 0x7f337c1395f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f336c009870 tx=0x7f336c01c400 comp rx=0 tx=0).ready entity=mon.5 client_cookie=b137a181b1bcb6e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 1 -- 172.21.15.43:0/3762769325 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f336c0b2070 con 0x7f337c0a5b80 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: dump: 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.819 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: _finish_auth 0 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 1 -- 172.21.15.43:0/3762769325 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f336c0ab7f0 con 0x7f337c0a5b80 2024-04-19T00:49:42.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 5 monclient: authenticate success, global_id 44726 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 4 set_mon_vals no callback set 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 1 -- 172.21.15.43:0/3762769325 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f336c0ab9e0 con 0x7f337c0a5b80 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3379d9b640 10 monclient: dump: 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.821 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 10 monclient: shutdown 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 -- 172.21.15.43:0/3762769325 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 msgr2=0x7f337c1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 --2- 172.21.15.43:0/3762769325 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 0x7f337c1395f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f336c009870 tx=0x7f336c01c400 comp rx=0 tx=0).stop 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 -- 172.21.15.43:0/3762769325 shutdown_connections 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 --2- 172.21.15.43:0/3762769325 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f337c1358b0 0x7f337c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 --2- 172.21.15.43:0/3762769325 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c0a5b80 0x7f337c1395f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 --2- 172.21.15.43:0/3762769325 >> v2:172.21.15.161:3300/0 conn(0x7f337c138cd0 0x7f337c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 -- 172.21.15.43:0/3762769325 >> 172.21.15.43:0/3762769325 conn(0x7f337c0ab2f0 msgr2=0x7f337c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 -- 172.21.15.43:0/3762769325 shutdown_connections 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 1 -- 172.21.15.43:0/3762769325 wait complete. 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.809+0000 7f3380d7f640 10 monclient: build_initial_monmap 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: monmap: 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:42.823 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 librados: starting msgr at 2024-04-19T00:49:42.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 librados: starting objecter 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 Processor -- start 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- start start 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 librados: setting wanted keys 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 librados: calling monclient init 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: init 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: _add_conns ranks=[5,7,3,2,8,1,0,6,4] 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c1358b0 0x7f337c134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): picked mon.noname-h con 0x7f337c1358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f337c138cd0 0x7f337c135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): picked mon.noname-i con 0x7f337c138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f337c1d0c40 0x7f337c1d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): picked mon.noname-g con 0x7f337c1d0c40 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:42.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f337c0a2010 con 0x7f337c1d0c40 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f337c0a1e90 con 0x7f337c1358b0 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f337c142aa0 con 0x7f337c138cd0 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): authenticate will time out at 1252.941022s 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c1358b0 0x7f337c134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f337c138cd0 0x7f337c135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f337c1d0c40 0x7f337c1d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c1358b0 0x7f337c134bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40194/0 (socket says 172.21.15.43:40194) 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 -- 172.21.15.43:0/1398016268 learned_addr learned my addr 172.21.15.43:0/1398016268 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:42.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): get_auth_request con 0x7f337c1358b0 auth_method 0 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): get_auth_request con 0x7f337c138cd0 auth_method 0 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): get_auth_request con 0x7f337c1d0c40 auth_method 0 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient(hunting): handle_auth_done global_id 44735 payload 824 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f337c138cd0 msgr2=0x7f337c135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f337c138cd0 0x7f337c135110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f337c1d0c40 msgr2=0x7f337c1d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f337c1d0c40 0x7f337c1d1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient: _finish_hunting 0 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 monclient: found mon.noname-h 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:42.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c0a3590 con 0x7f337c1358b0 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f337c138cd0 0x7f337c135110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337ad9d640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f337c1d0c40 0x7f337c1d1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337a59c640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c1358b0 0x7f337c134bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f336400b530 tx=0x7f336400b9f0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 client.?.objecter ms_handle_connect 0x7f337c1358b0 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3364013020 con 0x7f337c1358b0 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: dump: 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.829 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:42.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3364003bd0 con 0x7f337c1358b0 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 5 monclient: authenticate success, global_id 44735 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: _renew_subs 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337c0a3040 con 0x7f337c1358b0 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3364003dc0 con 0x7f337c1358b0 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 client.44735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3372ffd640 10 monclient: dump: 2024-04-19T00:49:42.831 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.832 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: _renew_subs 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f337c0a0540 con 0x7f337c1358b0 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f337959a640 4 set_mon_vals no callback set 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 librados: init done 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 0x7f337c13bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f337c06d140 con 0x7f337c1d15a0 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 0x7f337c13bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient: get_auth_request con 0x7f337c1d15a0 auth_method 0 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.813+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3373fff640 10 monclient(hunting): handle_auth_done global_id 23204 payload 824 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3373fff640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f337c0693b0 con 0x7f337c1d15a0 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3364016020 con 0x7f337c1358b0 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3373fff640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 0x7f337c13bdd0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f337c0a7450 tx=0x7f336c01aca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f335c038e10 0x7f335c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 client.44735.objecter ms_handle_connect 0x7f337c1d15a0 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 client.44735.objecter resend_mon_ops 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(222..222 src has 1..222) ==== 7009+0+0 (secure 0 0 0) 0x7f3364049330 con 0x7f337c1358b0 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 client.44735.objecter ms_dispatch 0x7f337c000ba0 osd_map(222..222 src has 1..222) 2024-04-19T00:49:42.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 3 client.44735.objecter handle_osd_map got epochs [222,222] > 0 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 3 client.44735.objecter handle_osd_map decoding full epoch 222 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 20 client.44735.objecter dump_active .. 0 homeless 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f336c0bc070 con 0x7f337c1d15a0 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 monclient: dump: 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.835 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f336c0036a0 con 0x7f337c1d15a0 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:42.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f337ad9d640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f335c038e10 0x7f335c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 msgr2=0x7f337c13bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 0x7f337c13bdd0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f337c0a7450 tx=0x7f336c01aca0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 msgr2=0x7f337c13bdd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c1d15a0 0x7f337c13bdd0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f337ad9d640 10 monclient: get_auth_request con 0x7f335c038e10 auth_method 0 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f3372ffd640 10 client.44735.objecter ms_handle_connect 0x7f335c038e10 2024-04-19T00:49:42.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.817+0000 7f337ad9d640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f335c038e10 0x7f335c03b2d0 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7f3374003d10 tx=0x7f337400d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3380d7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3380d7f640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 0x7f337c13c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3380d7f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f337c068240 con 0x7f337c13c310 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 0x7f337c13c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient: get_auth_request con 0x7f337c13c310 auth_method 0 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:42.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 10 monclient(hunting): handle_auth_done global_id 23213 payload 824 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 1 -- 172.21.15.43:0/1398016268 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f337c0a79d0 con 0x7f337c13c310 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3373fff640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 0x7f337c13c720 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097c0 tx=0x7f336c019200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3372ffd640 10 client.44735.objecter ms_handle_connect 0x7f337c13c310 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3372ffd640 10 client.44735.objecter resend_mon_ops 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f336c0bc040 con 0x7f337c13c310 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3372ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3372ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.821+0000 7f3372ffd640 10 monclient: dump: 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:42.840 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:42.841 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9401 (secure 0 0 0) 0x7f336c0c2e50 con 0x7f337c13c310 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 msgr2=0x7f337c13c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 0x7f337c13c720 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f336c0097c0 tx=0x7f336c019200 comp rx=0 tx=0).stop 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 msgr2=0x7f337c13c720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3372ffd640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f337c13c310 0x7f337c13c720 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:42.842 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 42, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 326320, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:42.843 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:42.844 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:42.846 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:42.847 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.850 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:42.851 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:42.855 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:42.857 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.858 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:42.859 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.860 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:42.861 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:42.863 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:42.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:42.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:42.866 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 10 librados: watch_flush enter 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 10 librados: watch_flush exit 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 20 client.44735.objecter shutdown clearing up homeless session... 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 10 client.44735.objecter successfully canceled tick 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f335c038e10 msgr2=0x7f335c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f335c038e10 0x7f335c03b2d0 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7f3374003d10 tx=0x7f337400d040 comp rx=0 tx=0).stop 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 10 monclient: shutdown 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c1358b0 msgr2=0x7f337c134bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 --2- 172.21.15.43:0/1398016268 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f337c1358b0 0x7f337c134bd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f336400b530 tx=0x7f336400b9f0 comp rx=0 tx=0).stop 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f337ad9d640 1 -- 172.21.15.43:0/1398016268 reap_dead start 2024-04-19T00:49:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f337ad9d640 1 -- 172.21.15.43:0/1398016268 reap_dead start 2024-04-19T00:49:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 shutdown_connections 2024-04-19T00:49:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 >> 172.21.15.43:0/1398016268 conn(0x7f337c0ab2f0 msgr2=0x7f337c137580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 shutdown_connections 2024-04-19T00:49:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 -- 172.21.15.43:0/1398016268 wait complete. 2024-04-19T00:49:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.825+0000 7f3380d7f640 1 librados: shutdown 2024-04-19T00:49:42.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:49:42.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon e 2024-04-19T00:49:42.869 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient: get_monmap_and_config 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient: build_initial_monmap 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient: monmap: 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:42.999040+0000 2024-04-19T00:49:42.997 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:42.999040+0000 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:42.998 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:42.999 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:42.999 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:42.999 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:42.999 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:42.999 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:42.999 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 Processor -- start 2024-04-19T00:49:43.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 -- start start 2024-04-19T00:49:43.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient: init 2024-04-19T00:49:43.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient: _add_conns ranks=[1,6,5,10,7,9,4,8,3,0,2] 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8570138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f857013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): picked mon.noname-d con 0x7f85700a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 0x7f85701407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): picked mon.noname-h con 0x7f85701397c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85700a3dc0 con 0x7f8570138cd0 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f85700a4410 con 0x7f85701397c0 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f85700a40a0 con 0x7f85700a5b80 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8576a69640 10 monclient(hunting): authenticate will time out at 1253.126160s 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f856ffff640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 msgr2=0x7f857013e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f856ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f857013e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8574805640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f8570138cd0 legacy=0x7f85701390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 0x7f85701407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 0x7f85701407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40202/0 (socket says 172.21.15.43:40202) 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 -- 172.21.15.43:0/3041178702 learned_addr learned my addr 172.21.15.43:0/3041178702 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): get_auth_request con 0x7f85701397c0 auth_method 0 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient(hunting): handle_auth_done global_id 44744 payload 274 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 -- 172.21.15.43:0/3041178702 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 msgr2=0x7f857013e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 --2- 172.21.15.43:0/3041178702 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f857013e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 -- 172.21.15.43:0/3041178702 >> v1:172.21.15.161:6789/0 conn(0x7f8570138cd0 legacy=0x7f85701390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 monclient: found mon.noname-h 2024-04-19T00:49:43.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:49:43.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:42.997+0000 7f8575006640 1 -- 172.21.15.43:0/3041178702 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85700a2ae0 con 0x7f85701397c0 2024-04-19T00:49:43.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.001+0000 7f8575006640 1 --2- 172.21.15.43:0/3041178702 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 0x7f85701407c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8560009870 tx=0x7f856001c460 comp rx=0 tx=0).ready entity=mon.5 client_cookie=52ac9f74bb805239 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 1 -- 172.21.15.43:0/3041178702 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85600b1070 con 0x7f85701397c0 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: dump: 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.014 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 5 monclient: authenticate success, global_id 44744 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 1 -- 172.21.15.43:0/3041178702 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85600ab3a0 con 0x7f85701397c0 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 4 set_mon_vals no callback set 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 1 -- 172.21.15.43:0/3041178702 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85700a20a0 con 0x7f85701397c0 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856f7fe640 10 monclient: dump: 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.016 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.017 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: shutdown 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- 172.21.15.43:0/3041178702 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 msgr2=0x7f85701407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 --2- 172.21.15.43:0/3041178702 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 0x7f85701407c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8560009870 tx=0x7f856001c460 comp rx=0 tx=0).stop 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- 172.21.15.43:0/3041178702 shutdown_connections 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 --2- 172.21.15.43:0/3041178702 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f85701397c0 0x7f85701407c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 --2- 172.21.15.43:0/3041178702 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f857013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- 172.21.15.43:0/3041178702 >> 172.21.15.43:0/3041178702 conn(0x7f85700ab2f0 msgr2=0x7f85700ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- 172.21.15.43:0/3041178702 shutdown_connections 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- 172.21.15.43:0/3041178702 wait complete. 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: monmap: 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.019 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 librados: starting msgr at 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 librados: starting objecter 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 Processor -- start 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- start start 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 librados: setting wanted keys 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 librados: calling monclient init 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: init 2024-04-19T00:49:43.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient: _add_conns ranks=[6,5,7,1,2,0,8,3,4] 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f85701358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): picked mon.noname-d con 0x7f85700a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8570138cd0 0x7f8570135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): picked mon.noname-h con 0x7f8570138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 0x7f8570136330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): picked mon.noname-i con 0x7f85701397c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f85701438b0 con 0x7f8570138cd0 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8570143730 con 0x7f85700a5b80 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8570143a30 con 0x7f85701397c0 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8576a69640 10 monclient(hunting): authenticate will time out at 1253.134611s 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8574805640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 msgr2=0x7f85701358b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8574805640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f85701358b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8570138cd0 0x7f8570135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 0x7f8570136330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8570138cd0 0x7f8570135df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40210/0 (socket says 172.21.15.43:40210) 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 0x7f8570136330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:36810/0 (socket says 172.21.15.43:36810) 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 1 -- 172.21.15.43:0/1906316484 learned_addr learned my addr 172.21.15.43:0/1906316484 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): get_auth_request con 0x7f8570138cd0 auth_method 0 2024-04-19T00:49:43.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): get_auth_request con 0x7f85701397c0 auth_method 0 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient(hunting): handle_auth_done global_id 19438 payload 824 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 msgr2=0x7f85701358b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f85700a5b80 0x7f85701358b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8570138cd0 msgr2=0x7f8570135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8570138cd0 0x7f8570135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 monclient: found mon.noname-i 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f8575006640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85700a3590 con 0x7f85701397c0 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.005+0000 7f856ffff640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8570138cd0 0x7f8570135df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8575006640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 0x7f8570136330 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f85600ac000 tx=0x7f8560019200 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 client.?.objecter ms_handle_connect 0x7f85701397c0 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85600b1040 con 0x7f85701397c0 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: dump: 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.025 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8560003cd0 con 0x7f85701397c0 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85600ab9c0 con 0x7f85701397c0 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856effd640 4 set_mon_vals no callback set 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: dump: 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.027 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 5 monclient: authenticate success, global_id 19438 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 monclient: _renew_subs 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85700a01f0 con 0x7f85701397c0 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 client.19438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 monclient: _renew_subs 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f857009ff10 con 0x7f85701397c0 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 1 librados: init done 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 0x7f857002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f857006d140 con 0x7f857009cd50 2024-04-19T00:49:43.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 0x7f857002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient: get_auth_request con 0x7f857009cd50 auth_method 0 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 10 monclient(hunting): handle_auth_done global_id 22904 payload 824 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8570068240 con 0x7f857009cd50 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f8574805640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 0x7f857002c450 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f856800bcc0 tx=0x7f8568009f30 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 client.19438.objecter ms_handle_connect 0x7f857009cd50 2024-04-19T00:49:43.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 client.19438.objecter resend_mon_ops 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.7 v2:172.21.15.161:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f85600be030 con 0x7f85701397c0 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f855c038e40 0x7f855c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.7 v2:172.21.15.161:3303/0 5 ==== osd_map(222..222 src has 1..222) ==== 7009+0+0 (secure 0 0 0) 0x7f85600bc020 con 0x7f85701397c0 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 client.19438.objecter ms_dispatch 0x7f8570000ba0 osd_map(222..222 src has 1..222) 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 3 client.19438.objecter handle_osd_map got epochs [222,222] > 0 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 3 client.19438.objecter handle_osd_map decoding full epoch 222 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 20 client.19438.objecter dump_active .. 0 homeless 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85680148a0 con 0x7f857009cd50 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: dump: 2024-04-19T00:49:43.031 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.032 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8568014a40 con 0x7f857009cd50 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 msgr2=0x7f857002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 0x7f857002c450 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f856800bcc0 tx=0x7f8568009f30 comp rx=0 tx=0).stop 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 msgr2=0x7f857002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009cd50 0x7f857002c450 secure :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f856800bcc0 tx=0x7f8568009f30 comp rx=0 tx=0).stop 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856ffff640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f855c038e40 0x7f855c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856ffff640 10 monclient: get_auth_request con 0x7f855c038e40 auth_method 0 2024-04-19T00:49:43.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856ffff640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f855c038e40 0x7f855c03b300 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f857009dc70 tx=0x7f8564009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.009+0000 7f856d7fa640 10 client.19438.objecter ms_handle_connect 0x7f855c038e40 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 0x7f857002c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f85700693b0 con 0x7f857009d130 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 0x7f857002c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient: get_auth_request con 0x7f857009d130 auth_method 0 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 10 monclient(hunting): handle_auth_done global_id 22913 payload 824 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 1 -- 172.21.15.43:0/1906316484 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f857013d400 con 0x7f857009d130 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 client.19438.objecter ms_handle_connect 0x7f857009d130 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 client.19438.objecter resend_mon_ops 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8574805640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 0x7f857002c990 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f8568014000 tx=0x7f8568005d60 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8568019040 con 0x7f857009d130 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 monclient: dump: 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.037 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.038 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9399 (secure 0 0 0) 0x7f8568014850 con 0x7f857009d130 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 msgr2=0x7f857002c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 0x7f857002c990 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f8568014000 tx=0x7f8568005d60 comp rx=0 tx=0).stop 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 msgr2=0x7f857002c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f856d7fa640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f857009d130 0x7f857002c990 secure :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f8568014000 tx=0x7f8568005d60 comp rx=0 tx=0).stop 2024-04-19T00:49:43.039 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 42, 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 326528, 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:43.040 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.041 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:43.042 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.043 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:43.044 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:43.045 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.046 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.047 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.048 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.049 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.050 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:43.051 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.052 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:43.053 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.054 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:43.055 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.056 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:43.057 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.058 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.060 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.062 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:43.065 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.066 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.068 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:43.068 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:43.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 10 librados: watch_flush enter 2024-04-19T00:49:43.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 10 librados: watch_flush exit 2024-04-19T00:49:43.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 20 client.19438.objecter shutdown clearing up homeless session... 2024-04-19T00:49:43.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 10 client.19438.objecter successfully canceled tick 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f855c038e40 msgr2=0x7f855c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f855c038e40 0x7f855c03b300 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7f857009dc70 tx=0x7f8564009190 comp rx=0 tx=0).stop 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 10 monclient: shutdown 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 msgr2=0x7f8570136330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 0x7f8570136330 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f85600ac000 tx=0x7f8560019200 comp rx=0 tx=0).stop 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8575006640 1 -- 172.21.15.43:0/1906316484 reap_dead start 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 shutdown_connections 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 --2- 172.21.15.43:0/1906316484 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f85701397c0 0x7f8570136330 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 >> 172.21.15.43:0/1906316484 conn(0x7f85700ab2f0 msgr2=0x7f8570137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 shutdown_connections 2024-04-19T00:49:43.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 -- 172.21.15.43:0/1906316484 wait complete. 2024-04-19T00:49:43.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.017+0000 7f8576a69640 1 librados: shutdown 2024-04-19T00:49:43.072 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:49:43.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-04-19T00:49:43.072 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient: get_monmap_and_config 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient: monmap: 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:43.190921+0000 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:43.190921+0000 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:43.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 Processor -- start 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 -- start start 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient: init 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient: _add_conns ranks=[0,5,7,4,2,10,1,8,3,9,6] 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc510138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 0x7fc51013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): picked mon.noname-h con 0x7fc5100a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc5101397c0 0x7fc5101407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): picked mon.noname-i con 0x7fc5101397c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5100a3dc0 con 0x7fc510138cd0 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc5100a4410 con 0x7fc5100a5b80 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc5100a40a0 con 0x7fc5101397c0 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc516490640 10 monclient(hunting): authenticate will time out at 1253.318066s 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50ffff640 1 -- >> v1:172.21.15.43:6789/0 conn(0x7fc510138cd0 legacy=0x7fc5101390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 0x7fc51013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc5101397c0 0x7fc5101407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 0x7fc51013e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40222/0 (socket says 172.21.15.43:40222) 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 -- 172.21.15.43:0/2198141146 learned_addr learned my addr 172.21.15.43:0/2198141146 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): get_auth_request con 0x7fc5100a5b80 auth_method 0 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): get_auth_request con 0x7fc5101397c0 auth_method 0 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient(hunting): handle_auth_done global_id 44762 payload 274 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 -- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc5101397c0 msgr2=0x7fc5101407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 --2- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc5101397c0 0x7fc5101407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 -- 172.21.15.43:0/2198141146 >> v1:172.21.15.43:6789/0 conn(0x7fc510138cd0 legacy=0x7fc5101390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 monclient: found mon.noname-h 2024-04-19T00:49:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:49:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc50f7fe640 1 -- 172.21.15.43:0/2198141146 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5100a2ae0 con 0x7fc5100a5b80 2024-04-19T00:49:43.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.189+0000 7fc514a2d640 1 --2- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc5101397c0 0x7fc5101407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50f7fe640 1 --2- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 0x7fc51013e270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc500009e50 tx=0x7fc50001c910 comp rx=0 tx=0).ready entity=mon.5 client_cookie=cbddb53361d31cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 1 -- 172.21.15.43:0/2198141146 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5000090d0 con 0x7fc5100a5b80 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: dump: 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.203 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.204 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 1 -- 172.21.15.43:0/2198141146 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5000a5a70 con 0x7fc5100a5b80 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 5 monclient: authenticate success, global_id 44762 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 4 set_mon_vals no callback set 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 1 -- 172.21.15.43:0/2198141146 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc5000a5c60 con 0x7fc5100a5b80 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc50effd640 10 monclient: dump: 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.206 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 10 monclient: shutdown 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 -- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 msgr2=0x7fc51013e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 --2- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 0x7fc51013e270 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc500009e50 tx=0x7fc50001c910 comp rx=0 tx=0).stop 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 -- 172.21.15.43:0/2198141146 shutdown_connections 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 --2- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc5101397c0 0x7fc5101407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 --2- 172.21.15.43:0/2198141146 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5100a5b80 0x7fc51013e270 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 -- 172.21.15.43:0/2198141146 >> 172.21.15.43:0/2198141146 conn(0x7fc5100ab2f0 msgr2=0x7fc5100ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 -- 172.21.15.43:0/2198141146 shutdown_connections 2024-04-19T00:49:43.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.193+0000 7fc516490640 1 -- 172.21.15.43:0/2198141146 wait complete. 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: monmap: 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.208 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 librados: starting msgr at 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 librados: starting objecter 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 Processor -- start 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- start start 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 librados: setting wanted keys 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 librados: calling monclient init 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: init 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: _add_conns ranks=[6,0,5,3,1,4,2,7,8] 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc5100a5b80 0x7fc510136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): picked mon.noname-d con 0x7fc5100a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc510138cd0 0x7fc51009d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): picked mon.noname-a con 0x7fc510138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 0x7fc51009de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): picked mon.noname-h con 0x7fc5101397c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fc510143730 con 0x7fc510138cd0 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc5100a20a0 con 0x7fc5101397c0 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fc5101438b0 con 0x7fc5100a5b80 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): authenticate will time out at 1253.325095s 2024-04-19T00:49:43.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50f7fe640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc510138cd0 msgr2=0x7fc51009d8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc510138cd0 0x7fc51009d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc5100a5b80 msgr2=0x7fc510136950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc5100a5b80 0x7fc510136950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 0x7fc51009de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 0x7fc51009de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40236/0 (socket says 172.21.15.43:40236) 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 -- 172.21.15.43:0/1083945556 learned_addr learned my addr 172.21.15.43:0/1083945556 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): get_auth_request con 0x7fc5101397c0 auth_method 0 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient(hunting): handle_auth_done global_id 44771 payload 824 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc5100a5b80 msgr2=0x7fc510136950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc5100a5b80 0x7fc510136950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc510138cd0 msgr2=0x7fc51009d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc510138cd0 0x7fc51009d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 monclient: found mon.noname-h 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5100a1580 con 0x7fc5101397c0 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc514a2d640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 0x7fc51009de20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e930 tx=0x7fc50800edf0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 client.?.objecter ms_handle_connect 0x7fc5101397c0 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:43.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc508017070 con 0x7fc5101397c0 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: dump: 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.213 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 5 monclient: authenticate success, global_id 44771 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc50800cdb0 con 0x7fc5101397c0 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: _renew_subs 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5100a3590 con 0x7fc5101397c0 2024-04-19T00:49:43.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc508013470 con 0x7fc5101397c0 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 client.44771.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50e7fc640 4 set_mon_vals no callback set 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 10 monclient: dump: 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.215 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: _renew_subs 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5100a12a0 con 0x7fc5101397c0 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 librados: init done 2024-04-19T00:49:43.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 0x7fc51002c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc5100a0140 con 0x7fc51002c450 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 0x7fc51002c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient: get_auth_request con 0x7fc51002c450 auth_method 0 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc50800c940 con 0x7fc5101397c0 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.197+0000 7fc50ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc4fc038e10 0x7fc4fc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(222..222 src has 1..222) ==== 7009+0+0 (secure 0 0 0) 0x7fc50800f040 con 0x7fc5101397c0 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50ffff640 10 monclient(hunting): handle_auth_done global_id 18903 payload 824 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 client.44771.objecter ms_dispatch 0x7fc510000ba0 osd_map(222..222 src has 1..222) 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 3 client.44771.objecter handle_osd_map got epochs [222,222] > 0 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 3 client.44771.objecter handle_osd_map decoding full epoch 222 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50ffff640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5100693b0 con 0x7fc51002c450 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 20 client.44771.objecter dump_active .. 0 homeless 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50f7fe640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc4fc038e10 0x7fc4fc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50f7fe640 10 monclient: get_auth_request con 0x7fc4fc038e10 auth_method 0 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50ffff640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 0x7fc51002c830 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fc504003990 tx=0x7fc50400b810 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 client.44771.objecter ms_handle_connect 0x7fc51002c450 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 client.44771.objecter resend_mon_ops 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc504016070 con 0x7fc51002c450 2024-04-19T00:49:43.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 monclient: dump: 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 client.44771.objecter ms_handle_connect 0x7fc4fc038e10 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50f7fe640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc4fc038e10 0x7fc4fc03b2d0 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ae000 tx=0x7fc5000a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc504003bd0 con 0x7fc51002c450 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 msgr2=0x7fc51002c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 0x7fc51002c830 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fc504003990 tx=0x7fc50400b810 comp rx=0 tx=0).stop 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 msgr2=0x7fc51002c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.201+0000 7fc50cff9640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002c450 0x7fc51002c830 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.205+0000 7fc516490640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:43.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.205+0000 7fc516490640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 0x7fc51002d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.205+0000 7fc516490640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.205+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fc51006d140 con 0x7fc51002cd70 2024-04-19T00:49:43.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 0x7fc51002d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient: get_auth_request con 0x7fc51002cd70 auth_method 0 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 10 monclient(hunting): handle_auth_done global_id 18912 payload 824 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 1 -- 172.21.15.43:0/1083945556 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc51013e4f0 con 0x7fc51002cd70 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50ffff640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 0x7fc51002d180 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fc504013000 tx=0x7fc504003800 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 client.44771.objecter ms_handle_connect 0x7fc51002cd70 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 client.44771.objecter resend_mon_ops 2024-04-19T00:49:43.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc504016040 con 0x7fc51002cd70 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 monclient: dump: 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.224 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9399 (secure 0 0 0) 0x7fc504021410 con 0x7fc51002cd70 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 msgr2=0x7fc51002d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 0x7fc51002d180 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fc504013000 tx=0x7fc504003800 comp rx=0 tx=0).stop 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 msgr2=0x7fc51002d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc50cff9640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fc51002cd70 0x7fc51002d180 secure :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7fc504013000 tx=0x7fc504003800 comp rx=0 tx=0).stop 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 42, 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 326689, 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:43.226 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.227 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.228 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:43.229 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.230 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:43.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.232 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.234 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:43.236 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.239 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.241 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.244 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:43.248 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:43.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 10 librados: watch_flush enter 2024-04-19T00:49:43.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 10 librados: watch_flush exit 2024-04-19T00:49:43.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 20 client.44771.objecter shutdown clearing up homeless session... 2024-04-19T00:49:43.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 10 client.44771.objecter successfully canceled tick 2024-04-19T00:49:43.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc4fc038e10 msgr2=0x7fc4fc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc4fc038e10 0x7fc4fc03b2d0 secure :-1 s=READY pgs=3432 cs=0 l=1 rev1=1 crypto rx=0x7fc5000ae000 tx=0x7fc5000a7070 comp rx=0 tx=0).stop 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 10 monclient: shutdown 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 msgr2=0x7fc51009de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc514a2d640 1 -- 172.21.15.43:0/1083945556 reap_dead start 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 0x7fc51009de20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc50800e930 tx=0x7fc50800edf0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 shutdown_connections 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 --2- 172.21.15.43:0/1083945556 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc5101397c0 0x7fc51009de20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 >> 172.21.15.43:0/1083945556 conn(0x7fc5100ab2f0 msgr2=0x7fc5100aa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 shutdown_connections 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 -- 172.21.15.43:0/1083945556 wait complete. 2024-04-19T00:49:43.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.209+0000 7fc516490640 1 librados: shutdown 2024-04-19T00:49:43.252 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:49:43.252 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-04-19T00:49:43.252 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:49:43.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.369+0000 7fa6732b8640 10 monclient: get_monmap_and_config 2024-04-19T00:49:43.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.369+0000 7fa6732b8640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.369+0000 7fa6732b8640 10 monclient: monmap: 2024-04-19T00:49:43.372 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:43.374298+0000 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:43.374298+0000 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.373 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.374 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.374 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.374 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:43.374 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:43.374 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 Processor -- start 2024-04-19T00:49:43.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 -- start start 2024-04-19T00:49:43.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient: init 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient: _add_conns ranks=[10,8,0,3,4,5,6,1,2,9,7] 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa66c09de40 0x7fa66c0a3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): picked mon.noname-f con 0x7fa66c09de40 addr v2:172.21.15.161:3300/0 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 0x7fa66c0a3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): picked mon.noname-e con 0x7fa66c09e220 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa66c09ae90 addr v1:172.21.15.43:6789/0 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa66c09bb10 con 0x7fa66c09ae90 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa66c09d020 con 0x7fa66c09de40 2024-04-19T00:49:43.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa66c0766c0 con 0x7fa66c09e220 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa6732b8640 10 monclient(hunting): authenticate will time out at 1253.501543s 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa671855640 1 -- >> v1:172.21.15.43:6789/0 conn(0x7fa66c09ae90 legacy=0x7fa66c13f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 0x7fa66c0a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa671054640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fa66c09de40 msgr2=0x7fa66c0a3050 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa671054640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa66c09de40 0x7fa66c0a3050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 0x7fa66c0a3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52198/0 (socket says 172.21.15.43:52198) 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 -- 172.21.15.43:0/1379826206 learned_addr learned my addr 172.21.15.43:0/1379826206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): get_auth_request con 0x7fa66c09e220 auth_method 0 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient(hunting): handle_auth_done global_id 22922 payload 274 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 -- 172.21.15.43:0/1379826206 >> v2:172.21.15.161:3300/0 conn(0x7fa66c09de40 msgr2=0x7fa66c0a3050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 --2- 172.21.15.43:0/1379826206 >> v2:172.21.15.161:3300/0 conn(0x7fa66c09de40 0x7fa66c0a3050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 -- 172.21.15.43:0/1379826206 >> v1:172.21.15.43:6789/0 conn(0x7fa66c09ae90 legacy=0x7fa66c13f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 monclient: found mon.noname-e 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.373+0000 7fa670853640 1 -- 172.21.15.43:0/1379826206 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c09c6f0 con 0x7fa66c09e220 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 --2- 172.21.15.43:0/1379826206 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 0x7fa66c0a3590 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009870 tx=0x7fa65c01c410 comp rx=0 tx=0).ready entity=mon.8 client_cookie=6b285b9120876113 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 1 -- 172.21.15.43:0/1379826206 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa65c0b0070 con 0x7fa66c09e220 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: dump: 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.381 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 1 -- 172.21.15.43:0/1379826206 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c0aa810 con 0x7fa66c09e220 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 5 monclient: authenticate success, global_id 22922 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 1 -- 172.21.15.43:0/1379826206 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa65c0a9300 con 0x7fa66c09e220 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 4 set_mon_vals no callback set 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:49:43.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa663fff640 10 monclient: dump: 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.384 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: shutdown 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- 172.21.15.43:0/1379826206 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 msgr2=0x7fa66c0a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 --2- 172.21.15.43:0/1379826206 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 0x7fa66c0a3590 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009870 tx=0x7fa65c01c410 comp rx=0 tx=0).stop 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- 172.21.15.43:0/1379826206 shutdown_connections 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 --2- 172.21.15.43:0/1379826206 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa66c09e220 0x7fa66c0a3590 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 --2- 172.21.15.43:0/1379826206 >> v2:172.21.15.161:3300/0 conn(0x7fa66c09de40 0x7fa66c0a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- 172.21.15.43:0/1379826206 >> 172.21.15.43:0/1379826206 conn(0x7fa66c0a4140 msgr2=0x7fa66c0a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- 172.21.15.43:0/1379826206 shutdown_connections 2024-04-19T00:49:43.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- 172.21.15.43:0/1379826206 wait complete. 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: monmap: 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 librados: starting msgr at 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 librados: starting objecter 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 Processor -- start 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- start start 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 librados: setting wanted keys 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 librados: calling monclient init 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: init 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient: _add_conns ranks=[0,3,4,1,5,8,6,2,7] 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa66c09ae90 0x7fa66c1d0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): picked mon.noname-a con 0x7fa66c09ae90 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:49:43.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 0x7fa66c099460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): picked mon.noname-g con 0x7fa66c09de40 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa66c09e220 0x7fa66c0999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): picked mon.noname-c con 0x7fa66c09e220 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa66c1438c0 con 0x7fa66c09ae90 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa66c143740 con 0x7fa66c09de40 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa66c143a40 con 0x7fa66c09e220 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa6732b8640 10 monclient(hunting): authenticate will time out at 1253.506618s 2024-04-19T00:49:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa671054640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa66c09ae90 msgr2=0x7fa66c1d0c80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa671855640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa66c09e220 msgr2=0x7fa66c0999a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa671054640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa66c09ae90 0x7fa66c1d0c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa671855640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa66c09e220 0x7fa66c0999a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 0x7fa66c099460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 0x7fa66c099460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:36062/0 (socket says 172.21.15.43:36062) 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 -- 172.21.15.43:0/3486870187 learned_addr learned my addr 172.21.15.43:0/3486870187 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): get_auth_request con 0x7fa66c09de40 auth_method 0 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient(hunting): handle_auth_done global_id 18921 payload 824 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa66c09e220 msgr2=0x7fa66c0999a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa66c09e220 0x7fa66c0999a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa66c09ae90 msgr2=0x7fa66c1d0c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa66c09ae90 0x7fa66c1d0c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 monclient: found mon.noname-g 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.377+0000 7fa670853640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa66c1402c0 con 0x7fa66c09de40 2024-04-19T00:49:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa670853640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 0x7fa66c099460 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa65c01ccb0 tx=0x7fa65c0191b0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 client.?.objecter ms_handle_connect 0x7fa66c09de40 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa65c0b0040 con 0x7fa66c09de40 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: dump: 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa65c0044a0 con 0x7fa66c09de40 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 5 monclient: authenticate success, global_id 18921 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 monclient: _renew_subs 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa66c092920 con 0x7fa66c09de40 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa65c003620 con 0x7fa66c09de40 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6637fe640 4 set_mon_vals no callback set 2024-04-19T00:49:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: dump: 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 client.18921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 monclient: _renew_subs 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa66c1d11c0 con 0x7fa66c09de40 2024-04-19T00:49:43.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 1 librados: init done 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 0x7fa66c13e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa66c06d140 con 0x7fa66c091750 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 0x7fa66c13e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient: get_auth_request con 0x7fa66c091750 auth_method 0 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa65c0bb030 con 0x7fa66c09de40 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa650038e10 0x7fa65003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(222..222 src has 1..222) ==== 7009+0+0 (secure 0 0 0) 0x7fa65c0b9020 con 0x7fa66c09de40 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 client.18921.objecter ms_dispatch 0x7fa66c000ba0 osd_map(222..222 src has 1..222) 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 3 client.18921.objecter handle_osd_map got epochs [222,222] > 0 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 3 client.18921.objecter handle_osd_map decoding full epoch 222 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 10 monclient(hunting): handle_auth_done global_id 44780 payload 824 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa66c0926a0 con 0x7fa66c091750 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 20 client.18921.objecter dump_active .. 0 homeless 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671855640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa650038e10 0x7fa65003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671855640 10 monclient: get_auth_request con 0x7fa650038e10 auth_method 0 2024-04-19T00:49:43.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 client.18921.objecter ms_handle_connect 0x7fa66c091750 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671054640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 0x7fa66c13e680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa66800b450 tx=0x7fa66800b910 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 client.18921.objecter resend_mon_ops 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa668003bd0 con 0x7fa66c091750 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: dump: 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 client.18921.objecter ms_handle_connect 0x7fa650038e10 2024-04-19T00:49:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa671855640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa650038e10 0x7fa65003b2d0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7fa66c09e7b0 tx=0x7fa664009230 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa668015070 con 0x7fa66c091750 2024-04-19T00:49:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 msgr2=0x7fa66c13e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 0x7fa66c13e680 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa66800b450 tx=0x7fa66800b910 comp rx=0 tx=0).stop 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 msgr2=0x7fa66c13e680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.381+0000 7fa661ffb640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c091750 0x7fa66c13e680 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa66800b450 tx=0x7fa66800b910 comp rx=0 tx=0).stop 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa6732b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa6732b8640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 0x7fa66c13efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa6732b8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa66c092e40 con 0x7fa66c13ebc0 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 0x7fa66c13efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient: get_auth_request con 0x7fa66c13ebc0 auth_method 0 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 10 monclient(hunting): handle_auth_done global_id 44789 payload 824 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 1 -- 172.21.15.43:0/3486870187 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa66c13f510 con 0x7fa66c13ebc0 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa671054640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 0x7fa66c13efd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa66800bfd0 tx=0x7fa6680059e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 10 client.18921.objecter ms_handle_connect 0x7fa66c13ebc0 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 10 client.18921.objecter resend_mon_ops 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa668005830 con 0x7fa66c13ebc0 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 10 monclient: dump: 2024-04-19T00:49:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.389+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9569 (secure 0 0 0) 0x7fa6680053c0 con 0x7fa66c13ebc0 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa661ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa661ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 msgr2=0x7fa66c13efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa661ffb640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 0x7fa66c13efd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa66800bfd0 tx=0x7fa6680059e0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa661ffb640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 msgr2=0x7fa66c13efd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa661ffb640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa66c13ebc0 0x7fa66c13efd0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa66800bfd0 tx=0x7fa6680059e0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.418 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:43.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:43.418 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:43.418 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:43.418 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 42, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 183120, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:43.419 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.420 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.421 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:43.422 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.423 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:43.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.426 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:43.427 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.428 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.429 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:43.430 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.430 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.430 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.430 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.430 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:43.430 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.431 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.432 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.433 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.434 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.435 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:43.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.437 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:43.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.443 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.444 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:43.445 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:43.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 10 librados: watch_flush enter 2024-04-19T00:49:43.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 10 librados: watch_flush exit 2024-04-19T00:49:43.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 20 client.18921.objecter shutdown clearing up homeless session... 2024-04-19T00:49:43.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 10 client.18921.objecter successfully canceled tick 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa650038e10 msgr2=0x7fa65003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa650038e10 0x7fa65003b2d0 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7fa66c09e7b0 tx=0x7fa664009230 comp rx=0 tx=0).stop 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 10 monclient: shutdown 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 msgr2=0x7fa66c099460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 0x7fa66c099460 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fa65c01ccb0 tx=0x7fa65c0191b0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa671855640 1 -- 172.21.15.43:0/3486870187 reap_dead start 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 shutdown_connections 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 --2- 172.21.15.43:0/3486870187 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa66c09de40 0x7fa66c099460 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 >> 172.21.15.43:0/3486870187 conn(0x7fa66c0a4140 msgr2=0x7fa66c09f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 shutdown_connections 2024-04-19T00:49:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 -- 172.21.15.43:0/3486870187 wait complete. 2024-04-19T00:49:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.393+0000 7fa6732b8640 1 librados: shutdown 2024-04-19T00:49:43.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:49:43.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-04-19T00:49:43.449 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:49:43.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient: get_monmap_and_config 2024-04-19T00:49:43.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient: monmap: 2024-04-19T00:49:43.568 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:49:43.570586+0000 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:49:43.570586+0000 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.569 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.570 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:49:43.570 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:49:43.570 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 Processor -- start 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 -- start start 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient: init 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient: _add_conns ranks=[1,7,4,2,0,5,8,3,9,10,6] 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f26dc0a4f50 addr v1:172.21.15.161:6789/0 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 0x7f26dc13bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): picked mon.noname-i con 0x7f26dc0731a0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:43.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26dc13c080 0x7f26dc13e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): picked mon.noname-c con 0x7f26dc13c080 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26dc1337a0 con 0x7f26dc0a4f50 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f26dc0a1c80 con 0x7f26dc13c080 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f26dc133a20 con 0x7f26dc0731a0 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26e0b4d640 10 monclient(hunting): authenticate will time out at 1253.697940s 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26dad9d640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26dc13c080 msgr2=0x7f26dc13e470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26dad9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26dc13c080 0x7f26dc13e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26da59c640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f26dc0a4f50 legacy=0x7f26dc0a1180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 0x7f26dc13bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 0x7f26dc13bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:36826/0 (socket says 172.21.15.43:36826) 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 -- 172.21.15.43:0/2415669759 learned_addr learned my addr 172.21.15.43:0/2415669759 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): get_auth_request con 0x7f26dc0731a0 auth_method 0 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_done global_id 19456 payload 274 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 -- 172.21.15.43:0/2415669759 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26dc13c080 msgr2=0x7f26dc13e470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 --2- 172.21.15.43:0/2415669759 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26dc13c080 0x7f26dc13e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 -- 172.21.15.43:0/2415669759 >> v1:172.21.15.161:6789/0 conn(0x7f26dc0a4f50 legacy=0x7f26dc0a1180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 monclient: found mon.noname-i 2024-04-19T00:49:43.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:49:43.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.569+0000 7f26d9d9b640 1 -- 172.21.15.43:0/2415669759 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc070380 con 0x7f26dc0731a0 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d9d9b640 1 --2- 172.21.15.43:0/2415669759 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 0x7f26dc13bb40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f26d00098a0 tx=0x7f26d001c280 comp rx=0 tx=0).ready entity=mon.7 client_cookie=17f57ac60ac92976 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 1 -- 172.21.15.43:0/2415669759 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26d00b0070 con 0x7f26dc0731a0 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: dump: 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.582 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 1 -- 172.21.15.43:0/2415669759 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d00aa810 con 0x7f26dc0731a0 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 1 -- 172.21.15.43:0/2415669759 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26d00a9300 con 0x7f26dc0731a0 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26d959a640 10 monclient: dump: 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.584 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 5 monclient: authenticate success, global_id 19456 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 10 monclient: get_monmap_and_config success 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 4 set_mon_vals no callback set 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 10 monclient: shutdown 2024-04-19T00:49:43.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 -- 172.21.15.43:0/2415669759 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 msgr2=0x7f26dc13bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/2415669759 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 0x7f26dc13bb40 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f26d00098a0 tx=0x7f26d001c280 comp rx=0 tx=0).stop 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 -- 172.21.15.43:0/2415669759 shutdown_connections 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/2415669759 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26dc13c080 0x7f26dc13e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/2415669759 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0731a0 0x7f26dc13bb40 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 -- 172.21.15.43:0/2415669759 >> 172.21.15.43:0/2415669759 conn(0x7f26dc0a4460 msgr2=0x7f26dc0a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 -- 172.21.15.43:0/2415669759 shutdown_connections 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 -- 172.21.15.43:0/2415669759 wait complete. 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 10 monclient: build_initial_monmap 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 10 monclient: monmap: 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:49:43.586 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 librados: starting msgr at 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.573+0000 7f26e0b4d640 1 librados: starting objecter 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 Processor -- start 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- start start 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 librados: setting wanted keys 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 librados: calling monclient init 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: init 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: _reopen_session rank -1 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: _add_conns ranks=[3,7,1,0,5,8,4,2,6] 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26dc0731a0 0x7f26dc1ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): picked mon.noname-g con 0x7f26dc0731a0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0a4f50 0x7f26dc13a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): picked mon.noname-i con 0x7f26dc0a4f50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26dc13c080 0x7f26dc13add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): picked mon.noname-f con 0x7f26dc13c080 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1411d0 con 0x7f26dc13c080 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f26dc0a1aa0 con 0x7f26dc0731a0 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f26dc141350 con 0x7f26dc0a4f50 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): _renew_subs 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): authenticate will time out at 1253.704832s 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26dad9d640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26dc13c080 msgr2=0x7f26dc13add0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26dad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26dc13c080 0x7f26dc13add0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26dc0731a0 0x7f26dc1ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0a4f50 0x7f26dc13a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26dc0731a0 0x7f26dc1ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:36068/0 (socket says 172.21.15.43:36068) 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 -- 172.21.15.43:0/546179671 learned_addr learned my addr 172.21.15.43:0/546179671 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): get_auth_request con 0x7f26dc0731a0 auth_method 0 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): get_auth_request con 0x7f26dc0a4f50 auth_method 0 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient(hunting): handle_auth_done global_id 18930 payload 824 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0a4f50 msgr2=0x7f26dc13a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc0a4f50 0x7f26dc13a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26dc13c080 msgr2=0x7f26dc13add0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26dc13c080 0x7f26dc13add0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient: _finish_hunting 0 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 monclient: found mon.noname-g 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc138350 con 0x7f26dc0731a0 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26da59c640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26dc0731a0 0x7f26dc1ce810 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f26d400b9e0 tx=0x7f26d400bea0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 client.?.objecter ms_handle_connect 0x7f26dc0731a0 2024-04-19T00:49:43.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26d400ce10 con 0x7f26dc0731a0 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: dump: 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.592 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: _finish_auth 0 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: _check_auth_tickets 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d4010070 con 0x7f26dc0731a0 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26d40158e0 con 0x7f26dc0731a0 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 5 monclient: authenticate success, global_id 18930 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: _renew_subs 2024-04-19T00:49:43.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26cbfff640 4 set_mon_vals no callback set 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc06fe80 con 0x7f26dc0731a0 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 10 monclient: dump: 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.594 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 client.18930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: _renew_subs 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc138070 con 0x7f26dc0731a0 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 librados: init done 2024-04-19T00:49:43.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 0x7f26dc02c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f26dc06d140 con 0x7f26dc134b90 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 0x7f26dc02c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient: get_auth_request con 0x7f26dc134b90 auth_method 0 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f26d4015470 con 0x7f26dc0731a0 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.577+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f26bc038e10 0x7f26bc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(222..222 src has 1..222) ==== 7009+0+0 (secure 0 0 0) 0x7f26d4049930 con 0x7f26dc0731a0 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 client.18930.objecter ms_dispatch 0x7f26dc000ba0 osd_map(222..222 src has 1..222) 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 3 client.18930.objecter handle_osd_map got epochs [222,222] > 0 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 3 client.18930.objecter handle_osd_map decoding full epoch 222 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_done global_id 19474 payload 824 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26d9d9b640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26dc1cf480 con 0x7f26dc134b90 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 20 client.18930.objecter dump_active .. 0 homeless 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26dad9d640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f26bc038e10 0x7f26bc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26dad9d640 10 monclient: get_auth_request con 0x7f26bc038e10 auth_method 0 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26d9d9b640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 0x7f26dc02c450 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f26dc069040 tx=0x7f26d0019020 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 client.18930.objecter ms_handle_connect 0x7f26dc134b90 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 client.18930.objecter resend_mon_ops 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26d00ba030 con 0x7f26dc134b90 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 monclient: dump: 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.598 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 client.18930.objecter ms_handle_connect 0x7f26bc038e10 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26dad9d640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f26bc038e10 0x7f26bc03b2d0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f26cc003940 tx=0x7f26cc006c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26d00036e0 con 0x7f26dc134b90 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 msgr2=0x7f26dc02c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 0x7f26dc02c450 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f26dc069040 tx=0x7f26d0019020 comp rx=0 tx=0).stop 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 msgr2=0x7f26dc02c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.581+0000 7f26caffd640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134b90 0x7f26dc02c450 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.585+0000 7f26e0b4d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:49:43.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 0x7f26dc02c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f26dc136910 con 0x7f26dc134f70 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 0x7f26dc02c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient: get_auth_request con 0x7f26dc134f70 auth_method 0 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 10 monclient(hunting): handle_auth_done global_id 19483 payload 824 2024-04-19T00:49:43.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 1 -- 172.21.15.43:0/546179671 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26dc09d800 con 0x7f26dc134f70 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26d9d9b640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 0x7f26dc02c990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097c0 tx=0x7f26d00a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 client.18930.objecter ms_handle_connect 0x7f26dc134f70 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 client.18930.objecter resend_mon_ops 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26d00ba040 con 0x7f26dc134f70 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 monclient: dump: 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:49:43.603 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9229 (secure 0 0 0) 0x7f26d00b6740 con 0x7f26dc134f70 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:49:43.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 msgr2=0x7f26dc02c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 0x7f26dc02c990 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097c0 tx=0x7f26d00a8000 comp rx=0 tx=0).stop 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 msgr2=0x7f26dc02c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26caffd640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26dc134f70 0x7f26dc02c990 secure :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097c0 tx=0x7f26d00a8000 comp rx=0 tx=0).stop 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:49:43.605 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 42, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 327049, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:49:43.606 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.607 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:49:43.609 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:49:43.610 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.612 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.614 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.615 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.617 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:49:43.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:49:43.621 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.624 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:49:43.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.626 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:49:43.627 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:49:43.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:49:43.628 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:49:43.628 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 10 librados: watch_flush enter 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 10 librados: watch_flush exit 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 20 client.18930.objecter shutdown clearing up homeless session... 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 10 client.18930.objecter successfully canceled tick 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f26bc038e10 msgr2=0x7f26bc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f26bc038e10 0x7f26bc03b2d0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7f26cc003940 tx=0x7f26cc006c50 comp rx=0 tx=0).stop 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 10 monclient: shutdown 2024-04-19T00:49:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26dc0731a0 msgr2=0x7f26dc1ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 --2- 172.21.15.43:0/546179671 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f26dc0731a0 0x7f26dc1ce810 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f26d400b9e0 tx=0x7f26d400bea0 comp rx=0 tx=0).stop 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26dad9d640 1 -- 172.21.15.43:0/546179671 reap_dead start 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26dad9d640 1 -- 172.21.15.43:0/546179671 reap_dead start 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 shutdown_connections 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 >> 172.21.15.43:0/546179671 conn(0x7f26dc0a4460 msgr2=0x7f26dc13c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 shutdown_connections 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 -- 172.21.15.43:0/546179671 wait complete. 2024-04-19T00:49:43.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:49:43.589+0000 7f26e0b4d640 1 librados: shutdown 2024-04-19T00:49:43.631 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:49:43.632 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: seed 28857 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (213 ms) 2024-04-19T00:49:44.615 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10749 ms) 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (57 ms) 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11019 ms total) 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6981 ms) 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8024 ms) 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10006 ms) 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: my_snaps [3] 2024-04-19T00:49:44.616 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: my_snaps [4,3] 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: promoting some heads 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: waiting for scrubs... 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: done waiting 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47030 ms) 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (27669 ms) 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-19T00:49:44.617 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7368 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7971 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7958 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10037 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8990 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9982 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13100 ms) 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-19T00:49:44.618 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7929 ms) 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7981 ms) 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (17846 ms) 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (12168 ms) 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-19T00:49:44.619 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (32011 ms) 2024-04-19T00:50:00.387 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-19T00:50:00.387 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: ok, hit_set contains 270:602f83fe:::foo:head 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9046 ms) 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg_num = 32 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 0 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 1 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 2 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 3 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 4 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 5 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 6 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 7 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 8 ls 1713487800,0 2024-04-19T00:50:00.388 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 9 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 10 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 11 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 12 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 13 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 14 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 15 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 16 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 17 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 18 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 19 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 20 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 21 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 22 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 23 ls 1713487800,0 2024-04-19T00:50:00.389 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 24 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 25 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 26 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 27 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 28 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 29 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 30 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg 31 ls 1713487800,0 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: pg_num = 32 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6cac518f:::0:head 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:02547ec2:::1:head 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f905c69b:::2:head 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:cfc208b3:::3:head 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d83876eb:::4:head 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b29083e3:::5:head 2024-04-19T00:50:00.390 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c4fdafeb:::6:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:5c6b0b28:::7:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:bd63b0f1:::8:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e960b815:::9:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:52ea6a34:::10:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:89d3ae78:::11:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:de5d7c5f:::12:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:566253c9:::13:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:62a1935d:::14:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:863748b0:::15:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3958e169:::16:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4d4dabf9:::17:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:8391935d:::18:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:28883081:::19:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:69259c59:::20:head 2024-04-19T00:50:00.391 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4bdb80b7:::21:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a11c5d71:::22:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:271af37b:::23:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:95b121be:::24:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:58d1031b:::25:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:0a050783:::26:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c709704c:::27:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:cbe56eaf:::28:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:86b4b162:::29:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:70d89383:::30:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:dd450c7c:::31:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6d5729b1:::32:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c388f3fb:::33:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:56cfea31:::34:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9dbc1bf7:::35:head 2024-04-19T00:50:00.392 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:40b74ccd:::36:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4d5aaf42:::37:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:920f362c:::38:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6cc53222:::39:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9cad833f:::40:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:1ea84d41:::41:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c4480ef6:::42:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a694361e:::43:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d1bd33e9:::44:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ddc2cd5d:::45:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:2b782207:::46:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7b187fca:::47:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:90ecdf6f:::48:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a5ed95fe:::49:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ea0eaa55:::50:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f33ef17b:::51:head 2024-04-19T00:50:00.393 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a0d1b2f6:::52:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:60c5229e:::53:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:edcbc575:::54:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:102cf253:::55:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:efb7fb0b:::56:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:50d0a326:::57:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d4dc5daf:::58:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3a130462:::59:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ec87ed71:::60:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d5bc9454:::61:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3ddfe313:::62:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7c2816b9:::63:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:47e00e4d:::64:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c6410c18:::65:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b48ed237:::66:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:cd63ad31:::67:head 2024-04-19T00:50:00.394 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b179e92b:::68:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:0d9f741a:::69:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6d3352ae:::70:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c6d5c19e:::71:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:bc4729c3:::72:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:77e930b9:::73:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:0abeecfd:::74:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b7c37e15:::75:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b6378398:::76:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:02bd68de:::77:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:cc795d2d:::78:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:630d4fea:::79:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e0d29ef5:::80:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:fd6f13d2:::81:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:606461d5:::82:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:eadbdc43:::83:head 2024-04-19T00:50:00.395 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:8761d0bb:::84:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9ef0186f:::85:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e0d41294:::86:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:961de695:::87:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:1423148f:::88:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:633a8fa2:::89:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a8653809:::90:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3dac8b33:::91:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:35aad435:::92:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f6dcc343:::93:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:dbbdad87:::94:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:1cb48ce0:::95:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:03cd461c:::96:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:17a4ea99:::97:head 2024-04-19T00:50:00.396 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9993c9a7:::98:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6394211c:::99:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:94c7ae57:::100:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6fdee5bb:::101:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9a477fd1:::102:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:eb850916:::103:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:affc56b9:::104:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b42dc814:::105:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f319f8f0:::106:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9a40b9de:::107:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:8b524f28:::108:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e3de589f:::109:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:90f90a5b:::110:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a7b4f1d7:::111:head 2024-04-19T00:50:00.397 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:af51766e:::112:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b6f90bd1:::113:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e0261208:::114:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c9569ef7:::115:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:61bebe50:::116:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:fe93412b:::117:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d3d38bee:::118:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3100ba0c:::119:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d0560ada:::120:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f0ea8b35:::121:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:766f231a:::122:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a07a2582:::123:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:bd7c6b3a:::124:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:fb2ddaff:::125:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4408e1fe:::126:head 2024-04-19T00:50:00.398 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ee1df7a7:::127:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c3002909:::128:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4f48ffa9:::129:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:edf38733:::130:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c08425c0:::131:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:5f902d98:::132:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:41ea2c93:::133:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:813cee13:::134:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:0131818d:::135:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:26ba5a85:::136:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:381b8a5a:::137:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:28797e47:::138:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:bfca7f22:::139:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:36807075:::140:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:80b03975:::141:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:5c15709b:::142:head 2024-04-19T00:50:00.399 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f39ea15e:::143:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ea992956:::144:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:48887b1c:::145:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:9f24a9dd:::146:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:987f100b:::147:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d2dd3581:::148:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7fed1808:::149:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c80b70e9:::150:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:85ed90f9:::151:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:36428b24:::152:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d044c34a:::153:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7c18bf58:::154:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d1c21232:::155:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a7a3c575:::156:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:87da0633:::157:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d5ac3822:::158:head 2024-04-19T00:50:00.400 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3f20522d:::159:head 2024-04-19T00:50:00.401 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6ca26563:::160:head 2024-04-19T00:50:00.401 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:532ce135:::161:head 2024-04-19T00:50:00.401 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:c78863e6:::162:head 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7462ddf6:o.RoundTripAppendPP (3036 ms) 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2917 ms) 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3019 ms) 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3016 ms) 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: setting pool EIO 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: max_success 94, min_failed 95 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3966 ms) 2024-04-19T00:50:20.720 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3013 ms) 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (139295 ms total) 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3007 ms) 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8991 ms) 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15014 ms) 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3033 ms) 2024-04-19T00:50:20.721 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30045 ms total) 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2973 ms) 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2973 ms total) 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13995 ms) 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7031 ms) 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-19T00:50:20.722 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6991 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6994 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (7825 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6216 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6964 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (31002 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7008 ms) 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-19T00:50:20.723 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7125 ms) 2024-04-19T00:50:20.724 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-19T00:50:20.724 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6922 ms) 2024-04-19T00:50:20.724 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-19T00:50:20.724 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6971 ms) 2024-04-19T00:50:20.724 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-19T00:50:20.724 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7076 ms) 2024-04-19T00:50:55.790 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2::163:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:5d165639:::164:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f43765fc:::165:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b4c720e9:::166:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e694b040:::167:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:afa38db2:::168:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:77ba9f53:::169:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:87495034:::170:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7c96bf0e:::171:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:dbe346cc:::172:head 2024-04-19T00:50:55.791 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:e943ec24:::173:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f97a9c0c:::174:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:6f26e74d:::175:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4f95e106:::176:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:0e6f2f8f:::177:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:05db05f1:::178:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:38a78d66:::179:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:d095610b:::180:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:a1a9d709:::181:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:1e5d39db:::182:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:f7df4fb9:::183:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:03a7f161:::184:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ba70721e:::185:head 2024-04-19T00:50:55.792 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:28e5662d:::186:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:973d52de:::187:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:4303eb1c:::188:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:b990b48e:::189:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:29b8165b:::190:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:3547f197:::191:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:7e260936:::192:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:1abec7b1:::193:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:10fdda93:::194:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:15817eea:::195:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:770bab57:::196:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:ed9e13e7:::197:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:71471a8f:::198:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: checking for 272:10fb1d02:::199:head 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8958 ms) 2024-04-19T00:50:55.793 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: first is 1713487810 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,1713487815,1713487816,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,1713487815,1713487816,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,1713487815,1713487816,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,1713487815,1713487816,1713487818,1713487819,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,1713487815,1713487816,1713487818,1713487819,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487810,1713487812,1713487813,1713487815,1713487816,1713487818,1713487819,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713487813,1713487815,1713487816,1713487818,1713487819,1713487821,1713487822,0 2024-04-19T00:50:55.794 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: first now 1713487813, trimmed 2024-04-19T00:50:55.795 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21012 ms) 2024-04-19T00:50:55.795 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-19T00:50:55.795 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: foo0 2024-04-19T00:50:55.795 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-19T00:50:55.795 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14065 ms) 2024-04-19T00:50:55.795 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-19T00:51:08.702 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ 2024-04-19T00:51:08.702 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2912 ms) 2024-04-19T00:51:08.705 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-19T00:51:08.705 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7049 ms) 2024-04-19T00:51:08.705 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7026 ms) 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6940 ms) 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6997 ms) 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7166 ms) 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (9887 ms) 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (170097 ms total) 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: 2024-04-19T00:51:08.706 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-19T00:51:08.707 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (342410 ms total) 2024-04-19T00:51:08.707 INFO:tasks.workunit.client.0.smithi161.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-19T00:51:13.634 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T00:51:13.635 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-19T00:51:13.635 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T00:51:13.635 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T00:51:13.638 INFO:tasks.ceph.mon.c:Started 2024-04-19T00:51:13.638 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T00:51:13.638 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-04-19T00:51:13.639 INFO:tasks.ceph.mon.d:Restarting daemon 2024-04-19T00:51:13.639 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-04-19T00:51:13.642 INFO:tasks.ceph.mon.d:Started 2024-04-19T00:51:13.642 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-04-19T00:51:13.642 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-04-19T00:51:13.642 INFO:tasks.ceph.mon.f:Restarting daemon 2024-04-19T00:51:13.642 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-04-19T00:51:13.645 INFO:tasks.ceph.mon.f:Started 2024-04-19T00:51:13.645 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T00:51:13.645 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-19T00:51:13.645 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T00:51:13.645 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T00:51:13.648 INFO:tasks.ceph.mon.a:Started 2024-04-19T00:51:13.648 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-04-19T00:51:13.648 INFO:tasks.ceph.mon.e:Sent signal 19 2024-04-19T00:51:13.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:51:13.747 INFO:tasks.ceph.mon.d.smithi043.stderr:2024-04-19T00:51:13.744+0000 7f12f2933c80 -1 mon.d@-1(???).log v0 log_external_backlog rewinding external_log_to from 222 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:51:13.748 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:51:13.744+0000 7efef6b7fc80 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 222 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:51:28.649 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-04-19T00:51:28.650 INFO:tasks.ceph.mon.e:Sent signal 18 2024-04-19T00:51:28.650 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:51:28.651 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:51:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient: get_monmap_and_config 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient: build_initial_monmap 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient: monmap: 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:28.779878+0000 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:28.779878+0000 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:28.779 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 1 Processor -- start 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 1 -- start start 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient: init 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient: _add_conns ranks=[0,4,1,10,6,9,3,2,8,5,7] 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9c409e430 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c4136e30 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe9c413c1b0 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c406c680 con 0x7fe9c409e430 2024-04-19T00:51:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c409bc10 con 0x7fe9c413c1b0 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe9c4137290 con 0x7fe9c4136e30 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c9a52640 10 monclient(hunting): authenticate will time out at 1358.907418s 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:35170/0 (socket says 172.21.15.43:35170) 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 learned_addr learned my addr 172.21.15.43:0/2703849293 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request con 0x7fe9c4136e30 auth_method 0 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:28.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.776+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request con 0x7fe9c4136e30 auth_method 0 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:28.980+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request con 0x7fe9c4136e30 auth_method 0 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:29.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:29.384+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:51:30.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request con 0x7fe9c4136e30 auth_method 0 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:30.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:30.188+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:51:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.108+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298192866 0 0) 0x7fe9c406c680 con 0x7fe9c409e430 2024-04-19T00:51:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.108+0000 7fe9c1ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:51:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.108+0000 7fe9c1ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.108+0000 7fe9c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.108+0000 7fe9c1ffb640 10 monclient(hunting): my global_id is 64098 2024-04-19T00:51:31.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.108+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9b0003530 con 0x7fe9c409e430 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1331772891 0 0) 0x7fe9b0003530 con 0x7fe9c409e430 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c413bc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9c413bc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 >> v1:172.21.15.161:6789/0 conn(0x7fe9c413c1b0 legacy=0x7fe9c413e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: _finish_hunting 0 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c409ef20 con 0x7fe9c409e430 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fe9ac001970 con 0x7fe9c409e430 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c9a52640 5 monclient: authenticate success, global_id 64098 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c9a52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: dump: 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 >> v1:172.21.15.43:6789/0 conn(0x7fe9c409e430 legacy=0x7fe9c409e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient: _add_conns ranks=[4,1,2,3,7,8,0,6,5] 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9b0004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient(hunting): picked mon.c con 0x7fe9c4136e30 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9b0004930 0x7fe9b0004db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.128+0000 7fe9c1ffb640 10 monclient(hunting): picked mon.f con 0x7fe9b0004930 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9b00099c0 0x7fe9b000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 10 monclient(hunting): picked mon.b con 0x7fe9b00099c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c4137290 con 0x7fe9b0004930 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b000c530 con 0x7fe9b00099c0 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe9b000c6e0 con 0x7fe9c4136e30 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c1ffb640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c9a52640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9b0004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9b00099c0 0x7fe9b000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9b0004930 0x7fe9b0004db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe9c4136e30 auth_method 0 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request con 0x7fe9b0004930 auth_method 0 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): get_auth_request con 0x7fe9b00099c0 auth_method 0 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient(hunting): handle_auth_done global_id 64098 payload 293 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9b00099c0 msgr2=0x7fe9b000bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9b00099c0 0x7fe9b000bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9b0004930 msgr2=0x7fe9b0004db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9b0004930 0x7fe9b0004db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 monclient: found mon.c 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 1 -- 172.21.15.43:0/2703849293 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b0003530 con 0x7fe9c4136e30 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:31.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.132+0000 7fe9c37fe640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9b00099c0 0x7fe9b000bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9b0004240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac001ec0 tx=0x7fe9ac01c880 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1b40db821113572f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9ac007a60 con 0x7fe9c4136e30 2024-04-19T00:51:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: dump: 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ac0a88a0 con 0x7fe9c4136e30 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c9a52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c9a52640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 1 -- 172.21.15.43:0/2703849293 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9ac0b0400 con 0x7fe9c4136e30 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c9a52640 4 set_mon_vals no callback set 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.136+0000 7fe9c1ffb640 10 monclient: dump: 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient: shutdown 2024-04-19T00:51:31.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 msgr2=0x7fe9b0004240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/2703849293 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4136e30 0x7fe9b0004240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac001ec0 tx=0x7fe9ac01c880 comp rx=0 tx=0).stop 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c37fe640 1 -- 172.21.15.43:0/2703849293 reap_dead start 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- 172.21.15.43:0/2703849293 shutdown_connections 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- 172.21.15.43:0/2703849293 >> 172.21.15.43:0/2703849293 conn(0x7fe9c40a85a0 msgr2=0x7fe9c40a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- 172.21.15.43:0/2703849293 shutdown_connections 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- 172.21.15.43:0/2703849293 wait complete. 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient: build_initial_monmap 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient: monmap: 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:31.146 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 librados: starting msgr at 2024-04-19T00:51:31.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 librados: starting objecter 2024-04-19T00:51:31.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 Processor -- start 2024-04-19T00:51:31.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- start start 2024-04-19T00:51:31.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 librados: setting wanted keys 2024-04-19T00:51:31.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 librados: calling monclient init 2024-04-19T00:51:31.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient: init 2024-04-19T00:51:31.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:31.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient: _add_conns ranks=[8,4,0,1,2,5,7,3,6] 2024-04-19T00:51:31.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 0x7fe9c41d0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): picked mon.noname-e con 0x7fe9c4136e30 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 0x7fe9c4135fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c4135bc0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9c406fff0 0x7fe9c4070470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c406fff0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c41404d0 con 0x7fe9c406fff0 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe9c40a20b0 con 0x7fe9c4135bc0 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe9c4140650 con 0x7fe9c4136e30 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c9a52640 10 monclient(hunting): authenticate will time out at 1361.271609s 2024-04-19T00:51:31.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 0x7fe9c4135fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 0x7fe9c4135fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50990/0 (socket says 172.21.15.43:50990) 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 1 -- 172.21.15.43:0/4063939416 learned_addr learned my addr 172.21.15.43:0/4063939416 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request con 0x7fe9c4135bc0 auth_method 0 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 0x7fe9c41d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c37fe640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9c406fff0 0x7fe9c4070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c2ffd640 10 monclient(hunting): get_auth_request con 0x7fe9c4136e30 auth_method 0 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c37fe640 10 monclient(hunting): get_auth_request con 0x7fe9c406fff0 auth_method 0 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:31.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.140+0000 7fe9c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 1 -- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c41d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 1 -- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c41d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 0x7fe9c41d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 0x7fe9c41d0910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 10 monclient(hunting): handle_auth_done global_id 64102 payload 824 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 -- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 msgr2=0x7fe9c41d0910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:51:31.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 0x7fe9c41d0910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 -- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9c406fff0 msgr2=0x7fe9c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9c406fff0 0x7fe9c4070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 10 monclient: _finish_hunting 0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 monclient: found mon.noname-c 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 -- 172.21.15.43:0/4063939416 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c4122fc0 con 0x7fe9c4135bc0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c37fe640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9c406fff0 0x7fe9c4070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 0x7fe9c4135fa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4017a80 tx=0x7fe9b4017e80 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 client.?.objecter ms_handle_connect 0x7fe9c4135bc0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9b400f040 con 0x7fe9c4135bc0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:31.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: dump: 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:31.157 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: _finish_auth 0 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b4023480 con 0x7fe9c4135bc0 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9b4023aa0 con 0x7fe9c4135bc0 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 5 monclient: authenticate success, global_id 64102 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 monclient: _renew_subs 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c17fa640 4 set_mon_vals no callback set 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c4071990 con 0x7fe9c4135bc0 2024-04-19T00:51:31.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 monclient: dump: 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:31.159 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 client.64102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 monclient: _renew_subs 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c41364e0 con 0x7fe9c4135bc0 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 1 librados: init done 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c4122a00 con 0x7fe9c4135bc0 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe9b4023630 con 0x7fe9c4135bc0 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9a4038e10 0x7fe9a403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(315..315 src has 1..315) ==== 6154+0+0 (secure 0 0 0) 0x7fe9b404aa00 con 0x7fe9c4135bc0 2024-04-19T00:51:31.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 client.64102.objecter ms_dispatch 0x7fe9c4000ba0 osd_map(315..315 src has 1..315) 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 3 client.64102.objecter handle_osd_map got epochs [315,315] > 0 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 3 client.64102.objecter handle_osd_map decoding full epoch 315 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 20 client.64102.objecter dump_active .. 0 homeless 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9a4038e10 0x7fe9a403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 10 monclient: get_auth_request con 0x7fe9a4038e10 auth_method 0 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9bbfff640 10 client.64102.objecter ms_handle_connect 0x7fe9a4038e10 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.144+0000 7fe9c2ffd640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9a4038e10 0x7fe9a403b2d0 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00057f0 tx=0x7fe9ac001c30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.148+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe9b4020df0 con 0x7fe9c4135bc0 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.148+0000 7fe9bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:31.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.148+0000 7fe9bbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:31.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.376+0000 7fe9c9a52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:51:31.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.376+0000 7fe9c9a52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:51:31.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.376+0000 7fe9c9a52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:31.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.376+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe9c40766e0 con 0x7fe9c4135bc0 2024-04-19T00:51:31.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9bbfff640 1 -- 172.21.15.43:0/4063939416 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7fe9b4021c10 con 0x7fe9c4135bc0 2024-04-19T00:51:31.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:51:31.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9bbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:31.381 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":46,"quorum":[0,2,3,4,5,6,7],"quorum_names":["a","b","g","c","h","d","i"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:51:31.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 10 librados: watch_flush enter 2024-04-19T00:51:31.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 10 librados: watch_flush exit 2024-04-19T00:51:31.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 20 client.64102.objecter shutdown clearing up homeless session... 2024-04-19T00:51:31.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 10 client.64102.objecter successfully canceled tick 2024-04-19T00:51:31.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9a4038e10 msgr2=0x7fe9a403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9a4038e10 0x7fe9a403b2d0 secure :-1 s=READY pgs=3452 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00057f0 tx=0x7fe9ac001c30 comp rx=0 tx=0).stop 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 10 monclient: shutdown 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 msgr2=0x7fe9c4135fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 0x7fe9c4135fa0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4017a80 tx=0x7fe9b4017e80 comp rx=0 tx=0).stop 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 shutdown_connections 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe9c406fff0 0x7fe9c4070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9c4135bc0 0x7fe9c4135fa0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9a4038e10 0x7fe9a403b2d0 unknown :-1 s=CLOSED pgs=3452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 --2- 172.21.15.43:0/4063939416 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe9c4136e30 0x7fe9c41d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 >> 172.21.15.43:0/4063939416 conn(0x7fe9c40a85a0 msgr2=0x7fe9c4137a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 shutdown_connections 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 -- 172.21.15.43:0/4063939416 wait complete. 2024-04-19T00:51:31.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:31.380+0000 7fe9c9a52640 1 librados: shutdown 2024-04-19T00:51:34.400 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:51:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.528+0000 7f7d22cf1640 10 monclient: get_monmap_and_config 2024-04-19T00:51:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.528+0000 7f7d22cf1640 10 monclient: build_initial_monmap 2024-04-19T00:51:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.528+0000 7f7d22cf1640 10 monclient: monmap: 2024-04-19T00:51:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:34.530 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:34.532509+0000 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:34.532509+0000 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:34.531 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:34.532 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:34.532 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:34.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.528+0000 7f7d22cf1640 1 Processor -- start 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.528+0000 7f7d22cf1640 1 -- start start 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.528+0000 7f7d22cf1640 10 monclient: init 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient: _add_conns ranks=[10,3,5,1,9,0,7,4,6,8,2] 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): picked mon.noname-f con 0x7f7d1c138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): picked mon.noname-g con 0x7f7d1c0a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): picked mon.noname-h con 0x7f7d1c1358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7d1c0a4410 con 0x7f7d1c138cd0 2024-04-19T00:51:34.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7d1c139b30 con 0x7f7d1c0a5b80 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7d1c0a3dc0 con 0x7f7d1c1358b0 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d22cf1640 10 monclient(hunting): authenticate will time out at 1364.660186s 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:39198/0 (socket says 172.21.15.43:39198) 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:51504/0 (socket says 172.21.15.43:51504) 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 learned_addr learned my addr 172.21.15.43:0/1064659496 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request con 0x7f7d1c138cd0 auth_method 0 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:34.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d1c1358b0 auth_method 0 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d1c0a5b80 auth_method 0 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:34.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:51:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:34.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.532+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request con 0x7f7d1c138cd0 auth_method 0 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d1c1358b0 auth_method 0 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d1c0a5b80 auth_method 0 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:34.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:34.736+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d1c0a5b80 auth_method 0 2024-04-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request con 0x7f7d1c138cd0 auth_method 0 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d1c1358b0 auth_method 0 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:51:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.140+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:51:35.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:35.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d1c1358b0 auth_method 0 2024-04-19T00:51:35.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d1c0a5b80 auth_method 0 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request con 0x7f7d1c138cd0 auth_method 0 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:35.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:51:35.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:35.944+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): tick 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 msgr2=0x7f7d1c135d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d1c1358b0 0x7f7d1c135d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 msgr2=0x7f7d1c1395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7d1c0a5b80 0x7f7d1c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 msgr2=0x7f7d1c1390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.161:3300/0 conn(0x7f7d1c138cd0 0x7f7d1c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:37.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient: _add_conns ranks=[1,8,9,6,7,0,2,4] 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7d04003800 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): picked mon.noname-e con 0x7f7d040073f0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f7d04009f40 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 -- 172.21.15.43:0/1064659496 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1c0a40a0 con 0x7f7d04003800 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 -- 172.21.15.43:0/1064659496 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f7d1c0a4410 con 0x7f7d04009f40 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 1 -- 172.21.15.43:0/1064659496 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7d1c139b30 con 0x7f7d040073f0 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1a7fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d040073f0 auth_method 0 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:37.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d04009f40 auth_method 0 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:37.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.532+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d04009f40 auth_method 0 2024-04-19T00:51:37.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d040073f0 auth_method 0 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:37.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:37.736+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d04009f40 auth_method 0 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d040073f0 auth_method 0 2024-04-19T00:51:38.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:51:38.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.140+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:51:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.740+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v1:172.21.15.161:6789/0 conn(0x7f7d04003800 legacy=0x7f7d04005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-19T00:51:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.740+0000 7f7d20a8d640 1 -- 172.21.15.43:0/1064659496 >> v1:172.21.15.161:6789/0 conn(0x7f7d04003800 legacy=0x7f7d04005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:51:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.740+0000 7f7d20a8d640 1 --1- 172.21.15.43:0/1064659496 >> v1:172.21.15.161:6789/0 conn(0x7f7d04003800 0x7f7d04005c10 :-1 s=OPENED pgs=53 cs=1 l=1).handle_message read tag failed 2024-04-19T00:51:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.740+0000 7f7d20a8d640 1 --1- 172.21.15.43:0/1064659496 >> v1:172.21.15.161:6789/0 conn(0x7f7d04003800 0x7f7d04005c10 :-1 s=OPENED pgs=53 cs=1 l=1).fault on lossy channel, failing 2024-04-19T00:51:38.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.740+0000 7f7d1b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.161:6789/0 2024-04-19T00:51:38.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d04009f40 auth_method 0 2024-04-19T00:51:38.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d040073f0 auth_method 0 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:38.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_done global_id 23003 payload 274 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 msgr2=0x7f7d0400c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> v2:172.21.15.43:3300/0 conn(0x7f7d04009f40 0x7f7d0400c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 >> v1:172.21.15.161:6789/0 conn(0x7f7d04003800 legacy=0x7f7d04005c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 1 monclient: found mon.noname-e 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d2128e640 1 -- 172.21.15.43:0/1064659496 reap_dead start 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:38.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.944+0000 7f7d1bfff640 1 -- 172.21.15.43:0/1064659496 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0400dc30 con 0x7f7d040073f0 2024-04-19T00:51:38.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009f10 tx=0x7f7d1001dd10 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3688aefd350f0904 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 1 -- 172.21.15.43:0/1064659496 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d100ac070 con 0x7f7d040073f0 2024-04-19T00:51:38.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: dump: 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:38.952 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 1 -- 172.21.15.43:0/1064659496 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d100a6d00 con 0x7f7d040073f0 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 1 -- 172.21.15.43:0/1064659496 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d10016840 con 0x7f7d040073f0 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d22cf1640 5 monclient: authenticate success, global_id 23003 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d22cf1640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d22cf1640 4 set_mon_vals no callback set 2024-04-19T00:51:38.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.948+0000 7f7d1b7fe640 10 monclient: dump: 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:38.954 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:38.955 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:38.955 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient: shutdown 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 msgr2=0x7f7d04009880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009f10 tx=0x7f7d1001dd10 comp rx=0 tx=0).stop 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- 172.21.15.43:0/1064659496 shutdown_connections 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/1064659496 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d040073f0 0x7f7d04009880 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- 172.21.15.43:0/1064659496 >> 172.21.15.43:0/1064659496 conn(0x7f7d1c0ab2f0 msgr2=0x7f7d1c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- 172.21.15.43:0/1064659496 shutdown_connections 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- 172.21.15.43:0/1064659496 wait complete. 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient: build_initial_monmap 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient: monmap: 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:38.960 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:38.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 librados: starting msgr at 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 librados: starting objecter 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 Processor -- start 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- start start 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 librados: setting wanted keys 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 librados: calling monclient init 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient: init 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient: _add_conns ranks=[5,8,1,7,4,3,6,0,2] 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d040073f0 0x7f7d1c1d07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): picked mon.noname-h con 0x7f7d040073f0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 0x7f7d1c1d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): picked mon.noname-e con 0x7f7d1c1d0ee0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 0x7f7d1c09e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:38.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): picked mon.noname-f con 0x7f7d1c09dba0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c135c90 con 0x7f7d1c09dba0 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7d1c0a20a0 con 0x7f7d040073f0 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7d1c135e10 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d22cf1640 10 monclient(hunting): authenticate will time out at 1369.083619s 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 0x7f7d1c1d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 0x7f7d1c1d12e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53524/0 (socket says 172.21.15.43:53524) 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 1 -- 172.21.15.43:0/3656378227 learned_addr learned my addr 172.21.15.43:0/3656378227 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request con 0x7f7d1c1d0ee0 auth_method 0 2024-04-19T00:51:38.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.952+0000 7f7d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 0x7f7d1c09e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d040073f0 0x7f7d1c1d07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 10 monclient(hunting): get_auth_request con 0x7f7d1c09dba0 auth_method 0 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request con 0x7f7d040073f0 auth_method 0 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:38.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 10 monclient(hunting): handle_auth_done global_id 23012 payload 824 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 1 -- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 msgr2=0x7f7d1c09e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 1 -- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 msgr2=0x7f7d1c09e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 0x7f7d1c09e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 -- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d040073f0 msgr2=0x7f7d1c1d07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d040073f0 0x7f7d1c1d07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 -- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 msgr2=0x7f7d1c09e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d2128e640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 0x7f7d1c09e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 0x7f7d1c09e020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 monclient: found mon.noname-e 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 -- 172.21.15.43:0/3656378227 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c09f830 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1bfff640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 0x7f7d1c1d12e0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f7d1001b9a0 tx=0x7f7d10007ac0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:38.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 client.?.objecter ms_handle_connect 0x7f7d1c1d0ee0 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d100ac040 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: dump: 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:38.966 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: _finish_auth 0 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: _check_auth_tickets 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d100b0ac0 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7d100b0c70 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 5 monclient: authenticate success, global_id 23012 2024-04-19T00:51:38.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 monclient: _renew_subs 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1c0a1840 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d1affd640 4 set_mon_vals no callback set 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 monclient: dump: 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:38.968 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 client.23012.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 monclient: _renew_subs 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1c0a0060 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 1 librados: init done 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7d100c1060 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:38.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d1c0693b0 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7d0403a1a0 0x7f7d0403c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(316..316 src has 1..316) ==== 6154+0+0 (secure 0 0 0) 0x7f7d100b5070 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 client.23012.objecter ms_dispatch 0x7f7d1c000ba0 osd_map(316..316 src has 1..316) 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 3 client.23012.objecter handle_osd_map got epochs [316,316] > 0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 3 client.23012.objecter handle_osd_map decoding full epoch 316 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 20 client.23012.objecter dump_active .. 0 homeless 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7d0403a1a0 0x7f7d0403c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 10 monclient: get_auth_request con 0x7f7d0403a1a0 auth_method 0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d197fa640 10 client.23012.objecter ms_handle_connect 0x7f7d0403a1a0 2024-04-19T00:51:38.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.956+0000 7f7d20a8d640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7d0403a1a0 0x7f7d0403c660 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007130 tx=0x7f7d0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:38.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.960+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7d100099e0 con 0x7f7d1c1d0ee0 2024-04-19T00:51:38.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.960+0000 7f7d197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:38.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:38.960+0000 7f7d197fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:39.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.196+0000 7f7d22cf1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:51:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.196+0000 7f7d22cf1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:51:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.196+0000 7f7d22cf1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.196+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7d1c076190 con 0x7f7d1c1d0ee0 2024-04-19T00:51:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d197fa640 1 -- 172.21.15.43:0/3656378227 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f7d100af7b0 con 0x7f7d1c1d0ee0 2024-04-19T00:51:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:51:39.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d197fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:39.201 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":50,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 10 librados: watch_flush enter 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 10 librados: watch_flush exit 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 20 client.23012.objecter shutdown clearing up homeless session... 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 10 client.23012.objecter successfully canceled tick 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7d0403a1a0 msgr2=0x7f7d0403c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7d0403a1a0 0x7f7d0403c660 secure :-1 s=READY pgs=3453 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c007130 tx=0x7f7d0c009040 comp rx=0 tx=0).stop 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 10 monclient: shutdown 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 msgr2=0x7f7d1c1d12e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 0x7f7d1c1d12e0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f7d1001b9a0 tx=0x7f7d10007ac0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 shutdown_connections 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d1c09dba0 0x7f7d1c09e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7d0403a1a0 0x7f7d0403c660 unknown :-1 s=CLOSED pgs=3453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7d1c1d0ee0 0x7f7d1c1d12e0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 --2- 172.21.15.43:0/3656378227 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7d040073f0 0x7f7d1c1d07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 >> 172.21.15.43:0/3656378227 conn(0x7f7d1c0ab2f0 msgr2=0x7f7d1c134d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 shutdown_connections 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 -- 172.21.15.43:0/3656378227 wait complete. 2024-04-19T00:51:39.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.200+0000 7f7d22cf1640 1 librados: shutdown 2024-04-19T00:51:39.222 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:51:39.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:51:39.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:51:39.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.340+0000 7f006f4e4640 10 monclient: get_monmap_and_config 2024-04-19T00:51:39.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.340+0000 7f006f4e4640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.340+0000 7f006f4e4640 10 monclient: monmap: 2024-04-19T00:51:39.343 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:39.345853+0000 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:39.345853+0000 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.344 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.345 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:39.345 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:39.345 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 Processor -- start 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 -- start start 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient: init 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient: _add_conns ranks=[5,6,10,2,4,3,9,1,7,8,0] 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0068138a50 0x7f0068138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): picked mon.noname-h con 0x7f0068138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 0x7f0068139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): picked mon.noname-d con 0x7f00680a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0068137960 0x7f0068137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): picked mon.noname-f con 0x7f0068137960 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f00680a1020 con 0x7f0068137960 2024-04-19T00:51:39.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f00681398b0 con 0x7f0068138a50 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f00680a0780 con 0x7f00680a5bb0 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient(hunting): authenticate will time out at 1369.473004s 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 0x7f0068139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 0x7f0068139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:52586/0 (socket says 172.21.15.43:52586) 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 -- 172.21.15.43:0/2029108987 learned_addr learned my addr 172.21.15.43:0/2029108987 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): get_auth_request con 0x7f00680a5bb0 auth_method 0 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006da81640 1 --2- 172.21.15.43:0/2029108987 >> v2:172.21.15.161:3300/0 conn(0x7f0068137960 0x7f0068137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient(hunting): handle_auth_done global_id 64104 payload 274 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 -- 172.21.15.43:0/2029108987 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0068138a50 msgr2=0x7f0068138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006d280640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0068138a50 0x7f0068138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0068138a50 0x7f0068138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 -- 172.21.15.43:0/2029108987 >> v2:172.21.15.161:3300/0 conn(0x7f0068137960 msgr2=0x7f0068137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 --2- 172.21.15.43:0/2029108987 >> v2:172.21.15.161:3300/0 conn(0x7f0068137960 0x7f0068137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 monclient: found mon.noname-d 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 -- 172.21.15.43:0/2029108987 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00680a0d40 con 0x7f00680a5bb0 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006da81640 10 monclient: get_auth_request con 0x7f0068137960 auth_method 0 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006d280640 10 monclient: get_auth_request con 0x7f0068138a50 auth_method 0 2024-04-19T00:51:39.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006da81640 1 --2- 172.21.15.43:0/2029108987 >> v2:172.21.15.161:3300/0 conn(0x7f0068137960 0x7f0068137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:39.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006d280640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0068138a50 0x7f0068138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006ca7f640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 0x7f0068139370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0058009870 tx=0x7f005801c3f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e416f5a22e58a2ed server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 1 -- 172.21.15.43:0/2029108987 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00580b2070 con 0x7f00680a5bb0 2024-04-19T00:51:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:39.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: dump: 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.358 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 1 -- 172.21.15.43:0/2029108987 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00580ab7b0 con 0x7f00680a5bb0 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 1 -- 172.21.15.43:0/2029108987 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00580abe50 con 0x7f00680a5bb0 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 5 monclient: authenticate success, global_id 64104 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f006f4e4640 4 set_mon_vals no callback set 2024-04-19T00:51:39.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.344+0000 7f005ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005ffff640 10 monclient: dump: 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.360 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: shutdown 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/2029108987 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 msgr2=0x7f0068139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 0x7f0068139370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0058009870 tx=0x7f005801c3f0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/2029108987 shutdown_connections 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- 172.21.15.43:0/2029108987 >> v2:172.21.15.161:3300/0 conn(0x7f0068137960 0x7f0068137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00680a5bb0 0x7f0068139370 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- 172.21.15.43:0/2029108987 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0068138a50 0x7f0068138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/2029108987 >> 172.21.15.43:0/2029108987 conn(0x7f00680a7cb0 msgr2=0x7f00680a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/2029108987 shutdown_connections 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/2029108987 wait complete. 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: monmap: 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:39.362 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 librados: starting msgr at 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 librados: starting objecter 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 Processor -- start 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- start start 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 librados: setting wanted keys 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 librados: calling monclient init 2024-04-19T00:51:39.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: init 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: _add_conns ranks=[7,8,2,4,3,0,6,5,1] 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00680a5bb0 0x7f00681d0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): picked mon.noname-i con 0x7f00680a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0068137960 0x7f00681368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): picked mon.noname-e con 0x7f0068137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0068138a50 0x7f0068136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): picked mon.noname-b con 0x7f0068138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00680a2100 con 0x7f0068138a50 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f00680a1f80 con 0x7f00680a5bb0 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0068142810 con 0x7f0068137960 2024-04-19T00:51:39.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient(hunting): authenticate will time out at 1369.478252s 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0068137960 0x7f00681368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0068138a50 0x7f0068136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0068137960 0x7f00681368a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53530/0 (socket says 172.21.15.43:53530) 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 -- 172.21.15.43:0/3487849572 learned_addr learned my addr 172.21.15.43:0/3487849572 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): get_auth_request con 0x7f0068137960 auth_method 0 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): get_auth_request con 0x7f0068138a50 auth_method 0 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006d280640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00680a5bb0 0x7f00681d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006d280640 10 monclient(hunting): get_auth_request con 0x7f00680a5bb0 auth_method 0 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006d280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006d280640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006d280640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient(hunting): handle_auth_done global_id 23021 payload 824 2024-04-19T00:51:39.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00680a5bb0 msgr2=0x7f00681d0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00680a5bb0 0x7f00681d0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0068138a50 msgr2=0x7f0068136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0068138a50 0x7f0068136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 monclient: found mon.noname-e 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f006809fc90 con 0x7f0068137960 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006da81640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0068138a50 0x7f0068136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:39.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006ca7f640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0068137960 0x7f00681368a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f00580b0040 tx=0x7f0058019190 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 client.?.objecter ms_handle_connect 0x7f0068137960 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00580b2070 con 0x7f0068137960 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: dump: 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.368 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0058003620 con 0x7f0068137960 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0058003c40 con 0x7f0068137960 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005dffb640 10 monclient: dump: 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.370 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f005f7fe640 4 set_mon_vals no callback set 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 5 monclient: authenticate success, global_id 23021 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: _renew_subs 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f006809ee60 con 0x7f0068137960 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 client.23021.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: _renew_subs 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0068122260 con 0x7f0068137960 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.348+0000 7f006f4e4640 1 librados: init done 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006f4e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006f4e4640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 0x7f006813cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f006813ec50 con 0x7f006813c7c0 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 0x7f006813cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f00580bf060 con 0x7f0068137960 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient: get_auth_request con 0x7f006813c7c0 auth_method 0 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f004c038e10 0x7f004c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(316..316 src has 1..316) ==== 6154+0+0 (secure 0 0 0) 0x7f00580e3170 con 0x7f0068137960 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 client.23021.objecter ms_dispatch 0x7f0068000ba0 osd_map(316..316 src has 1..316) 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 3 client.23021.objecter handle_osd_map got epochs [316,316] > 0 2024-04-19T00:51:39.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 3 client.23021.objecter handle_osd_map decoding full epoch 316 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 20 client.23021.objecter dump_active .. 0 homeless 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 10 monclient(hunting): handle_auth_done global_id 64125 payload 824 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00681d1750 con 0x7f006813c7c0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006da81640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f004c038e10 0x7f004c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 client.23021.objecter ms_handle_connect 0x7f006813c7c0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 client.23021.objecter resend_mon_ops 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006d280640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 0x7f006813cba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f006400e770 tx=0x7f006400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006da81640 10 monclient: get_auth_request con 0x7f004c038e10 auth_method 0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0064019020 con 0x7f006813c7c0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 monclient: dump: 2024-04-19T00:51:39.374 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.375 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f006400a780 con 0x7f006813c7c0 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 msgr2=0x7f006813cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 0x7f006813cba0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f006400e770 tx=0x7f006400ec30 comp rx=0 tx=0).stop 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 msgr2=0x7f006813cba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813c7c0 0x7f006813cba0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f006400e770 tx=0x7f006400ec30 comp rx=0 tx=0).stop 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f005dffb640 10 client.23021.objecter ms_handle_connect 0x7f004c038e10 2024-04-19T00:51:39.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.352+0000 7f006da81640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f004c038e10 0x7f004c03b2d0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f0068121bd0 tx=0x7f0060007780 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 0x7f006809c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f006806d140 con 0x7f006813d0e0 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 0x7f006809c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient: get_auth_request con 0x7f006813d0e0 auth_method 0 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 10 monclient(hunting): handle_auth_done global_id 64134 payload 824 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 1 -- 172.21.15.43:0/3487849572 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0068076190 con 0x7f006813d0e0 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 client.23021.objecter ms_handle_connect 0x7f006813d0e0 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 client.23021.objecter resend_mon_ops 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006d280640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 0x7f006809c470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0064000c10 tx=0x7f00640046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0064019040 con 0x7f006813d0e0 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 monclient: dump: 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.379 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0064015b00 con 0x7f006813d0e0 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 msgr2=0x7f006809c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 0x7f006809c470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0064000c10 tx=0x7f00640046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 msgr2=0x7f006809c470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f005dffb640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f006813d0e0 0x7f006809c470 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0064000c10 tx=0x7f00640046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:39.381 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 25631, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.382 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.383 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.384 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:39.385 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.386 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:39.387 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.388 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:39.390 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.393 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:39.394 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.395 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.397 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.400 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.402 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.403 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:39.404 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 10 librados: watch_flush enter 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 10 librados: watch_flush exit 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 20 client.23021.objecter shutdown clearing up homeless session... 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 10 client.23021.objecter successfully canceled tick 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f004c038e10 msgr2=0x7f004c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f004c038e10 0x7f004c03b2d0 secure :-1 s=READY pgs=3454 cs=0 l=1 rev1=1 crypto rx=0x7f0068121bd0 tx=0x7f0060007780 comp rx=0 tx=0).stop 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 10 monclient: shutdown 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0068137960 msgr2=0x7f00681368a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 --2- 172.21.15.43:0/3487849572 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0068137960 0x7f00681368a0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f00580b0040 tx=0x7f0058019190 comp rx=0 tx=0).stop 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006da81640 1 -- 172.21.15.43:0/3487849572 reap_dead start 2024-04-19T00:51:39.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006da81640 1 -- 172.21.15.43:0/3487849572 reap_dead start 2024-04-19T00:51:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 shutdown_connections 2024-04-19T00:51:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 >> 172.21.15.43:0/3487849572 conn(0x7f00680a7cb0 msgr2=0x7f006809ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 shutdown_connections 2024-04-19T00:51:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 -- 172.21.15.43:0/3487849572 wait complete. 2024-04-19T00:51:39.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.360+0000 7f006f4e4640 1 librados: shutdown 2024-04-19T00:51:39.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:39.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:51:39.407 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:51:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient: get_monmap_and_config 2024-04-19T00:51:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient: monmap: 2024-04-19T00:51:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:39.531159+0000 2024-04-19T00:51:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:39.531159+0000 2024-04-19T00:51:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.530 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.531 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:39.532 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:39.532 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 Processor -- start 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 -- start start 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient: init 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient: _add_conns ranks=[0,7,9,8,4,10,6,1,3,2,5] 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7f4138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd7f40a5b80 0x7fd7f413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): picked mon.noname-i con 0x7fd7f40a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:39.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7f41397c0 0x7fd7f41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f41397c0 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7f40a3dc0 con 0x7fd7f4138cd0 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fd7f40a4410 con 0x7fd7f41397c0 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd7f40a40a0 con 0x7fd7f40a5b80 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7fadb9640 10 monclient(hunting): authenticate will time out at 1369.658259s 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7f41397c0 0x7fd7f41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd7f41397c0 0x7fd7f41407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38108/0 (socket says 172.21.15.43:38108) 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f8b55640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fd7f4138cd0 0x7fd7f41390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44526/0 (socket says 172.21.15.43:44526) 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 -- 172.21.15.43:0/1500313635 learned_addr learned my addr 172.21.15.43:0/1500313635 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): get_auth_request con 0x7fd7f41397c0 auth_method 0 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f3fff640 1 --2- 172.21.15.43:0/1500313635 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd7f40a5b80 0x7fd7f413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f3fff640 10 monclient(hunting): get_auth_request con 0x7fd7f40a5b80 auth_method 0 2024-04-19T00:51:39.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient(hunting): handle_auth_done global_id 64143 payload 274 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 -- 172.21.15.43:0/1500313635 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd7f40a5b80 msgr2=0x7fd7f413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 --2- 172.21.15.43:0/1500313635 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd7f40a5b80 0x7fd7f413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 -- 172.21.15.43:0/1500313635 >> v1:172.21.15.43:6789/0 conn(0x7fd7f4138cd0 legacy=0x7fd7f41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 monclient: found mon.noname-a 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f37fe640 1 -- 172.21.15.43:0/1500313635 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3706908605 0 0) 0x7fd7f40a3dc0 con 0x7fd7f4138cd0 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f9356640 1 -- 172.21.15.43:0/1500313635 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f40a2ae0 con 0x7fd7f41397c0 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.528+0000 7fd7f37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:51:39.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 1 --2- 172.21.15.43:0/1500313635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f41407c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009870 tx=0x7fd7e401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2613e44e2f9492e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 1 -- 172.21.15.43:0/1500313635 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7e40b0070 con 0x7fd7f41397c0 2024-04-19T00:51:39.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:39.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: dump: 2024-04-19T00:51:39.557 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.558 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.558 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.558 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.558 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.558 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.558 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.559 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.559 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.559 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.559 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.559 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.559 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.560 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.560 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.560 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 1 -- 172.21.15.43:0/1500313635 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40aa810 con 0x7fd7f41397c0 2024-04-19T00:51:39.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 1 -- 172.21.15.43:0/1500313635 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7e40aae30 con 0x7fd7f41397c0 2024-04-19T00:51:39.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 5 monclient: authenticate success, global_id 64143 2024-04-19T00:51:39.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:39.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 4 set_mon_vals no callback set 2024-04-19T00:51:39.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:39.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f37fe640 10 monclient: dump: 2024-04-19T00:51:39.562 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.562 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.563 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.563 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.563 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.563 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.563 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.564 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.564 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.564 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.564 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.564 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.564 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.565 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.565 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.565 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: shutdown 2024-04-19T00:51:39.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- 172.21.15.43:0/1500313635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 msgr2=0x7fd7f41407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/1500313635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f41407c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd7e4009870 tx=0x7fd7e401c420 comp rx=0 tx=0).stop 2024-04-19T00:51:39.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- 172.21.15.43:0/1500313635 shutdown_connections 2024-04-19T00:51:39.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/1500313635 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f41407c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/1500313635 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd7f40a5b80 0x7fd7f413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- 172.21.15.43:0/1500313635 >> 172.21.15.43:0/1500313635 conn(0x7fd7f40ab2f0 msgr2=0x7fd7f40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- 172.21.15.43:0/1500313635 shutdown_connections 2024-04-19T00:51:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- 172.21.15.43:0/1500313635 wait complete. 2024-04-19T00:51:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: monmap: 2024-04-19T00:51:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:39.567 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:39.568 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:39.568 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.568 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:39.568 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:39.568 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:39.568 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.569 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 librados: starting msgr at 2024-04-19T00:51:39.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 librados: starting objecter 2024-04-19T00:51:39.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 Processor -- start 2024-04-19T00:51:39.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- start start 2024-04-19T00:51:39.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 librados: setting wanted keys 2024-04-19T00:51:39.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 librados: calling monclient init 2024-04-19T00:51:39.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: init 2024-04-19T00:51:39.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient: _add_conns ranks=[6,3,0,4,2,1,8,5,7] 2024-04-19T00:51:39.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd7f40a5b80 0x7fd7f4135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): picked mon.noname-d con 0x7fd7f40a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:39.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd7f4138cd0 0x7fd7f41361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): picked mon.noname-g con 0x7fd7f4138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:39.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f409c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f41397c0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:39.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f4143730 con 0x7fd7f41397c0 2024-04-19T00:51:39.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f40a20a0 con 0x7fd7f4138cd0 2024-04-19T00:51:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd7f41438b0 con 0x7fd7f40a5b80 2024-04-19T00:51:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7fadb9640 10 monclient(hunting): authenticate will time out at 1369.663506s 2024-04-19T00:51:39.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd7f40a5b80 0x7fd7f4135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f409c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd7f40a5b80 0x7fd7f4135c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:52598/0 (socket says 172.21.15.43:52598) 2024-04-19T00:51:39.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 1 -- 172.21.15.43:0/2697492732 learned_addr learned my addr 172.21.15.43:0/2697492732 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): get_auth_request con 0x7fd7f40a5b80 auth_method 0 2024-04-19T00:51:39.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): get_auth_request con 0x7fd7f41397c0 auth_method 0 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f3fff640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd7f4138cd0 0x7fd7f41361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.532+0000 7fd7f9356640 10 monclient(hunting): handle_auth_done global_id 64161 payload 824 2024-04-19T00:51:39.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd7f40a5b80 msgr2=0x7fd7f4135c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd7f40a5b80 0x7fd7f4135c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd7f4138cd0 msgr2=0x7fd7f41361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd7f4138cd0 0x7fd7f41361b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 monclient: found mon.noname-a 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:39.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f409e0d0 con 0x7fd7f41397c0 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd7f40a5b80 0x7fd7f4135c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f9356640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f409c4f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40ab000 tx=0x7fd7e40191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 client.?.objecter ms_handle_connect 0x7fd7f41397c0 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7e40b9070 con 0x7fd7f41397c0 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:39.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: dump: 2024-04-19T00:51:39.579 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.579 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.579 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.579 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.579 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.580 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7e40044b0 con 0x7fd7f41397c0 2024-04-19T00:51:39.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7e40b0040 con 0x7fd7f41397c0 2024-04-19T00:51:39.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 5 monclient: authenticate success, global_id 64161 2024-04-19T00:51:39.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 monclient: _renew_subs 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f40a3040 con 0x7fd7f41397c0 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f2ffd640 4 set_mon_vals no callback set 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: dump: 2024-04-19T00:51:39.583 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.584 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.585 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 client.64161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 monclient: _renew_subs 2024-04-19T00:51:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f40a0430 con 0x7fd7f41397c0 2024-04-19T00:51:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 1 librados: init done 2024-04-19T00:51:39.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 0x7fd7f402c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f406d140 con 0x7fd7f40a0be0 2024-04-19T00:51:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd7e40aac70 con 0x7fd7f41397c0 2024-04-19T00:51:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 0x7fd7f402c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient: get_auth_request con 0x7fd7f40a0be0 auth_method 0 2024-04-19T00:51:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd7d4038e00 0x7fd7d403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(316..316 src has 1..316) ==== 6154+0+0 (secure 0 0 0) 0x7fd7e40e2390 con 0x7fd7f41397c0 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_done global_id 23285 payload 824 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 client.64161.objecter ms_dispatch 0x7fd7f4000ba0 osd_map(316..316 src has 1..316) 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 3 client.64161.objecter handle_osd_map got epochs [316,316] > 0 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 3 client.64161.objecter handle_osd_map decoding full epoch 316 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7f40a3590 con 0x7fd7f40a0be0 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 20 client.64161.objecter dump_active .. 0 homeless 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f8b55640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 0x7fd7f402c450 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004a20 tx=0x7fd7ec004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 client.64161.objecter ms_handle_connect 0x7fd7f40a0be0 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 client.64161.objecter resend_mon_ops 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f3fff640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd7d4038e00 0x7fd7d403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7ec0133f0 con 0x7fd7f40a0be0 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: dump: 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.590 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f3fff640 10 monclient: get_auth_request con 0x7fd7d4038e00 auth_method 0 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7ec018070 con 0x7fd7f40a0be0 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 msgr2=0x7fd7f402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 0x7fd7f402c450 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004a20 tx=0x7fd7ec004ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 msgr2=0x7fd7f402c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f40a0be0 0x7fd7f402c450 secure :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004a20 tx=0x7fd7ec004ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f17fa640 10 client.64161.objecter ms_handle_connect 0x7fd7d4038e00 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.536+0000 7fd7f3fff640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd7d4038e00 0x7fd7d403b2c0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7fd7f409dc50 tx=0x7fd7dc00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 0x7fd7f402cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f409fb90 con 0x7fd7f402c990 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 0x7fd7f402cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient: get_auth_request con 0x7fd7f402c990 auth_method 0 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 10 monclient(hunting): handle_auth_done global_id 23294 payload 824 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 1 -- 172.21.15.43:0/2697492732 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7f40693b0 con 0x7fd7f402c990 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f8b55640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 0x7fd7f402cda0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004a20 tx=0x7fd7ec000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 client.64161.objecter ms_handle_connect 0x7fd7f402c990 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 client.64161.objecter resend_mon_ops 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd7ec013a70 con 0x7fd7f402c990 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 monclient: dump: 2024-04-19T00:51:39.593 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.594 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fd7ec013600 con 0x7fd7f402c990 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 msgr2=0x7fd7f402cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 0x7fd7f402cda0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004a20 tx=0x7fd7ec000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 msgr2=0x7fd7f402cda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f17fa640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd7f402c990 0x7fd7f402cda0 secure :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004a20 tx=0x7fd7ec000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.595 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 443043, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:39.596 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.597 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:39.599 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:39.600 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.601 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.603 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.606 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:39.608 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:39.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.613 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.615 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:39.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:39.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.618 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:39.619 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 10 librados: watch_flush enter 2024-04-19T00:51:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 10 librados: watch_flush exit 2024-04-19T00:51:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 20 client.64161.objecter shutdown clearing up homeless session... 2024-04-19T00:51:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 10 client.64161.objecter successfully canceled tick 2024-04-19T00:51:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd7d4038e00 msgr2=0x7fd7d403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd7d4038e00 0x7fd7d403b2c0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7fd7f409dc50 tx=0x7fd7dc00b040 comp rx=0 tx=0).stop 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 10 monclient: shutdown 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 msgr2=0x7fd7f409c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 --2- 172.21.15.43:0/2697492732 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd7f41397c0 0x7fd7f409c4f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40ab000 tx=0x7fd7e40191c0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f9356640 1 -- 172.21.15.43:0/2697492732 reap_dead start 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7f9356640 1 -- 172.21.15.43:0/2697492732 reap_dead start 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 shutdown_connections 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 >> 172.21.15.43:0/2697492732 conn(0x7fd7f40ab2f0 msgr2=0x7fd7f4137440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 shutdown_connections 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 -- 172.21.15.43:0/2697492732 wait complete. 2024-04-19T00:51:39.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.544+0000 7fd7fadb9640 1 librados: shutdown 2024-04-19T00:51:39.624 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:39.625 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:51:39.625 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient: get_monmap_and_config 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient: monmap: 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:39.750531+0000 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:39.750531+0000 2024-04-19T00:51:39.748 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:39.749 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 Processor -- start 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 -- start start 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient: init 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient: _add_conns ranks=[10,2,4,3,0,5,9,1,7,6,8] 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0d58072980 0x7f0d58139070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): picked mon.noname-f con 0x7f0d58072980 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 0x7f0d580a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): picked mon.noname-b con 0x7f0d58072d60 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d580a4fe0 0x7f0d580a5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): picked mon.noname-c con 0x7f0d580a4fe0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0d5806e750 con 0x7f0d58072980 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5806f7f0 con 0x7f0d58072d60 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0d5806f230 con 0x7f0d580a4fe0 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d5d715640 10 monclient(hunting): authenticate will time out at 1369.877658s 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 0x7f0d580a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d577fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d580a4fe0 0x7f0d580a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 0x7f0d580a4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:54518/0 (socket says 172.21.15.43:54518) 2024-04-19T00:51:39.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d577fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d580a4fe0 0x7f0d580a5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:51012/0 (socket says 172.21.15.43:51012) 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 -- 172.21.15.43:0/3218839193 learned_addr learned my addr 172.21.15.43:0/3218839193 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): get_auth_request con 0x7f0d58072d60 auth_method 0 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d577fe640 10 monclient(hunting): get_auth_request con 0x7f0d580a4fe0 auth_method 0 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d577fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 1 --2- 172.21.15.43:0/3218839193 >> v2:172.21.15.161:3300/0 conn(0x7f0d58072980 0x7f0d58139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): get_auth_request con 0x7f0d58072980 auth_method 0 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient(hunting): handle_auth_done global_id 23303 payload 274 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 -- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d580a4fe0 msgr2=0x7f0d580a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 --2- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d580a4fe0 0x7f0d580a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 -- 172.21.15.43:0/3218839193 >> v2:172.21.15.161:3300/0 conn(0x7f0d58072980 msgr2=0x7f0d58139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 --2- 172.21.15.43:0/3218839193 >> v2:172.21.15.161:3300/0 conn(0x7f0d58072980 0x7f0d58139070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 monclient: found mon.noname-b 2024-04-19T00:51:39.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:51:39.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.748+0000 7f0d567fc640 1 -- 172.21.15.43:0/3218839193 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5806f510 con 0x7f0d58072d60 2024-04-19T00:51:39.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 1 --2- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 0x7f0d580a4aa0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009870 tx=0x7f0d5001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22180d99a9ea2124 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 1 -- 172.21.15.43:0/3218839193 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d500b0070 con 0x7f0d58072d60 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: dump: 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.763 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 5 monclient: authenticate success, global_id 23303 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 1 -- 172.21.15.43:0/3218839193 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d500aa810 con 0x7f0d58072d60 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 4 set_mon_vals no callback set 2024-04-19T00:51:39.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 1 -- 172.21.15.43:0/3218839193 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d500a9300 con 0x7f0d58072d60 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d55ffb640 10 monclient: dump: 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.765 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: shutdown 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 msgr2=0x7f0d580a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 0x7f0d580a4aa0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f0d50009870 tx=0x7f0d5001c420 comp rx=0 tx=0).stop 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- 172.21.15.43:0/3218839193 shutdown_connections 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d580a4fe0 0x7f0d580a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- 172.21.15.43:0/3218839193 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0d58072d60 0x7f0d580a4aa0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- 172.21.15.43:0/3218839193 >> v2:172.21.15.161:3300/0 conn(0x7f0d58072980 0x7f0d58139070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- 172.21.15.43:0/3218839193 >> 172.21.15.43:0/3218839193 conn(0x7f0d580a63e0 msgr2=0x7f0d5806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- 172.21.15.43:0/3218839193 shutdown_connections 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- 172.21.15.43:0/3218839193 wait complete. 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: monmap: 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.767 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 librados: starting msgr at 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 librados: starting objecter 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 Processor -- start 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- start start 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 librados: setting wanted keys 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 librados: calling monclient init 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: init 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient: _add_conns ranks=[7,1,6,2,3,5,0,8,4] 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0d58072980 0x7f0d58135df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): picked mon.noname-i con 0x7f0d58072980 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d58072d60 0x7f0d5813df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): picked mon.noname-f con 0x7f0d58072d60 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 0x7f0d5813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): picked mon.noname-d con 0x7f0d580a4fe0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0d580732e0 con 0x7f0d58072d60 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0d58073160 con 0x7f0d580a4fe0 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0d581426d0 con 0x7f0d58072980 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d5d715640 10 monclient(hunting): authenticate will time out at 1369.883102s 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 0x7f0d5813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 0x7f0d5813e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:52608/0 (socket says 172.21.15.43:52608) 2024-04-19T00:51:39.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 -- 172.21.15.43:0/1070186578 learned_addr learned my addr 172.21.15.43:0/1070186578 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): get_auth_request con 0x7f0d580a4fe0 auth_method 0 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d58072d60 0x7f0d5813df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 10 monclient(hunting): get_auth_request con 0x7f0d58072d60 auth_method 0 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d56ffd640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0d58072980 0x7f0d58135df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d56ffd640 10 monclient(hunting): get_auth_request con 0x7f0d58072980 auth_method 0 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient(hunting): handle_auth_done global_id 64122 payload 824 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0d58072980 msgr2=0x7f0d58135df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0d58072980 0x7f0d58135df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d58072d60 msgr2=0x7f0d5813df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d58072d60 0x7f0d5813df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 monclient: found mon.noname-d 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:39.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5806e4b0 con 0x7f0d580a4fe0 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d567fc640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d58072d60 0x7f0d5813df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d577fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 0x7f0d5813e4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0d4800d800 tx=0x7f0d4800dcc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d4f7fe640 10 client.?.objecter ms_handle_connect 0x7f0d580a4fe0 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.752+0000 7f0d4f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d4800bad0 con 0x7f0d580a4fe0 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: dump: 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.772 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d48019070 con 0x7f0d580a4fe0 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 5 monclient: authenticate success, global_id 64122 2024-04-19T00:51:39.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 monclient: _renew_subs 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5806e900 con 0x7f0d580a4fe0 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d4800bcb0 con 0x7f0d580a4fe0 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: dump: 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.774 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 client.64122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 monclient: _renew_subs 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:39.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d58121790 con 0x7f0d580a4fe0 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 1 librados: init done 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d557fa640 4 set_mon_vals no callback set 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 0x7f0d5813e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0d5806d140 con 0x7f0d58120ef0 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0d4800b660 con 0x7f0d580a4fe0 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 0x7f0d5813e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient: get_auth_request con 0x7f0d58120ef0 auth_method 0 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d3c038e10 0x7f0d3c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(316..316 src has 1..316) ==== 6154+0+0 (secure 0 0 0) 0x7f0d4800f040 con 0x7f0d580a4fe0 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 client.64122.objecter ms_dispatch 0x7f0d58000ba0 osd_map(316..316 src has 1..316) 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 3 client.64122.objecter handle_osd_map got epochs [316,316] > 0 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 3 client.64122.objecter handle_osd_map decoding full epoch 316 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_done global_id 64120 payload 824 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d5813fca0 con 0x7f0d58120ef0 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 20 client.64122.objecter dump_active .. 0 homeless 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d567fc640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d3c038e10 0x7f0d3c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 client.64122.objecter ms_handle_connect 0x7f0d58120ef0 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 client.64122.objecter resend_mon_ops 2024-04-19T00:51:39.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d56ffd640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 0x7f0d5813e9f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009f60 tx=0x7f0d40009f90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d567fc640 10 monclient: get_auth_request con 0x7f0d3c038e10 auth_method 0 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d40003e10 con 0x7f0d58120ef0 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: dump: 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.778 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d400039a0 con 0x7f0d58120ef0 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 msgr2=0x7f0d5813e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 0x7f0d5813e9f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009f60 tx=0x7f0d40009f90 comp rx=0 tx=0).stop 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 msgr2=0x7f0d5813e9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d58120ef0 0x7f0d5813e9f0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d40009f60 tx=0x7f0d40009f90 comp rx=0 tx=0).stop 2024-04-19T00:51:39.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d567fc640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d3c038e10 0x7f0d3c03b2d0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f0d500ad000 tx=0x7f0d500a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.756+0000 7f0d4f7fe640 10 client.64122.objecter ms_handle_connect 0x7f0d3c038e10 2024-04-19T00:51:39.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:39.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 0x7f0d58139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0d5806ca90 con 0x7f0d5813ef30 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 0x7f0d58139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient: get_auth_request con 0x7f0d5813ef30 auth_method 0 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 10 monclient(hunting): handle_auth_done global_id 64129 payload 824 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 1 -- 172.21.15.43:0/1070186578 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d5813f340 con 0x7f0d5813ef30 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d56ffd640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 0x7f0d58139c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d40000f90 tx=0x7f0d40004a50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 client.64122.objecter ms_handle_connect 0x7f0d5813ef30 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 client.64122.objecter resend_mon_ops 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d40004870 con 0x7f0d5813ef30 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 monclient: dump: 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.783 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0d40004400 con 0x7f0d5813ef30 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 msgr2=0x7f0d58139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 0x7f0d58139c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d40000f90 tx=0x7f0d40004a50 comp rx=0 tx=0).stop 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 msgr2=0x7f0d58139c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d4f7fe640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0d5813ef30 0x7f0d58139c50 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d40000f90 tx=0x7f0d40004a50 comp rx=0 tx=0).stop 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 26035, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:39.785 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.786 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.788 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.789 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.791 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.792 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.793 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.794 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.795 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.796 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:39.797 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:39.798 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.799 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:39.801 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:39.803 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.804 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.805 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.806 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:39.806 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:39.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 10 librados: watch_flush enter 2024-04-19T00:51:39.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 10 librados: watch_flush exit 2024-04-19T00:51:39.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 20 client.64122.objecter shutdown clearing up homeless session... 2024-04-19T00:51:39.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 10 client.64122.objecter successfully canceled tick 2024-04-19T00:51:39.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d3c038e10 msgr2=0x7f0d3c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d3c038e10 0x7f0d3c03b2d0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f0d500ad000 tx=0x7f0d500a6000 comp rx=0 tx=0).stop 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 10 monclient: shutdown 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 msgr2=0x7f0d5813e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d577fe640 1 -- 172.21.15.43:0/1070186578 reap_dead start 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 0x7f0d5813e4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0d4800d800 tx=0x7f0d4800dcc0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 shutdown_connections 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 --2- 172.21.15.43:0/1070186578 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d580a4fe0 0x7f0d5813e4b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 >> 172.21.15.43:0/1070186578 conn(0x7f0d580a63e0 msgr2=0x7f0d58137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 shutdown_connections 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 -- 172.21.15.43:0/1070186578 wait complete. 2024-04-19T00:51:39.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.764+0000 7f0d5d715640 1 librados: shutdown 2024-04-19T00:51:39.808 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:39.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:51:39.809 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:51:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient: get_monmap_and_config 2024-04-19T00:51:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient: monmap: 2024-04-19T00:51:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:39.934731+0000 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:39.934731+0000 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:39.933 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:39.934 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 Processor -- start 2024-04-19T00:51:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 -- start start 2024-04-19T00:51:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient: init 2024-04-19T00:51:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient: _add_conns ranks=[5,6,3,7,0,2,9,8,1,10,4] 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb3a409e4c0 0x7fb3a4136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): picked mon.noname-h con 0x7fb3a409e4c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 0x7fb3a40a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): picked mon.noname-d con 0x7fb3a409e8a0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb3a409eca0 0x7fb3a40a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): picked mon.noname-g con 0x7fb3a409eca0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a40a1c50 con 0x7fb3a409eca0 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb3a40a1e20 con 0x7fb3a409e4c0 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb3a406c760 con 0x7fb3a409e8a0 2024-04-19T00:51:39.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3ab7d8640 10 monclient(hunting): authenticate will time out at 1370.061810s 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 0x7fb3a40a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 0x7fb3a40a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:52618/0 (socket says 172.21.15.43:52618) 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 -- 172.21.15.43:0/3111251349 learned_addr learned my addr 172.21.15.43:0/3111251349 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): get_auth_request con 0x7fb3a409e8a0 auth_method 0 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a9574640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb3a409e4c0 0x7fb3a4136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a9d75640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb3a409eca0 0x7fb3a40a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_done global_id 64131 payload 274 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 -- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb3a409e4c0 msgr2=0x7fb3a4136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb3a409e4c0 0x7fb3a4136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 -- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb3a409eca0 msgr2=0x7fb3a40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb3a409eca0 0x7fb3a40a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 monclient: found mon.noname-d 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a8d73640 1 -- 172.21.15.43:0/3111251349 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a40a5510 con 0x7fb3a409e8a0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a9d75640 10 monclient: get_auth_request con 0x7fb3a409eca0 auth_method 0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a9d75640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb3a409eca0 0x7fb3a40a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a9574640 10 monclient: get_auth_request con 0x7fb3a409e4c0 auth_method 0 2024-04-19T00:51:39.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.932+0000 7fb3a9574640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb3a409e4c0 0x7fb3a4136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 0x7fb3a40a3940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3980098a0 tx=0x7fb39801c4d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7f7183f3ed58908 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 1 -- 172.21.15.43:0/3111251349 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb3980b0070 con 0x7fb3a409e8a0 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: dump: 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.941 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 1 -- 172.21.15.43:0/3111251349 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3980aa810 con 0x7fb3a409e8a0 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 1 -- 172.21.15.43:0/3111251349 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb3980a9300 con 0x7fb3a409e8a0 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 5 monclient: authenticate success, global_id 64131 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 4 set_mon_vals no callback set 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a3fff640 10 monclient: dump: 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.943 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: shutdown 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3111251349 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 msgr2=0x7fb3a40a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 0x7fb3a40a3940 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3980098a0 tx=0x7fb39801c4d0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3111251349 shutdown_connections 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb3a409eca0 0x7fb3a40a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a409e8a0 0x7fb3a40a3940 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3111251349 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb3a409e4c0 0x7fb3a4136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3111251349 >> 172.21.15.43:0/3111251349 conn(0x7fb3a40a85c0 msgr2=0x7fb3a40a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3111251349 shutdown_connections 2024-04-19T00:51:39.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3111251349 wait complete. 2024-04-19T00:51:39.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: build_initial_monmap 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: monmap: 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:39.952 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 librados: starting msgr at 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 librados: starting objecter 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 Processor -- start 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- start start 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 librados: setting wanted keys 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 librados: calling monclient init 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: init 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:39.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient: _add_conns ranks=[8,4,2,7,3,5,0,6,1] 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 0x7fb3a409daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): picked mon.noname-e con 0x7fb3a409e4c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb3a409e8a0 0x7fb3a4072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): picked mon.noname-c con 0x7fb3a409e8a0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3a409eca0 0x7fb3a40726a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): picked mon.noname-b con 0x7fb3a409eca0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a41406e0 con 0x7fb3a409eca0 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fb3a40a47d0 con 0x7fb3a409e8a0 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb3a4140860 con 0x7fb3a409e4c0 2024-04-19T00:51:39.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3ab7d8640 10 monclient(hunting): authenticate will time out at 1370.067020s 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 0x7fb3a409daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb3a409e8a0 0x7fb3a4072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 0x7fb3a409daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53534/0 (socket says 172.21.15.43:53534) 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 -- 172.21.15.43:0/3033110765 learned_addr learned my addr 172.21.15.43:0/3033110765 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb3a409e8a0 0x7fb3a4072160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:51040/0 (socket says 172.21.15.43:51040) 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): get_auth_request con 0x7fb3a409e4c0 auth_method 0 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): get_auth_request con 0x7fb3a409e8a0 auth_method 0 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3a409eca0 0x7fb3a40726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): get_auth_request con 0x7fb3a409eca0 auth_method 0 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9d75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient(hunting): handle_auth_done global_id 23030 payload 824 2024-04-19T00:51:39.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb3a409e8a0 msgr2=0x7fb3a4072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb3a409e8a0 0x7fb3a4072160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3a409eca0 msgr2=0x7fb3a40726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb3a409eca0 0x7fb3a40726a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient: _finish_hunting 0 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 monclient: found mon.noname-e 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a4070400 con 0x7fb3a409e4c0 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb3a409e8a0 0x7fb3a4072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a9574640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 0x7fb3a409daf0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00c7c0 tx=0x7fb39c00cc80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a1ffb640 10 client.?.objecter ms_handle_connect 0x7fb3a409e4c0 2024-04-19T00:51:39.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.936+0000 7fb3a1ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb39c007ca0 con 0x7fb3a409e4c0 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: dump: 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.958 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb39c0113f0 con 0x7fb3a409e4c0 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 5 monclient: authenticate success, global_id 23030 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 monclient: _renew_subs 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a41cee20 con 0x7fb3a409e4c0 2024-04-19T00:51:39.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb39c011a10 con 0x7fb3a409e4c0 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: dump: 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.960 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 client.23030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 monclient: _renew_subs 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a37fe640 4 set_mon_vals no callback set 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a4070950 con 0x7fb3a409e4c0 2024-04-19T00:51:39.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 1 librados: init done 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 0x7fb3a41320d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb3a406d400 con 0x7fb3a4133cf0 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 0x7fb3a41320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient: get_auth_request con 0x7fb3a4133cf0 auth_method 0 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_done global_id 64140 payload 824 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3a4131c60 con 0x7fb3a4133cf0 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 0x7fb3a41320d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb398009870 tx=0x7fb398019270 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 client.23030.objecter ms_handle_connect 0x7fb3a4133cf0 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 client.23030.objecter resend_mon_ops 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb39c011bb0 con 0x7fb3a409e4c0 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb390038e10 0x7fb39003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(317..317 src has 1..317) ==== 5769+0+0 (secure 0 0 0) 0x7fb39c049440 con 0x7fb3a409e4c0 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 client.23030.objecter ms_dispatch 0x7fb3a4000ba0 osd_map(317..317 src has 1..317) 2024-04-19T00:51:39.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 3 client.23030.objecter handle_osd_map got epochs [317,317] > 0 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 3 client.23030.objecter handle_osd_map decoding full epoch 317 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 20 client.23030.objecter dump_active .. 0 homeless 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb3980ba070 con 0x7fb3a4133cf0 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: dump: 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.964 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb3980aaa80 con 0x7fb3a4133cf0 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:39.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 msgr2=0x7fb3a41320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 0x7fb3a41320d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb398009870 tx=0x7fb398019270 comp rx=0 tx=0).stop 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 msgr2=0x7fb3a41320d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4133cf0 0x7fb3a41320d0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb398009870 tx=0x7fb398019270 comp rx=0 tx=0).stop 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a9d75640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb390038e10 0x7fb39003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a9d75640 10 monclient: get_auth_request con 0x7fb390038e10 auth_method 0 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a9d75640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb390038e10 0x7fb39003b2d0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7fb394007f70 tx=0x7fb3940076e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.940+0000 7fb3a1ffb640 10 client.23030.objecter ms_handle_connect 0x7fb390038e10 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 0x7fb3a4132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb3a4132f60 con 0x7fb3a4132610 2024-04-19T00:51:39.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 0x7fb3a4132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient: get_auth_request con 0x7fb3a4132610 auth_method 0 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 10 monclient(hunting): handle_auth_done global_id 64149 payload 824 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 1 -- 172.21.15.43:0/3033110765 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb3a41cf7b0 con 0x7fb3a4132610 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 client.23030.objecter ms_handle_connect 0x7fb3a4132610 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 client.23030.objecter resend_mon_ops 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a8d73640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 0x7fb3a4132a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb39801c510 tx=0x7fb3980a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb3980b0040 con 0x7fb3a4132610 2024-04-19T00:51:39.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 monclient: dump: 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:39.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7fb3980038f0 con 0x7fb3a4132610 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 msgr2=0x7fb3a4132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 0x7fb3a4132a20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb39801c510 tx=0x7fb3980a8000 comp rx=0 tx=0).stop 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 msgr2=0x7fb3a4132a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a1ffb640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb3a4132610 0x7fb3a4132a20 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb39801c510 tx=0x7fb3980a8000 comp rx=0 tx=0).stop 2024-04-19T00:51:39.969 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 26223, 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:39.970 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.971 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.972 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:39.973 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:39.974 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.975 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.976 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:39.977 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.978 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:39.980 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.982 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.985 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.988 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.989 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.991 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.992 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:39.993 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 10 librados: watch_flush enter 2024-04-19T00:51:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 10 librados: watch_flush exit 2024-04-19T00:51:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 20 client.23030.objecter shutdown clearing up homeless session... 2024-04-19T00:51:39.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 10 client.23030.objecter successfully canceled tick 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb390038e10 msgr2=0x7fb39003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb390038e10 0x7fb39003b2d0 secure :-1 s=READY pgs=3457 cs=0 l=1 rev1=1 crypto rx=0x7fb394007f70 tx=0x7fb3940076e0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 10 monclient: shutdown 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 msgr2=0x7fb3a409daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 0x7fb3a409daf0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00c7c0 tx=0x7fb39c00cc80 comp rx=0 tx=0).stop 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3a9d75640 1 -- 172.21.15.43:0/3033110765 reap_dead start 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 shutdown_connections 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 --2- 172.21.15.43:0/3033110765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb3a409e4c0 0x7fb3a409daf0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 >> 172.21.15.43:0/3033110765 conn(0x7fb3a40a85c0 msgr2=0x7fb3a4130d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 shutdown_connections 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 -- 172.21.15.43:0/3033110765 wait complete. 2024-04-19T00:51:39.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:39.948+0000 7fb3ab7d8640 1 librados: shutdown 2024-04-19T00:51:39.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:39.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:51:39.997 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:51:40.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient: get_monmap_and_config 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient: monmap: 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:40.119111+0000 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:40.119111+0000 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.117 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.118 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.118 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.118 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.118 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:40.118 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:40.118 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 Processor -- start 2024-04-19T00:51:40.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 -- start start 2024-04-19T00:51:40.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient: init 2024-04-19T00:51:40.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient: _add_conns ranks=[1,10,8,9,0,3,5,2,7,4,6] 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f02b0138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f02b00a5b80 0x7f02b013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): picked mon.noname-f con 0x7f02b00a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 0x7f02b01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): picked mon.noname-e con 0x7f02b01397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b00a3dc0 con 0x7f02b0138cd0 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f02b00a4410 con 0x7f02b00a5b80 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f02b00a40a0 con 0x7f02b01397c0 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b8024640 10 monclient(hunting): authenticate will time out at 1370.246258s 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 0x7f02b01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 0x7f02b01407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53538/0 (socket says 172.21.15.43:53538) 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 -- 172.21.15.43:0/2580722105 learned_addr learned my addr 172.21.15.43:0/2580722105 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): get_auth_request con 0x7f02b01397c0 auth_method 0 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b55bf640 1 --2- 172.21.15.43:0/2580722105 >> v2:172.21.15.161:3300/0 conn(0x7f02b00a5b80 0x7f02b013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b55bf640 10 monclient(hunting): get_auth_request con 0x7f02b00a5b80 auth_method 0 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b55bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b55bf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b55bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient(hunting): handle_auth_done global_id 23039 payload 274 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 -- 172.21.15.43:0/2580722105 >> v2:172.21.15.161:3300/0 conn(0x7f02b00a5b80 msgr2=0x7f02b013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 --2- 172.21.15.43:0/2580722105 >> v2:172.21.15.161:3300/0 conn(0x7f02b00a5b80 0x7f02b013e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 -- 172.21.15.43:0/2580722105 >> v1:172.21.15.161:6789/0 conn(0x7f02b0138cd0 legacy=0x7f02b01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 monclient: found mon.noname-e 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:40.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.116+0000 7f02b65c1640 1 -- 172.21.15.43:0/2580722105 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b00a2ae0 con 0x7f02b01397c0 2024-04-19T00:51:40.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b65c1640 1 --2- 172.21.15.43:0/2580722105 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 0x7f02b01407c0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f02b00a7360 tx=0x7f02a801c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=f1c271546236ca0c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 1 -- 172.21.15.43:0/2580722105 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02a80b2070 con 0x7f02b01397c0 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: dump: 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.131 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 1 -- 172.21.15.43:0/2580722105 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a80ab7f0 con 0x7f02b01397c0 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 5 monclient: authenticate success, global_id 23039 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 1 -- 172.21.15.43:0/2580722105 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02b00a20a0 con 0x7f02b01397c0 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 4 set_mon_vals no callback set 2024-04-19T00:51:40.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b4dbe640 10 monclient: dump: 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.133 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: shutdown 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- 172.21.15.43:0/2580722105 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 msgr2=0x7f02b01407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 --2- 172.21.15.43:0/2580722105 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 0x7f02b01407c0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f02b00a7360 tx=0x7f02a801c3e0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- 172.21.15.43:0/2580722105 shutdown_connections 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 --2- 172.21.15.43:0/2580722105 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b01397c0 0x7f02b01407c0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 --2- 172.21.15.43:0/2580722105 >> v2:172.21.15.161:3300/0 conn(0x7f02b00a5b80 0x7f02b013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- 172.21.15.43:0/2580722105 >> 172.21.15.43:0/2580722105 conn(0x7f02b00ab2f0 msgr2=0x7f02b00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- 172.21.15.43:0/2580722105 shutdown_connections 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- 172.21.15.43:0/2580722105 wait complete. 2024-04-19T00:51:40.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: monmap: 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.135 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 librados: starting msgr at 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 librados: starting objecter 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 Processor -- start 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- start start 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 librados: setting wanted keys 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 librados: calling monclient init 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: init 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient: _add_conns ranks=[6,1,0,4,3,7,5,2,8] 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f02b00a5b80 0x7f02b01d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): picked mon.noname-d con 0x7f02b00a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02b0138cd0 0x7f02b00a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): picked mon.noname-f con 0x7f02b0138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02b01397c0 0x7f02b00a0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): picked mon.noname-a con 0x7f02b01397c0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f02b01438b0 con 0x7f02b01397c0 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f02b0143730 con 0x7f02b0138cd0 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f02b0143a30 con 0x7f02b00a5b80 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b8024640 10 monclient(hunting): authenticate will time out at 1370.251810s 2024-04-19T00:51:40.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b65c1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02b01397c0 0x7f02b00a0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.120+0000 7f02b5dc0640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f02b00a5b80 0x7f02b01d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02b01397c0 0x7f02b00a0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38124/0 (socket says 172.21.15.43:38124) 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 -- 172.21.15.43:0/666341721 learned_addr learned my addr 172.21.15.43:0/666341721 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): get_auth_request con 0x7f02b01397c0 auth_method 0 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): get_auth_request con 0x7f02b00a5b80 auth_method 0 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02b0138cd0 0x7f02b00a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 10 monclient(hunting): get_auth_request con 0x7f02b0138cd0 auth_method 0 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient(hunting): handle_auth_done global_id 64170 payload 824 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f02b00a5b80 msgr2=0x7f02b01d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f02b00a5b80 0x7f02b01d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02b0138cd0 msgr2=0x7f02b00a0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02b0138cd0 0x7f02b00a0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 monclient: found mon.noname-a 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b009d400 con 0x7f02b01397c0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f02b0138cd0 0x7f02b00a0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f02b00a5b80 0x7f02b01d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b65c1640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02b01397c0 0x7f02b00a0e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02a80b0040 tx=0x7f02a8019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 client.?.objecter ms_handle_connect 0x7f02b01397c0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02a80bd070 con 0x7f02b01397c0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: dump: 2024-04-19T00:51:40.140 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.141 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a80044b0 con 0x7f02b01397c0 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 5 monclient: authenticate success, global_id 64170 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02a80b2040 con 0x7f02b01397c0 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 monclient: _renew_subs 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b0137280 con 0x7f02b01397c0 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a3fff640 4 set_mon_vals no callback set 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 client.64170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: dump: 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.143 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 monclient: _renew_subs 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b009ce40 con 0x7f02b01397c0 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 1 librados: init done 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 0x7f02b0135690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f02b006d140 con 0x7f02b009bc70 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f02a8004040 con 0x7f02b01397c0 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 0x7f02b0135690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient: get_auth_request con 0x7f02b009bc70 auth_method 0 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0294038e10 0x7f029403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 5769+0+0 (secure 0 0 0) 0x7f02a80e29d0 con 0x7f02b01397c0 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 client.64170.objecter ms_dispatch 0x7f02b0000ba0 osd_map(317..317 src has 1..317) 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 3 client.64170.objecter handle_osd_map got epochs [317,317] > 0 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 3 client.64170.objecter handle_osd_map decoding full epoch 317 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 20 client.64170.objecter dump_active .. 0 homeless 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_done global_id 23048 payload 824 2024-04-19T00:51:40.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02b0068240 con 0x7f02b009bc70 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 client.64170.objecter ms_handle_connect 0x7f02b009bc70 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 client.64170.objecter resend_mon_ops 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b5dc0640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 0x7f02b0135690 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fae0 tx=0x7f02a4004510 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0294038e10 0x7f029403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 10 monclient: get_auth_request con 0x7f0294038e10 auth_method 0 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02a4005030 con 0x7f02b009bc70 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: dump: 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.146 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.147 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f02a400dad0 con 0x7f02b009bc70 2024-04-19T00:51:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 msgr2=0x7f02b0135690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 0x7f02b0135690 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fae0 tx=0x7f02a4004510 comp rx=0 tx=0).stop 2024-04-19T00:51:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 msgr2=0x7f02b0135690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b009bc70 0x7f02b0135690 secure :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fae0 tx=0x7f02a4004510 comp rx=0 tx=0).stop 2024-04-19T00:51:40.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02a27fc640 10 client.64170.objecter ms_handle_connect 0x7f0294038e10 2024-04-19T00:51:40.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.124+0000 7f02b55bf640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0294038e10 0x7f029403b2d0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f02ac003fd0 tx=0x7f02ac0039e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:40.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 0x7f02b0135fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f02b0136520 con 0x7f02b0135bd0 2024-04-19T00:51:40.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 0x7f02b0135fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient: get_auth_request con 0x7f02b0135bd0 auth_method 0 2024-04-19T00:51:40.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 10 monclient(hunting): handle_auth_done global_id 23057 payload 824 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 1 -- 172.21.15.43:0/666341721 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02b00693b0 con 0x7f02b0135bd0 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 client.64170.objecter ms_handle_connect 0x7f02b0135bd0 2024-04-19T00:51:40.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b5dc0640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 0x7f02b0135fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fae0 tx=0x7f02a4003770 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 client.64170.objecter resend_mon_ops 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f02a4004a50 con 0x7f02b0135bd0 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 monclient: dump: 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.156 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.157 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f02a40045e0 con 0x7f02b0135bd0 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 msgr2=0x7f02b0135fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 0x7f02b0135fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fae0 tx=0x7f02a4003770 comp rx=0 tx=0).stop 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 msgr2=0x7f02b0135fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02a27fc640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f02b0135bd0 0x7f02b0135fe0 secure :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f02a400fae0 tx=0x7f02a4003770 comp rx=0 tx=0).stop 2024-04-19T00:51:40.158 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 443643, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:40.159 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.160 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.161 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.162 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:40.164 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.166 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.167 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.168 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.169 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.170 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.171 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.172 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.173 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.175 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:40.176 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.177 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.178 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.180 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:40.181 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 10 librados: watch_flush enter 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 10 librados: watch_flush exit 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 20 client.64170.objecter shutdown clearing up homeless session... 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 10 client.64170.objecter successfully canceled tick 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0294038e10 msgr2=0x7f029403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0294038e10 0x7f029403b2d0 secure :-1 s=READY pgs=3458 cs=0 l=1 rev1=1 crypto rx=0x7f02ac003fd0 tx=0x7f02ac0039e0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 10 monclient: shutdown 2024-04-19T00:51:40.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02b01397c0 msgr2=0x7f02b00a0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 --2- 172.21.15.43:0/666341721 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f02b01397c0 0x7f02b00a0e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f02a80b0040 tx=0x7f02a8019180 comp rx=0 tx=0).stop 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b65c1640 1 -- 172.21.15.43:0/666341721 reap_dead start 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b65c1640 1 -- 172.21.15.43:0/666341721 reap_dead start 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 shutdown_connections 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 >> 172.21.15.43:0/666341721 conn(0x7f02b00ab2f0 msgr2=0x7f02b0131ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 shutdown_connections 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 -- 172.21.15.43:0/666341721 wait complete. 2024-04-19T00:51:40.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.132+0000 7f02b8024640 1 librados: shutdown 2024-04-19T00:51:40.184 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:40.185 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:51:40.185 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:51:40.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient: get_monmap_and_config 2024-04-19T00:51:40.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient: monmap: 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:40.306758+0000 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:40.306758+0000 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.305 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:40.306 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:40.306 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 Processor -- start 2024-04-19T00:51:40.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 -- start start 2024-04-19T00:51:40.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient: init 2024-04-19T00:51:40.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient: _add_conns ranks=[10,0,5,2,9,7,3,4,8,6,1] 2024-04-19T00:51:40.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f01d00a4aa0 0x7f01d0071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): picked mon.noname-f con 0x7f01d00a4aa0 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01d00a4e80 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01d0142570 0x7f01d0144960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): picked mon.noname-h con 0x7f01d0142570 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d006db80 con 0x7f01d00a4e80 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f01d006c4d0 con 0x7f01d00a4aa0 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f01d006f830 con 0x7f01d0142570 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 10 monclient(hunting): authenticate will time out at 1370.433965s 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f01d00a4e80 0x7f01d00a5260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44538/0 (socket says 172.21.15.43:44538) 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 1 -- 172.21.15.43:0/4089018843 learned_addr learned my addr 172.21.15.43:0/4089018843 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:51:40.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 --2- 172.21.15.43:0/4089018843 >> v2:172.21.15.161:3300/0 conn(0x7f01d00a4aa0 0x7f01d0071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01d0142570 0x7f01d0144960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): get_auth_request con 0x7f01d00a4aa0 auth_method 0 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): get_auth_request con 0x7f01d0142570 auth_method 0 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1821061696 0 0) 0x7f01d006db80 con 0x7f01d00a4e80 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): _negotiate 2024-04-19T00:51:40.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): my global_id is 64179 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b8003530 con 0x7f01d00a4e80 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1716067405 0 0) 0x7f01b8003530 con 0x7f01d00a4e80 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01d0142570 msgr2=0x7f01d0144960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f01d0142570 0x7f01d0144960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 >> v2:172.21.15.161:3300/0 conn(0x7f01d00a4aa0 msgr2=0x7f01d0071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 --2- 172.21.15.43:0/4089018843 >> v2:172.21.15.161:3300/0 conn(0x7f01d00a4aa0 0x7f01d0071c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d006f270 con 0x7f01d00a4e80 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f01c4001cf0 con 0x7f01d00a4e80 2024-04-19T00:51:40.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: dump: 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.323 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.324 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 >> v1:172.21.15.43:6789/0 conn(0x7f01d00a4e80 legacy=0x7f01d00a5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient: _add_conns ranks=[4,8,3,2,1,7,0,6,5] 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01b8004210 0x7f01b8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): picked mon.c con 0x7f01b8004210 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f01b8004da0 0x7f01b8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): picked mon.e con 0x7f01b8004da0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01b8009ab0 0x7f01b800bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): picked mon.g con 0x7f01b8009ab0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f01d006c4d0 con 0x7f01b8009ab0 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f01d006f830 con 0x7f01b8004210 2024-04-19T00:51:40.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f01b800c850 con 0x7f01b8004da0 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d551d640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01b8004210 0x7f01b8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f01b8004da0 0x7f01b8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): get_auth_request con 0x7f01b8004210 auth_method 0 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): get_auth_request con 0x7f01b8004da0 auth_method 0 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:40.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d5d1e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01b8009ab0 0x7f01b800bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): get_auth_request con 0x7f01b8009ab0 auth_method 0 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient(hunting): handle_auth_done global_id 64179 payload 293 2024-04-19T00:51:40.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 -- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f01b8004da0 msgr2=0x7f01b8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f01b8004da0 0x7f01b8005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 -- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01b8009ab0 msgr2=0x7f01b800bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01b8009ab0 0x7f01b800bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 monclient: found mon.c 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d6d20640 1 -- 172.21.15.43:0/4089018843 reap_dead start 2024-04-19T00:51:40.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 1 -- 172.21.15.43:0/4089018843 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b8003530 con 0x7f01b8004210 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d651f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 5 monclient: authenticate success, global_id 64179 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.304+0000 7f01d8783640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d651f640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01b8004210 0x7f01b8004810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009ab0 tx=0x7f01c801c9d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f6aed2cac46a9eda server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c80096b0 con 0x7f01b8004210 2024-04-19T00:51:40.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: dump: 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.332 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.333 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c80a6970 con 0x7f01b8004210 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 4 set_mon_vals no callback set 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 1 -- 172.21.15.43:0/4089018843 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c80a5430 con 0x7f01b8004210 2024-04-19T00:51:40.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d551d640 10 monclient: dump: 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.335 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: shutdown 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 -- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01b8004210 msgr2=0x7f01b8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01b8004210 0x7f01b8004810 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009ab0 tx=0x7f01c801c9d0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 -- 172.21.15.43:0/4089018843 shutdown_connections 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 --2- 172.21.15.43:0/4089018843 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f01b8004210 0x7f01b8004810 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 -- 172.21.15.43:0/4089018843 >> 172.21.15.43:0/4089018843 conn(0x7f01d00a6410 msgr2=0x7f01d006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 -- 172.21.15.43:0/4089018843 shutdown_connections 2024-04-19T00:51:40.336 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 -- 172.21.15.43:0/4089018843 wait complete. 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: monmap: 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.337 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 librados: starting msgr at 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 librados: starting objecter 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 Processor -- start 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 -- start start 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 librados: setting wanted keys 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 librados: calling monclient init 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: init 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 10 monclient: _add_conns ranks=[6,3,0,7,1,4,8,5,2] 2024-04-19T00:51:40.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.308+0000 7f01d8783640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01b8004210 0x7f01d01d5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): picked mon.noname-d con 0x7f01b8004210 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01d013b4c0 0x7f01d013b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): picked mon.noname-g con 0x7f01d013b4c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d009ca70 0x7f01d009cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): picked mon.noname-a con 0x7f01d009ca70 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f01d013d3d0 con 0x7f01d009ca70 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f01d013d250 con 0x7f01d013b4c0 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f01d013d550 con 0x7f01b8004210 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): authenticate will time out at 1370.440082s 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01b8004210 0x7f01d01d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d009ca70 0x7f01d009cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d009ca70 0x7f01d009cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38132/0 (socket says 172.21.15.43:38132) 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 -- 172.21.15.43:0/556870696 learned_addr learned my addr 172.21.15.43:0/556870696 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): get_auth_request con 0x7f01b8004210 auth_method 0 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): get_auth_request con 0x7f01d009ca70 auth_method 0 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01d013b4c0 0x7f01d013b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 10 monclient(hunting): get_auth_request con 0x7f01d013b4c0 auth_method 0 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient(hunting): handle_auth_done global_id 64188 payload 824 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01b8004210 msgr2=0x7f01d01d5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f01b8004210 0x7f01d01d5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01d013b4c0 msgr2=0x7f01d013b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01d013b4c0 0x7f01d013b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 monclient: found mon.noname-a 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d0128ba0 con 0x7f01d009ca70 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d6d20640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d009ca70 0x7f01d009cef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01d00a0040 tx=0x7f01c0011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 client.?.objecter ms_handle_connect 0x7f01d009ca70 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c0019020 con 0x7f01d009ca70 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: dump: 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.342 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c0004300 con 0x7f01d009ca70 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c00044b0 con 0x7f01d009ca70 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 5 monclient: authenticate success, global_id 64188 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient: _renew_subs 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d009f250 con 0x7f01d009ca70 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d4d1c640 4 set_mon_vals no callback set 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: dump: 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.344 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 client.64188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient: _renew_subs 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d01288c0 con 0x7f01d009ca70 2024-04-19T00:51:40.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 librados: init done 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 0x7f01d00a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01d006d140 con 0x7f01d009d430 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f01c001c020 con 0x7f01d009ca70 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f01bc038e10 0x7f01bc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 0x7f01d00a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 5769+0+0 (secure 0 0 0) 0x7f01c0049460 con 0x7f01d009ca70 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 client.64188.objecter ms_dispatch 0x7f01d0000ba0 osd_map(317..317 src has 1..317) 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 3 client.64188.objecter handle_osd_map got epochs [317,317] > 0 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 3 client.64188.objecter handle_osd_map decoding full epoch 317 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient: get_auth_request con 0x7f01d009d430 auth_method 0 2024-04-19T00:51:40.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 20 client.64188.objecter dump_active .. 0 homeless 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f01bc038e10 0x7f01bc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 10 monclient: get_auth_request con 0x7f01bc038e10 auth_method 0 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 10 monclient(hunting): handle_auth_done global_id 64144 payload 824 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01d009efa0 con 0x7f01d009d430 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 client.64188.objecter ms_handle_connect 0x7f01d009d430 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 client.64188.objecter resend_mon_ops 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d651f640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 0x7f01d00a0490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01b8004790 tx=0x7f01b8005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c80bc070 con 0x7f01d009d430 2024-04-19T00:51:40.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01d5d1e640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f01bc038e10 0x7f01bc03b2d0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f01d01283f0 tx=0x7f01c4006040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 monclient: dump: 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.348 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.312+0000 7f01ceffd640 10 client.64188.objecter ms_handle_connect 0x7f01bc038e10 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f01c8009770 con 0x7f01d009d430 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 msgr2=0x7f01d00a0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 0x7f01d00a0490 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01b8004790 tx=0x7f01b8005840 comp rx=0 tx=0).stop 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 msgr2=0x7f01d00a0490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.316+0000 7f01ceffd640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d009d430 0x7f01d00a0490 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f01b8004790 tx=0x7f01b8005840 comp rx=0 tx=0).stop 2024-04-19T00:51:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d8783640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d8783640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 0x7f01d00a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d8783640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01d009e780 con 0x7f01d00a09d0 2024-04-19T00:51:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 0x7f01d00a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient: get_auth_request con 0x7f01d00a09d0 auth_method 0 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 10 monclient(hunting): handle_auth_done global_id 64153 payload 824 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.320+0000 7f01d651f640 1 -- 172.21.15.43:0/556870696 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01d0076440 con 0x7f01d00a09d0 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d651f640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 0x7f01d00a0de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009ab0 tx=0x7f01c8009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 client.64188.objecter ms_handle_connect 0x7f01d00a09d0 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 client.64188.objecter resend_mon_ops 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01c80b8460 con 0x7f01d00a09d0 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 monclient: dump: 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.353 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f01c8003ba0 con 0x7f01d00a09d0 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 msgr2=0x7f01d00a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 0x7f01d00a0de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009ab0 tx=0x7f01c8009190 comp rx=0 tx=0).stop 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 msgr2=0x7f01d00a0de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01ceffd640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f01d00a09d0 0x7f01d00a0de0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009ab0 tx=0x7f01c8009190 comp rx=0 tx=0).stop 2024-04-19T00:51:40.355 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:40.355 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:40.355 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:40.355 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:40.355 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 26587, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:40.356 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.357 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.358 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:40.359 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:40.360 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.361 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.362 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.363 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.365 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:40.367 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.368 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:40.372 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:40.374 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.375 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.377 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.378 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.379 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.379 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:40.379 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:40.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 10 librados: watch_flush enter 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 10 librados: watch_flush exit 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 20 client.64188.objecter shutdown clearing up homeless session... 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 10 client.64188.objecter successfully canceled tick 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f01bc038e10 msgr2=0x7f01bc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f01bc038e10 0x7f01bc03b2d0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7f01d01283f0 tx=0x7f01c4006040 comp rx=0 tx=0).stop 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 10 monclient: shutdown 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d009ca70 msgr2=0x7f01d009cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 --2- 172.21.15.43:0/556870696 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f01d009ca70 0x7f01d009cef0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f01d00a0040 tx=0x7f01c0011920 comp rx=0 tx=0).stop 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d6d20640 1 -- 172.21.15.43:0/556870696 reap_dead start 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d6d20640 1 -- 172.21.15.43:0/556870696 reap_dead start 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 shutdown_connections 2024-04-19T00:51:40.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 >> 172.21.15.43:0/556870696 conn(0x7f01d00a6410 msgr2=0x7f01d013c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 shutdown_connections 2024-04-19T00:51:40.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 -- 172.21.15.43:0/556870696 wait complete. 2024-04-19T00:51:40.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.324+0000 7f01d8783640 1 librados: shutdown 2024-04-19T00:51:40.382 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:40.382 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:51:40.382 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:51:40.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient: get_monmap_and_config 2024-04-19T00:51:40.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient: monmap: 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:40.511103+0000 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:40.511103+0000 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.509 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.510 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.510 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:40.510 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:40.510 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 1 Processor -- start 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 1 -- start start 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient: init 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient: _add_conns ranks=[0,9,1,8,4,3,6,10,7,5,2] 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17ec0a1e00 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17ec0a1190 0x7f17ec0a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient(hunting): picked mon.noname-a con 0x7f17ec0a1190 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:40.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f17ec13eac0 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17ec06c690 con 0x7f17ec0a1e00 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17ec0735a0 con 0x7f17ec13eac0 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f17ec09fff0 con 0x7f17ec0a1190 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17f2922640 10 monclient(hunting): authenticate will time out at 1370.638938s 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17ec0a1190 0x7f17ec0a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17ec0a1190 0x7f17ec0a3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38140/0 (socket says 172.21.15.43:38140) 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 -- 172.21.15.43:0/3587765557 learned_addr learned my addr 172.21.15.43:0/3587765557 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request con 0x7f17ec0a1190 auth_method 0 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17ebfff640 1 --1- 172.21.15.43:0/3587765557 >> v1:172.21.15.43:6789/0 conn(0x7f17ec0a1e00 0x7f17ec0a38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44542/0 (socket says 172.21.15.43:44542) 2024-04-19T00:51:40.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_done global_id 64197 payload 274 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 -- 172.21.15.43:0/3587765557 >> v1:172.21.15.161:6789/0 conn(0x7f17ec13eac0 legacy=0x7f17ec140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 -- 172.21.15.43:0/3587765557 >> v1:172.21.15.43:6789/0 conn(0x7f17ec0a1e00 legacy=0x7f17ec0a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 monclient: found mon.noname-a 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.508+0000 7f17eb7fe640 1 -- 172.21.15.43:0/3587765557 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec06c330 con 0x7f17ec0a1190 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eb7fe640 1 --2- 172.21.15.43:0/3587765557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17ec0a1190 0x7f17ec0a3fe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009870 tx=0x7f17e001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5da0b5aa4d95f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 1 -- 172.21.15.43:0/3587765557 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e00b0070 con 0x7f17ec0a1190 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: dump: 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.521 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 1 -- 172.21.15.43:0/3587765557 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e00aa810 con 0x7f17ec0a1190 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 1 -- 172.21.15.43:0/3587765557 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e00a9300 con 0x7f17ec0a1190 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 5 monclient: authenticate success, global_id 64197 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 4 set_mon_vals no callback set 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17eaffd640 10 monclient: dump: 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.523 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 10 monclient: shutdown 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 -- 172.21.15.43:0/3587765557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17ec0a1190 msgr2=0x7f17ec0a3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 --2- 172.21.15.43:0/3587765557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17ec0a1190 0x7f17ec0a3fe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f17e0009870 tx=0x7f17e001c420 comp rx=0 tx=0).stop 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 -- 172.21.15.43:0/3587765557 shutdown_connections 2024-04-19T00:51:40.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 --2- 172.21.15.43:0/3587765557 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17ec0a1190 0x7f17ec0a3fe0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 -- 172.21.15.43:0/3587765557 >> 172.21.15.43:0/3587765557 conn(0x7f17ec0a84c0 msgr2=0x7f17ec0a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 -- 172.21.15.43:0/3587765557 shutdown_connections 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 -- 172.21.15.43:0/3587765557 wait complete. 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 10 monclient: monmap: 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.525 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 librados: starting msgr at 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 librados: starting objecter 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 Processor -- start 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 -- start start 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 librados: setting wanted keys 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 1 librados: calling monclient init 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.512+0000 7f17f2922640 10 monclient: init 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient: _add_conns ranks=[5,8,3,7,6,0,4,1,2] 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17ec0a1190 0x7f17ec09ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): picked mon.noname-h con 0x7f17ec0a1190 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17ec0a1e00 0x7f17ec09bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): picked mon.noname-e con 0x7f17ec0a1e00 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec13eac0 0x7f17ec09c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): picked mon.noname-g con 0x7f17ec13eac0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f17ec0a21e0 con 0x7f17ec13eac0 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f17ec0a4730 con 0x7f17ec0a1190 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f17ec0a2360 con 0x7f17ec0a1e00 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): authenticate will time out at 1370.644149s 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17ec0a1e00 0x7f17ec09bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17ec0a1e00 0x7f17ec09bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:53582/0 (socket says 172.21.15.43:53582) 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 -- 172.21.15.43:0/217270871 learned_addr learned my addr 172.21.15.43:0/217270871 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request con 0x7f17ec0a1e00 auth_method 0 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17ec0a1190 0x7f17ec09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f0ebf640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec13eac0 0x7f17ec09c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient(hunting): handle_auth_done global_id 23066 payload 824 2024-04-19T00:51:40.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17ec0a1190 msgr2=0x7f17ec09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17ec0a1190 0x7f17ec09ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec13eac0 msgr2=0x7f17ec09c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec13eac0 0x7f17ec09c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 monclient: found mon.noname-e 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec073270 con 0x7f17ec0a1e00 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient: get_auth_request con 0x7f17ec0a1190 auth_method 0 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f0ebf640 10 monclient: get_auth_request con 0x7f17ec13eac0 auth_method 0 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17ec0a1190 0x7f17ec09ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f0ebf640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec13eac0 0x7f17ec09c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:40.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17eb7fe640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17ec0a1e00 0x7f17ec09bfb0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a050 tx=0x7f17e00191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 client.?.objecter ms_handle_connect 0x7f17ec0a1e00 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e00b0040 con 0x7f17ec0a1e00 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: dump: 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.530 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e00b9070 con 0x7f17ec0a1e00 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e00044b0 con 0x7f17ec0a1e00 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: dump: 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.532 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e9ffb640 4 set_mon_vals no callback set 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 5 monclient: authenticate success, global_id 23066 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient: _renew_subs 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec13faa0 con 0x7f17ec0a1e00 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 client.23066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient: _renew_subs 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec13f7c0 con 0x7f17ec0a1e00 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 librados: init done 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 0x7f17ec1d0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f17ec141ca0 con 0x7f17ec1d0be0 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f17e0004040 con 0x7f17ec0a1e00 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17d8038e10 0x7f17d803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(317..317 src has 1..317) ==== 5769+0+0 (secure 0 0 0) 0x7f17e00e3060 con 0x7f17ec0a1e00 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 client.23066.objecter ms_dispatch 0x7f17ec000ba0 osd_map(317..317 src has 1..317) 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 3 client.23066.objecter handle_osd_map got epochs [317,317] > 0 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 3 client.23066.objecter handle_osd_map decoding full epoch 317 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 0x7f17ec1d0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient: get_auth_request con 0x7f17ec1d0be0 auth_method 0 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 20 client.23066.objecter dump_active .. 0 homeless 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f0ebf640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17d8038e10 0x7f17d803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f0ebf640 10 monclient: get_auth_request con 0x7f17d8038e10 auth_method 0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 10 monclient(hunting): handle_auth_done global_id 19029 payload 824 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17ec073050 con 0x7f17ec1d0be0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 client.23066.objecter ms_handle_connect 0x7f17ec1d0be0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 client.23066.objecter resend_mon_ops 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17ebfff640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 0x7f17ec1d0fc0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001ec0 tx=0x7f17dc00d950 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 client.23066.objecter ms_handle_connect 0x7f17d8038e10 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17f0ebf640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17d8038e10 0x7f17d803b2d0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007fd0 tx=0x7f17e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17dc018070 con 0x7f17ec1d0be0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.516+0000 7f17e8ff9640 10 monclient: dump: 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.536 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17dc0072e0 con 0x7f17ec1d0be0 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 msgr2=0x7f17ec1d0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 0x7f17ec1d0fc0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001ec0 tx=0x7f17dc00d950 comp rx=0 tx=0).stop 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 msgr2=0x7f17ec1d0fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.520+0000 7f17e8ff9640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d0be0 0x7f17ec1d0fc0 secure :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001ec0 tx=0x7f17dc00d950 comp rx=0 tx=0).stop 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17f2922640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17f2922640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 0x7f17ec0a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17f2922640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f17ec09dff0 con 0x7f17ec1d1500 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 0x7f17ec0a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient: get_auth_request con 0x7f17ec1d1500 auth_method 0 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.524+0000 7f17ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17ebfff640 10 monclient(hunting): handle_auth_done global_id 19038 payload 824 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17ebfff640 1 -- 172.21.15.43:0/217270871 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17ec06c8d0 con 0x7f17ec1d1500 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17ebfff640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 0x7f17ec0a5cc0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001ec0 tx=0x7f17dc000ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 client.23066.objecter ms_handle_connect 0x7f17ec1d1500 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 client.23066.objecter resend_mon_ops 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17dc018040 con 0x7f17ec1d1500 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 monclient: dump: 2024-04-19T00:51:40.540 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.541 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f17dc013470 con 0x7f17ec1d1500 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 msgr2=0x7f17ec0a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 0x7f17ec0a5cc0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001ec0 tx=0x7f17dc000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 msgr2=0x7f17ec0a5cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17e8ff9640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17ec1d1500 0x7f17ec0a5cc0 secure :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f17dc001ec0 tx=0x7f17dc000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.542 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 444008, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:40.543 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.544 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.545 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.546 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.547 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.549 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:40.552 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:40.553 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.556 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:40.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.559 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:40.561 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.563 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:40.564 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:40.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 10 librados: watch_flush enter 2024-04-19T00:51:40.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 10 librados: watch_flush exit 2024-04-19T00:51:40.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 20 client.23066.objecter shutdown clearing up homeless session... 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 10 client.23066.objecter successfully canceled tick 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17d8038e10 msgr2=0x7f17d803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17d8038e10 0x7f17d803b2d0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7f17e4007fd0 tx=0x7f17e4009040 comp rx=0 tx=0).stop 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 10 monclient: shutdown 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17ec0a1e00 msgr2=0x7f17ec09bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 --2- 172.21.15.43:0/217270871 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17ec0a1e00 0x7f17ec09bfb0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f17e001a050 tx=0x7f17e00191c0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f0ebf640 1 -- 172.21.15.43:0/217270871 reap_dead start 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f0ebf640 1 -- 172.21.15.43:0/217270871 reap_dead start 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 shutdown_connections 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 >> 172.21.15.43:0/217270871 conn(0x7f17ec0a84c0 msgr2=0x7f17ec139190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 shutdown_connections 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 -- 172.21.15.43:0/217270871 wait complete. 2024-04-19T00:51:40.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.528+0000 7f17f2922640 1 librados: shutdown 2024-04-19T00:51:40.568 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:40.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:51:40.568 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:51:40.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.688+0000 7fe1118a1640 10 monclient: get_monmap_and_config 2024-04-19T00:51:40.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.688+0000 7fe1118a1640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.688+0000 7fe1118a1640 10 monclient: monmap: 2024-04-19T00:51:40.692 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.692 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.692 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:40.694351+0000 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:40.694351+0000 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:40.693 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 Processor -- start 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 -- start start 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient: init 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient: _add_conns ranks=[1,6,5,8,3,0,10,9,4,2,7] 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe10c138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): picked mon.noname-d con 0x7fe10c139540 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c0a5bb0 0x7fe10c140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): picked mon.noname-h con 0x7fe10c0a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:40.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10c0a0d40 con 0x7fe10c138a50 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe10c0a1020 con 0x7fe10c0a5bb0 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fe10c0a0780 con 0x7fe10c139540 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe1118a1640 10 monclient(hunting): authenticate will time out at 1370.821859s 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:52654/0 (socket says 172.21.15.43:52654) 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 -- 172.21.15.43:0/3489728931 learned_addr learned my addr 172.21.15.43:0/3489728931 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): get_auth_request con 0x7fe10c139540 auth_method 0 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10b7fe640 1 --2- 172.21.15.43:0/3489728931 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c0a5bb0 0x7fe10c140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10b7fe640 10 monclient(hunting): get_auth_request con 0x7fe10c0a5bb0 auth_method 0 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_done global_id 64176 payload 274 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 -- 172.21.15.43:0/3489728931 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c0a5bb0 msgr2=0x7fe10c140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 --2- 172.21.15.43:0/3489728931 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c0a5bb0 0x7fe10c140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 -- 172.21.15.43:0/3489728931 >> v1:172.21.15.161:6789/0 conn(0x7fe10c138a50 legacy=0x7fe10c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 monclient: found mon.noname-d 2024-04-19T00:51:40.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:40.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.692+0000 7fe10a7fc640 1 -- 172.21.15.43:0/3489728931 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c0a0a60 con 0x7fe10c139540 2024-04-19T00:51:40.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 1 --2- 172.21.15.43:0/3489728931 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13dfe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40098a0 tx=0x7fe0f401c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=cdabaf43ce53d91d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 1 -- 172.21.15.43:0/3489728931 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0f40b1070 con 0x7fe10c139540 2024-04-19T00:51:40.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:40.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: dump: 2024-04-19T00:51:40.706 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.707 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.708 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.708 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.708 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 5 monclient: authenticate success, global_id 64176 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 1 -- 172.21.15.43:0/3489728931 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f40aa7b0 con 0x7fe10c139540 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 4 set_mon_vals no callback set 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 1 -- 172.21.15.43:0/3489728931 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0f40aae50 con 0x7fe10c139540 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe109ffb640 10 monclient: dump: 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.710 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.711 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: shutdown 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- 172.21.15.43:0/3489728931 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 msgr2=0x7fe10c13dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 --2- 172.21.15.43:0/3489728931 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13dfe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40098a0 tx=0x7fe0f401c440 comp rx=0 tx=0).stop 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- 172.21.15.43:0/3489728931 shutdown_connections 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 --2- 172.21.15.43:0/3489728931 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c0a5bb0 0x7fe10c140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 --2- 172.21.15.43:0/3489728931 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13dfe0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- 172.21.15.43:0/3489728931 >> 172.21.15.43:0/3489728931 conn(0x7fe10c0a7cb0 msgr2=0x7fe10c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- 172.21.15.43:0/3489728931 shutdown_connections 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- 172.21.15.43:0/3489728931 wait complete. 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: monmap: 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.713 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 librados: starting msgr at 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 librados: starting objecter 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 Processor -- start 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- start start 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 librados: setting wanted keys 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 librados: calling monclient init 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: init 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient: _add_conns ranks=[7,8,6,5,0,2,3,4,1] 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe10c0a5bb0 0x7fe10c1d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): picked mon.noname-i con 0x7fe10c0a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe10c138a50 0x7fe10c137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): picked mon.noname-e con 0x7fe10c138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): picked mon.noname-d con 0x7fe10c139540 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fe10c143620 con 0x7fe10c139540 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe10c1434a0 con 0x7fe10c0a5bb0 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe10c1437a0 con 0x7fe10c138a50 2024-04-19T00:51:40.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe1118a1640 10 monclient(hunting): authenticate will time out at 1370.827074s 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe10c138a50 0x7fe10c137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:52668/0 (socket says 172.21.15.43:52668) 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 -- 172.21.15.43:0/1071100594 learned_addr learned my addr 172.21.15.43:0/1071100594 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): get_auth_request con 0x7fe10c139540 auth_method 0 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): get_auth_request con 0x7fe10c138a50 auth_method 0 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10affd640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe10c0a5bb0 0x7fe10c1d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10affd640 10 monclient(hunting): get_auth_request con 0x7fe10c0a5bb0 auth_method 0 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient(hunting): handle_auth_done global_id 64185 payload 824 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe10c138a50 msgr2=0x7fe10c137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe10c138a50 0x7fe10c137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe10c0a5bb0 msgr2=0x7fe10c1d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe10c0a5bb0 0x7fe10c1d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 monclient: found mon.noname-d 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c09fc90 con 0x7fe10c139540 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10a7fc640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe10c138a50 0x7fe10c137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe10b7fe640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13cb90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe10400d830 tx=0x7fe10400dcf0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe103fff640 10 client.?.objecter ms_handle_connect 0x7fe10c139540 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.696+0000 7fe103fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe10400bad0 con 0x7fe10c139540 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: dump: 2024-04-19T00:51:40.718 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.719 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe104019070 con 0x7fe10c139540 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 5 monclient: authenticate success, global_id 64185 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 monclient: _renew_subs 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10c09ee60 con 0x7fe10c139540 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe10400bcb0 con 0x7fe10c139540 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1097fa640 4 set_mon_vals no callback set 2024-04-19T00:51:40.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 client.64185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: dump: 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.721 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 monclient: _renew_subs 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10c1224f0 con 0x7fe10c139540 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 1 librados: init done 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 0x7fe10c13d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe10c06d140 con 0x7fe10c1380a0 2024-04-19T00:51:40.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe10401e460 con 0x7fe10c139540 2024-04-19T00:51:40.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe0ec038e10 0x7fe0ec03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 0x7fe10c13d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(317..317 src has 1..317) ==== 5769+0+0 (secure 0 0 0) 0x7fe10400f040 con 0x7fe10c139540 2024-04-19T00:51:40.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 client.64185.objecter ms_dispatch 0x7fe10c000ba0 osd_map(317..317 src has 1..317) 2024-04-19T00:51:40.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 3 client.64185.objecter handle_osd_map got epochs [317,317] > 0 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 3 client.64185.objecter handle_osd_map decoding full epoch 317 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient: get_auth_request con 0x7fe10c1380a0 auth_method 0 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 20 client.64185.objecter dump_active .. 0 homeless 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10a7fc640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe0ec038e10 0x7fe0ec03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10a7fc640 10 monclient: get_auth_request con 0x7fe0ec038e10 auth_method 0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 10 monclient(hunting): handle_auth_done global_id 44897 payload 824 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe10c09eaa0 con 0x7fe10c1380a0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 client.64185.objecter ms_handle_connect 0x7fe10c1380a0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 client.64185.objecter resend_mon_ops 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10affd640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 0x7fe10c13d0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e840 tx=0x7fe0fc00ed00 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 client.64185.objecter ms_handle_connect 0x7fe0ec038e10 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe10a7fc640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe0ec038e10 0x7fe0ec03b2d0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40ae000 tx=0x7fe0f40a6070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0fc0050d0 con 0x7fe10c1380a0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: dump: 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.725 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe0fc018070 con 0x7fe10c1380a0 2024-04-19T00:51:40.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:40.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 msgr2=0x7fe10c13d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 0x7fe10c13d0d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e840 tx=0x7fe0fc00ed00 comp rx=0 tx=0).stop 2024-04-19T00:51:40.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 msgr2=0x7fe10c13d0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.700+0000 7fe103fff640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c1380a0 0x7fe10c13d0d0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00e840 tx=0x7fe0fc00ed00 comp rx=0 tx=0).stop 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe1118a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe1118a1640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 0x7fe10c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe1118a1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe10c13da20 con 0x7fe10c13d610 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 0x7fe10c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient: get_auth_request con 0x7fe10c13d610 auth_method 0 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 10 monclient(hunting): handle_auth_done global_id 44906 payload 824 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 1 -- 172.21.15.43:0/1071100594 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe10c1387e0 con 0x7fe10c13d610 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe10affd640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 0x7fe10c0a3650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009010 tx=0x7fe0fc0014b0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe103fff640 10 client.64185.objecter ms_handle_connect 0x7fe10c13d610 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe103fff640 10 client.64185.objecter resend_mon_ops 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe0fc005540 con 0x7fe10c13d610 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe103fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.708+0000 7fe103fff640 10 monclient: dump: 2024-04-19T00:51:40.730 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.731 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7fe0fc0050d0 con 0x7fe10c13d610 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 msgr2=0x7fe10c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 0x7fe10c0a3650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009010 tx=0x7fe0fc0014b0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 msgr2=0x7fe10c0a3650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe103fff640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe10c13d610 0x7fe10c0a3650 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009010 tx=0x7fe0fc0014b0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:40.732 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 300438, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:40.733 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.734 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.735 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:40.736 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.737 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.738 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.739 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.740 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.741 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.742 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.743 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:40.746 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.749 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.751 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:40.752 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:40.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.754 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:40.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.756 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.757 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.758 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.758 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:40.758 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 10 librados: watch_flush enter 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 10 librados: watch_flush exit 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 20 client.64185.objecter shutdown clearing up homeless session... 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 10 client.64185.objecter successfully canceled tick 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe0ec038e10 msgr2=0x7fe0ec03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe0ec038e10 0x7fe0ec03b2d0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7fe0f40ae000 tx=0x7fe0f40a6070 comp rx=0 tx=0).stop 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 10 monclient: shutdown 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 msgr2=0x7fe10c13cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe10b7fe640 1 -- 172.21.15.43:0/1071100594 reap_dead start 2024-04-19T00:51:40.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13cb90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe10400d830 tx=0x7fe10400dcf0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 shutdown_connections 2024-04-19T00:51:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 --2- 172.21.15.43:0/1071100594 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe10c139540 0x7fe10c13cb90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 >> 172.21.15.43:0/1071100594 conn(0x7fe10c0a7cb0 msgr2=0x7fe10c0a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 shutdown_connections 2024-04-19T00:51:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 -- 172.21.15.43:0/1071100594 wait complete. 2024-04-19T00:51:40.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.712+0000 7fe1118a1640 1 librados: shutdown 2024-04-19T00:51:40.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:40.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:51:40.762 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient: get_monmap_and_config 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient: monmap: 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:40.883309+0000 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:40.883309+0000 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.881 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:40.882 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 Processor -- start 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 -- start start 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient: init 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient: _add_conns ranks=[10,9,1,3,5,7,0,2,6,8,4] 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f93f0138cd0 0x7f93f01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): picked mon.noname-f con 0x7f93f0138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93f00a5b80 0x7f93f01395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): picked mon.noname-a con 0x7f93f00a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:40.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f93f01358b0 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93f00a3dc0 con 0x7f93f01358b0 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f93f00a4410 con 0x7f93f00a5b80 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f93f0139b30 con 0x7f93f0138cd0 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f7b85640 10 monclient(hunting): authenticate will time out at 1371.010441s 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93f00a5b80 0x7f93f01395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f93f00a5b80 0x7f93f01395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:38148/0 (socket says 172.21.15.43:38148) 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 -- 172.21.15.43:0/2594101335 learned_addr learned my addr 172.21.15.43:0/2594101335 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): get_auth_request con 0x7f93f00a5b80 auth_method 0 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5921640 1 --2- 172.21.15.43:0/2594101335 >> v2:172.21.15.161:3300/0 conn(0x7f93f0138cd0 0x7f93f01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5921640 10 monclient(hunting): get_auth_request con 0x7f93f0138cd0 auth_method 0 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5921640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5921640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient(hunting): handle_auth_done global_id 64215 payload 274 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 -- 172.21.15.43:0/2594101335 >> v2:172.21.15.161:3300/0 conn(0x7f93f0138cd0 msgr2=0x7f93f01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 --2- 172.21.15.43:0/2594101335 >> v2:172.21.15.161:3300/0 conn(0x7f93f0138cd0 0x7f93f01390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 -- 172.21.15.43:0/2594101335 >> v1:172.21.15.161:6789/0 conn(0x7f93f01358b0 legacy=0x7f93f0135ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:40.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 monclient: found mon.noname-a 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.880+0000 7f93f5120640 1 -- 172.21.15.43:0/2594101335 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f00a40a0 con 0x7f93f00a5b80 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f5120640 1 --2- 172.21.15.43:0/2594101335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93f00a5b80 0x7f93f01395f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93f00a7450 tx=0x7f93e401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bea312928dc39d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 1 -- 172.21.15.43:0/2594101335 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93e40b2070 con 0x7f93f00a5b80 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: dump: 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.896 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 5 monclient: authenticate success, global_id 64215 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 1 -- 172.21.15.43:0/2594101335 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e40a9070 con 0x7f93f00a5b80 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 4 set_mon_vals no callback set 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 1 -- 172.21.15.43:0/2594101335 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93e40abc40 con 0x7f93f00a5b80 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f491f640 10 monclient: dump: 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.898 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.899 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: shutdown 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- 172.21.15.43:0/2594101335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93f00a5b80 msgr2=0x7f93f01395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 --2- 172.21.15.43:0/2594101335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93f00a5b80 0x7f93f01395f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f93f00a7450 tx=0x7f93e401c380 comp rx=0 tx=0).stop 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- 172.21.15.43:0/2594101335 shutdown_connections 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 --2- 172.21.15.43:0/2594101335 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f93f00a5b80 0x7f93f01395f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 --2- 172.21.15.43:0/2594101335 >> v2:172.21.15.161:3300/0 conn(0x7f93f0138cd0 0x7f93f01390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- 172.21.15.43:0/2594101335 >> 172.21.15.43:0/2594101335 conn(0x7f93f00ab2f0 msgr2=0x7f93f00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- 172.21.15.43:0/2594101335 shutdown_connections 2024-04-19T00:51:40.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- 172.21.15.43:0/2594101335 wait complete. 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: build_initial_monmap 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: monmap: 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:40.901 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 librados: starting msgr at 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 librados: starting objecter 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 Processor -- start 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- start start 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 librados: setting wanted keys 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 librados: calling monclient init 2024-04-19T00:51:40.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: init 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient: _add_conns ranks=[3,5,2,6,1,0,4,7,8] 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93f00a5b80 0x7f93f012aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): picked mon.noname-g con 0x7f93f00a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93f01358b0 0x7f93f012afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): picked mon.noname-h con 0x7f93f01358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93f0138cd0 0x7f93f0132580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): picked mon.noname-b con 0x7f93f0138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f93f0143c30 con 0x7f93f0138cd0 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f93f0143ab0 con 0x7f93f00a5b80 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f93f0143db0 con 0x7f93f01358b0 2024-04-19T00:51:40.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f7b85640 10 monclient(hunting): authenticate will time out at 1371.015624s 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93f0138cd0 0x7f93f0132580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93f0138cd0 0x7f93f0132580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:53754/0 (socket says 172.21.15.43:53754) 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 1 -- 172.21.15.43:0/3136971775 learned_addr learned my addr 172.21.15.43:0/3136971775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 10 monclient(hunting): get_auth_request con 0x7f93f0138cd0 auth_method 0 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f6122640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.884+0000 7f93f5921640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93f00a5b80 0x7f93f012aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93f01358b0 0x7f93f012afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): get_auth_request con 0x7f93f00a5b80 auth_method 0 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): get_auth_request con 0x7f93f01358b0 auth_method 0 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 10 monclient(hunting): handle_auth_done global_id 23321 payload 824 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93f01358b0 msgr2=0x7f93f012afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93f01358b0 0x7f93f012afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93f00a5b80 msgr2=0x7f93f012aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f93f00a5b80 0x7f93f012aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 10 monclient: _finish_hunting 0 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 monclient: found mon.noname-b 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f009c280 con 0x7f93f0138cd0 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f6122640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93f0138cd0 0x7f93f0132580 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f93e800c780 tx=0x7f93e800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 client.?.objecter ms_handle_connect 0x7f93f0138cd0 2024-04-19T00:51:40.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93e8004de0 con 0x7f93f0138cd0 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: dump: 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.907 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93e800a850 con 0x7f93f0138cd0 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 5 monclient: authenticate success, global_id 23321 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 monclient: _renew_subs 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:40.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f009cc90 con 0x7f93f0138cd0 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93e8005030 con 0x7f93f0138cd0 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e3fff640 4 set_mon_vals no callback set 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 client.23321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: dump: 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.909 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.910 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 monclient: _renew_subs 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f0133da0 con 0x7f93f0138cd0 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 1 librados: init done 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 0x7f93f0132ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f93f006d140 con 0x7f93f0132ac0 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f93e8004970 con 0x7f93f0138cd0 2024-04-19T00:51:40.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 0x7f93f0132ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93d4038e10 0x7f93d403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient: get_auth_request con 0x7f93f0132ac0 auth_method 0 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(318..318 src has 1..318) ==== 6141+0+0 (secure 0 0 0) 0x7f93e8049550 con 0x7f93f0138cd0 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 client.23321.objecter ms_dispatch 0x7f93f0000ba0 osd_map(318..318 src has 1..318) 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 3 client.23321.objecter handle_osd_map got epochs [318,318] > 0 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 3 client.23321.objecter handle_osd_map decoding full epoch 318 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 20 client.23321.objecter dump_active .. 0 homeless 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93d4038e10 0x7f93d403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 10 monclient: get_auth_request con 0x7f93d4038e10 auth_method 0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 10 monclient(hunting): handle_auth_done global_id 19618 payload 824 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93f00693b0 con 0x7f93f0132ac0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 client.23321.objecter ms_handle_connect 0x7f93f0132ac0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 client.23321.objecter resend_mon_ops 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5921640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 0x7f93f0132ea0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f93ec010820 tx=0x7f93ec010ce0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 client.23321.objecter ms_handle_connect 0x7f93d4038e10 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93f5120640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93d4038e10 0x7f93d403b2d0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f93e40af000 tx=0x7f93e40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93ec00eea0 con 0x7f93f0132ac0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.888+0000 7f93e27fc640 10 monclient: dump: 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.913 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f93ec00ea30 con 0x7f93f0132ac0 2024-04-19T00:51:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 msgr2=0x7f93f0132ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 0x7f93f0132ea0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f93ec010820 tx=0x7f93ec010ce0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 msgr2=0x7f93f0132ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.892+0000 7f93e27fc640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f0132ac0 0x7f93f0132ea0 secure :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f93ec010820 tx=0x7f93ec010ce0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f7b85640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f7b85640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 0x7f93f012b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f7b85640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f93f0068240 con 0x7f93f01333e0 2024-04-19T00:51:40.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 0x7f93f012b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient: get_auth_request con 0x7f93f01333e0 auth_method 0 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 10 monclient(hunting): handle_auth_done global_id 19627 payload 824 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 1 -- 172.21.15.43:0/3136971775 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93f00a2ff0 con 0x7f93f01333e0 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93f5921640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 0x7f93f012b910 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f93ec010820 tx=0x7f93ec0046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93e27fc640 10 client.23321.objecter ms_handle_connect 0x7f93f01333e0 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93e27fc640 10 client.23321.objecter resend_mon_ops 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93ec003e00 con 0x7f93f01333e0 2024-04-19T00:51:40.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93e27fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.896+0000 7f93e27fc640 10 monclient: dump: 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:40.918 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f93ec003990 con 0x7f93f01333e0 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 msgr2=0x7f93f012b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 0x7f93f012b910 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f93ec010820 tx=0x7f93ec0046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 msgr2=0x7f93f012b910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:40.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93e27fc640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93f01333e0 0x7f93f012b910 secure :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f93ec010820 tx=0x7f93ec0046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 444356, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:40.920 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.921 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.922 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:40.923 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:40.924 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:40.925 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:40.926 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.928 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.929 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.930 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.931 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.932 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.934 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.935 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.936 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.937 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:40.939 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:40.940 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.941 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:40.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:40.943 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 10 librados: watch_flush enter 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 10 librados: watch_flush exit 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 20 client.23321.objecter shutdown clearing up homeless session... 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 10 client.23321.objecter successfully canceled tick 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93d4038e10 msgr2=0x7f93d403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f93d4038e10 0x7f93d403b2d0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f93e40af000 tx=0x7f93e40a7000 comp rx=0 tx=0).stop 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 10 monclient: shutdown 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93f0138cd0 msgr2=0x7f93f0132580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:40.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 --2- 172.21.15.43:0/3136971775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93f0138cd0 0x7f93f0132580 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f93e800c780 tx=0x7f93e800cc40 comp rx=0 tx=0).stop 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f6122640 1 -- 172.21.15.43:0/3136971775 reap_dead start 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f6122640 1 -- 172.21.15.43:0/3136971775 reap_dead start 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 shutdown_connections 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 >> 172.21.15.43:0/3136971775 conn(0x7f93f00ab2f0 msgr2=0x7f93f00a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 shutdown_connections 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 -- 172.21.15.43:0/3136971775 wait complete. 2024-04-19T00:51:40.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:40.900+0000 7f93f7b85640 1 librados: shutdown 2024-04-19T00:51:40.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:40.946 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T00:51:40.947 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.060+0000 7f92195bd640 10 monclient: get_monmap_and_config 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.060+0000 7f92195bd640 10 monclient: build_initial_monmap 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.060+0000 7f92195bd640 10 monclient: monmap: 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:41.065094+0000 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:41.065094+0000 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:41.063 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:41.064 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.060+0000 7f92195bd640 1 Processor -- start 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.060+0000 7f92195bd640 1 -- start start 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.060+0000 7f92195bd640 10 monclient: init 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient: _add_conns ranks=[4,8,9,0,5,6,3,1,2,7,10] 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9214138a50 0x7f9214138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): picked mon.noname-c con 0x7f9214138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92140a5bb0 0x7f9214139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): picked mon.noname-e con 0x7f92140a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9214137960 0x7f9214137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): picked mon.noname-a con 0x7f9214137960 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f92140a1020 con 0x7f9214137960 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f92141398b0 con 0x7f9214138a50 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f92140a0780 con 0x7f92140a5bb0 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient(hunting): authenticate will time out at 1371.192222s 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9214137960 0x7f9214137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9214138a50 0x7f9214138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f9214137960 0x7f9214137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39956/0 (socket says 172.21.15.43:39956) 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 1 -- 172.21.15.43:0/1451058257 learned_addr learned my addr 172.21.15.43:0/1451058257 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): get_auth_request con 0x7f9214138a50 auth_method 0 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:41.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): get_auth_request con 0x7f9214137960 auth_method 0 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 1 --2- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92140a5bb0 0x7f9214139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): get_auth_request con 0x7f92140a5bb0 auth_method 0 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:41.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f920bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient(hunting): handle_auth_done global_id 64147 payload 274 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 -- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92140a5bb0 msgr2=0x7f9214139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 --2- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92140a5bb0 0x7f9214139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 -- 172.21.15.43:0/1451058257 >> v2:172.21.15.43:3300/0 conn(0x7f9214137960 msgr2=0x7f9214137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 --2- 172.21.15.43:0/1451058257 >> v2:172.21.15.43:3300/0 conn(0x7f9214137960 0x7f9214137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 monclient: found mon.noname-c 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:51:41.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 -- 172.21.15.43:0/1451058257 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92140a0d40 con 0x7f9214138a50 2024-04-19T00:51:41.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f9212ffd640 1 --2- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9214138a50 0x7f9214138e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92040097c0 tx=0x7f920401c330 comp rx=0 tx=0).ready entity=mon.4 client_cookie=db0a136cf8ddd5ad server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:41.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 1 -- 172.21.15.43:0/1451058257 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92040b0070 con 0x7f9214138a50 2024-04-19T00:51:41.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:41.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:51:41.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: dump: 2024-04-19T00:51:41.075 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:41.076 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 1 -- 172.21.15.43:0/1451058257 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92040aa810 con 0x7f9214138a50 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 1 -- 172.21.15.43:0/1451058257 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92040a9300 con 0x7f9214138a50 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 5 monclient: authenticate success, global_id 64147 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92195bd640 4 set_mon_vals no callback set 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.064+0000 7f92127fc640 10 monclient: dump: 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:41.077 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:41.078 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: shutdown 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9214138a50 msgr2=0x7f9214138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9214138a50 0x7f9214138e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f92040097c0 tx=0x7f920401c330 comp rx=0 tx=0).stop 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/1451058257 shutdown_connections 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- 172.21.15.43:0/1451058257 >> v2:172.21.15.43:3300/0 conn(0x7f9214137960 0x7f9214137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92140a5bb0 0x7f9214139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- 172.21.15.43:0/1451058257 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9214138a50 0x7f9214138e30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/1451058257 >> 172.21.15.43:0/1451058257 conn(0x7f92140a7cb0 msgr2=0x7f92140a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/1451058257 shutdown_connections 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/1451058257 wait complete. 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: build_initial_monmap 2024-04-19T00:51:41.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: monmap: 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:41.080 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 librados: starting msgr at 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 librados: starting objecter 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 Processor -- start 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- start start 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 librados: setting wanted keys 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 librados: calling monclient init 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: init 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _add_conns ranks=[5,7,2,4,3,8,0,1,6] 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92140a5bb0 0x7f92141d0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): picked mon.noname-h con 0x7f92140a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9214137960 0x7f9214136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): picked mon.noname-i con 0x7f9214137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 0x7f921413c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): picked mon.noname-b con 0x7f9214138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f92140a2100 con 0x7f9214138a50 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f92140a1f80 con 0x7f92140a5bb0 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9214142810 con 0x7f9214137960 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient(hunting): authenticate will time out at 1371.197726s 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 0x7f921413c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 0x7f921413c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:53768/0 (socket says 172.21.15.43:53768) 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 -- 172.21.15.43:0/3157162819 learned_addr learned my addr 172.21.15.43:0/3157162819 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): get_auth_request con 0x7f9214138a50 auth_method 0 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f9212ffd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92140a5bb0 0x7f92141d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f9212ffd640 10 monclient(hunting): get_auth_request con 0x7f92140a5bb0 auth_method 0 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f9212ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f9212ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f9212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920bfff640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9214137960 0x7f9214136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient(hunting): handle_auth_done global_id 23330 payload 824 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 -- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9214137960 msgr2=0x7f9214136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9214137960 0x7f9214136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 -- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92140a5bb0 msgr2=0x7f92141d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92140a5bb0 0x7f92141d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 monclient: found mon.noname-b 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 -- 172.21.15.43:0/3157162819 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92141222f0 con 0x7f9214138a50 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920bfff640 10 monclient: get_auth_request con 0x7f9214137960 auth_method 0 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920bfff640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9214137960 0x7f9214136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92137fe640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 0x7f921413c6c0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f920c00efc0 tx=0x7f920c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 client.?.objecter ms_handle_connect 0x7f9214138a50 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:41.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f920c0039a0 con 0x7f9214138a50 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: dump: 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:41.085 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 5 monclient: authenticate success, global_id 23330 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f920c003b80 con 0x7f9214138a50 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f920c003d30 con 0x7f9214138a50 2024-04-19T00:51:41.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _renew_subs 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921409ee60 con 0x7f9214138a50 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f9211ffb640 4 set_mon_vals no callback set 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 monclient: dump: 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:41.087 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 client.23330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _renew_subs 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f921413e390 con 0x7f9214138a50 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 librados: init done 2024-04-19T00:51:41.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f920c012050 con 0x7f9214138a50 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921413ec70 con 0x7f9214138a50 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91f4038e10 0x7f91f403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(318..318 src has 1..318) ==== 6141+0+0 (secure 0 0 0) 0x7f920c0498d0 con 0x7f9214138a50 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 10 client.23330.objecter ms_dispatch 0x7f9214000ba0 osd_map(318..318 src has 1..318) 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 3 client.23330.objecter handle_osd_map got epochs [318,318] > 0 2024-04-19T00:51:41.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.068+0000 7f920b7fe640 3 client.23330.objecter handle_osd_map decoding full epoch 318 2024-04-19T00:51:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.072+0000 7f920b7fe640 20 client.23330.objecter dump_active .. 0 homeless 2024-04-19T00:51:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.072+0000 7f9212ffd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91f4038e10 0x7f91f403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.072+0000 7f9212ffd640 10 monclient: get_auth_request con 0x7f91f4038e10 auth_method 0 2024-04-19T00:51:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.072+0000 7f920b7fe640 10 client.23330.objecter ms_handle_connect 0x7f91f4038e10 2024-04-19T00:51:41.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.072+0000 7f9212ffd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91f4038e10 0x7f91f403b2d0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f92040ad000 tx=0x7f92040a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.076+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f920c022020 con 0x7f9214138a50 2024-04-19T00:51:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.076+0000 7f920b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:41.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.076+0000 7f920b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:41.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.300+0000 7f92195bd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T00:51:41.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.300+0000 7f92195bd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T00:51:41.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.300+0000 7f92195bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:41.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.300+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9214076190 con 0x7f9214138a50 2024-04-19T00:51:41.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.304+0000 7f920b7fe640 1 -- 172.21.15.43:0/3157162819 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f920c010040 con 0x7f9214138a50 2024-04-19T00:51:41.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.304+0000 7f920b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T00:51:41.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.304+0000 7f920b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:41.306 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.306 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=12,logm=63} crc {auth=3236372504,config=2586702516,health=792476472,kv=4196696685,logm=2205632158}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=12,logm=61} crc {auth=3236372504,config=2586702516,health=2941319762,kv=4196696685,logm=1621770308}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=1429551716,kv=8226665,logm=3423075860}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=872527168,kv=4196696685,logm=635744901}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.307 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=12,config=2,health=12,kv=11,logm=63} crc {auth=217515142,config=2586702516,health=2941319762,kv=801871553,logm=3249772935}) 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=13,config=2,health=10,kv=11,logm=64} crc {auth=3236372504,config=2586702516,health=1826223829,kv=2173196854,logm=706228084}) 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=3222042181,config=2586702516,health=2377988878,kv=2349598857,logm=3739954148}) 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3236372504,config=2586702516,health=3876625136,kv=4196696685,logm=1688993725}) 2024-04-19T00:51:41.308 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.304+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=217515141,config=2586702516,health=3167573400,kv=1928767899,logm=375664089}) 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 10 librados: watch_flush enter 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 10 librados: watch_flush exit 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 20 client.23330.objecter shutdown clearing up homeless session... 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 10 client.23330.objecter successfully canceled tick 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91f4038e10 msgr2=0x7f91f403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91f4038e10 0x7f91f403b2d0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f92040ad000 tx=0x7f92040a6000 comp rx=0 tx=0).stop 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 10 monclient: shutdown 2024-04-19T00:51:41.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 msgr2=0x7f921413c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 0x7f921413c6c0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f920c00efc0 tx=0x7f920c00c490 comp rx=0 tx=0).stop 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 shutdown_connections 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91f4038e10 0x7f91f403b2d0 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9214138a50 0x7f921413c6c0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9214137960 0x7f9214136950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 --2- 172.21.15.43:0/3157162819 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92140a5bb0 0x7f92141d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 >> 172.21.15.43:0/3157162819 conn(0x7f92140a7cb0 msgr2=0x7f921406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 shutdown_connections 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 -- 172.21.15.43:0/3157162819 wait complete. 2024-04-19T00:51:41.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:41.308+0000 7f92195bd640 1 librados: shutdown 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358894108}) 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235325710}) 2024-04-19T00:51:41.314 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=125665637}) 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391285137}) 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2931132153}) 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.315 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1614117356}) 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3781781186}) 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788579379}) 2024-04-19T00:51:41.316 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.308+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1479333808}) 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3589874958}) 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548510117}) 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.318 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325365977}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2209018594}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2157463005}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3849134586}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1932982089}) 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.319 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732705786}) 2024-04-19T00:51:41.320 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.312+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3179627871}) 2024-04-19T00:51:41.324 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.324 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.324 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3929295955}) 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687545884}) 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3425118674}) 2024-04-19T00:51:41.325 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2607973342}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2223845571}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2453022043}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3291495933}) 2024-04-19T00:51:41.326 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898721120}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.316+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3618066793}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3171629251}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59736763}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1134676927}) 2024-04-19T00:51:41.327 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3486876936}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2461084707}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2852699038}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2571081311}) 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.328 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938562338}) 2024-04-19T00:51:41.329 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.320+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3331619421}) 2024-04-19T00:51:41.332 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.332 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.332 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607625973}) 2024-04-19T00:51:41.332 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512770067}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4087553646}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=36077909}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2198530734}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.333 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1244688199}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1419905047}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2662775678}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.324+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=379615390}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107621088}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=42456381}) 2024-04-19T00:51:41.334 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3164192180}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1973344203}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=561106457}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2864928862}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.335 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1576948042}) 2024-04-19T00:51:41.336 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.336 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396114449}) 2024-04-19T00:51:41.336 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.328+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=484042604}) 2024-04-19T00:51:41.337 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T00:51:41.342 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.342 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.342 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791617769}) 2024-04-19T00:51:41.342 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.342 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.342 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1751846044}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3658673783}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2872165250}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3277715095}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1768381834}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.343 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3719072764}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012492825}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.332+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=973165166}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174482728}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898851787}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1028850078}) 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.344 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2663899598}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2215593577}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2588331010}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2245792517}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144479915}) 2024-04-19T00:51:41.345 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.336+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3549326122}) 2024-04-19T00:51:41.346 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.346 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.346 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153969456}) 2024-04-19T00:51:41.346 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.346 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.346 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747090557}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=13819167}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1265139070}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2721812396}) 2024-04-19T00:51:41.347 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1272935602}) 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=522346653}) 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685612301}) 2024-04-19T00:51:41.348 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.340+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3535645197}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964123146}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3987731803}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3462675176}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.353 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1356755496}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3361324997}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1573946906}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=62300918}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862616278}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.344+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3378296911}) 2024-04-19T00:51:41.354 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462583587}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645168893}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1111064106}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2370287130}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.355 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1662436823}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3574349831}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2185438223}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=342277897}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1779954441}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.356 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450198675}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3674844411}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3516560629}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.348+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2231184452}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1741484176}) 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.357 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4167157095}) 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2411263440}) 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3227305459}) 2024-04-19T00:51:41.358 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1838929965}) 2024-04-19T00:51:41.371 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.371 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4197508278}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3070240232}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1204448035}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626036916}) 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.372 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2406207005}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1246420264}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1342560284}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623059667}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.352+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=833243640}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1376920988}) 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.373 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807642567}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1358437999}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3178019023}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2028447910}) 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.374 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4215548395}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1750501949}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4290842651}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.356+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1311702064}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365780752}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.375 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599421878}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3856002882}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3855956415}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1588846243}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=139298200}) 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.376 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2898494388}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2565074688}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.360+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2541599879}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146324078}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303654451}) 2024-04-19T00:51:41.377 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2602141854}) 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3753876510}) 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1572130567}) 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.378 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=75606030}) 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4023761118}) 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=226740292}) 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.364+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3264204158}) 2024-04-19T00:51:41.379 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.380 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.380 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=318915107}) 2024-04-19T00:51:41.380 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.380 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.380 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1089918730}) 2024-04-19T00:51:41.380 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.381 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.381 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1373496934}) 2024-04-19T00:51:41.381 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.381 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.381 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3441725199}) 2024-04-19T00:51:41.381 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3705181320}) 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=174172225}) 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.382 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.383 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1660569744}) 2024-04-19T00:51:41.383 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.383 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4227020293}) 2024-04-19T00:51:41.383 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.368+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1393740485}) 2024-04-19T00:51:41.406 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.406 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.406 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122413518}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570939838}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2728397368}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3825936403}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1455693849}) 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.407 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4281792552}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1805893362}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325416522}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.372+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=615561843}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.408 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330508501}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357288444}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2883238019}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=130013506}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.409 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3286781492}) 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3846900180}) 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2669028971}) 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3033357369}) 2024-04-19T00:51:41.410 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1009866138}) 2024-04-19T00:51:41.411 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.411 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.411 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872223004}) 2024-04-19T00:51:41.411 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.411 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.411 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057141206}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1622164833}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2548216918}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=437478826}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.412 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2609427609}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3557348861}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=343303432}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.376+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2265037570}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1251310392}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.413 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=438710100}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1249060433}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3975578568}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=615492179}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.414 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3851504609}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1819770008}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619300087}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.380+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1552466608}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462372041}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2487407547}) 2024-04-19T00:51:41.415 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2390813102}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3651087489}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2085255146}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=457510724}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3964767836}) 2024-04-19T00:51:41.416 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170400241}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.384+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=608532959}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458223122}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2771369458}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137579463}) 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.417 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3230180819}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3370720716}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4047366729}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3063357155}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308514178}) 2024-04-19T00:51:41.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3908812067}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755038144}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1520228750}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3841211281}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1021618001}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4224108477}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.388+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3257196557}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3039846255}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690661202}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2297359036}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2306175259}) 2024-04-19T00:51:41.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558259605}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2366780360}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2041900576}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3631765288}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2533287793}) 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1728442871}) 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081860256}) 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.392+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1881812067}) 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795424823}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211874485}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2571459218}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2063100742}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=220562133}) 2024-04-19T00:51:41.423 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1631339454}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2185194993}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1226866171}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.396+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1840069709}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159078705}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655245042}) 2024-04-19T00:51:41.424 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=700534040}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2898251461}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1497692683}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=684817023}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.425 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1077087300}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022568685}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.400+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1975223048}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232113533}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872643062}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.426 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2423508594}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2873111852}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3946214923}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1642545880}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2145216552}) 2024-04-19T00:51:41.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2064838033}) 2024-04-19T00:51:41.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3446188428}) 2024-04-19T00:51:41.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043056924}) 2024-04-19T00:51:41.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637213915}) 2024-04-19T00:51:41.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=33583967}) 2024-04-19T00:51:41.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4179752654}) 2024-04-19T00:51:41.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=124205145}) 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3393657115}) 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1960896940}) 2024-04-19T00:51:41.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=617378816}) 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.404+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1426654940}) 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249181429}) 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.504 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3093305766}) 2024-04-19T00:51:41.505 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.505 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.505 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1423464666}) 2024-04-19T00:51:41.505 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.505 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.505 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2152289882}) 2024-04-19T00:51:41.506 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.506 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.506 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=141103661}) 2024-04-19T00:51:41.506 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.506 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.506 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3303628908}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1831701323}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207731858}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.408+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1133388974}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274174643}) 2024-04-19T00:51:41.507 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938990253}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3759934747}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2481977816}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=580397213}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.508 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3467994393}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2147584487}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=678717980}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2794871852}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282457011}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818663988}) 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.509 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3810215341}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=487057643}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=103102912}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4048163698}) 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.510 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1440215620}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3268114275}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.412+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4058823079}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4032281904}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2145383218}) 2024-04-19T00:51:41.511 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2686602005}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3824304003}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2661419022}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1938619077}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.512 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=623414737}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274278494}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.416+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4072299059}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309932152}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766572580}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1240055}) 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.513 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1102007031}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2055146814}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2543296212}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.514 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4256777943}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1495807540}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.420+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2101077880}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544827496}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2872723309}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=126676507}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.515 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1625692348}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3565078052}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=957896798}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3874038441}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275126438}) 2024-04-19T00:51:41.516 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1104379026}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482594544}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686970125}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=894128024}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2545244086}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.517 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1640817949}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3635203520}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3969169109}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953657901}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.424+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4252067041}) 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.518 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=553418047}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=43089322}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2910690102}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2529601428}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2281795561}) 2024-04-19T00:51:41.519 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2497084946}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3357127843}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207132669}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.428+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2360289451}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=465310822}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.520 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119460615}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1983132842}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2248949804}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=829332608}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3934992959}) 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.521 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=541385525}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529973756}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=896569577}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056346320}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3421646614}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.522 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=129898625}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1876141163}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=995829238}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=80351795}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3037122551}) 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.523 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2753333948}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.432+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1468535925}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2796601629}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161843195}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1561895742}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=804397738}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4137120185}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1254904634}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2293875341}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274640829}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.436+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1683980667}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253675985}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3463083197}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2546239328}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4277464114}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=648458920}) 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2402750116}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=5195346}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610110765}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.440+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=244638743}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977595100}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2773087136}) 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.527 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1282806483}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2734298690}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3272055486}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=59904331}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2691565707}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375015993}) 2024-04-19T00:51:41.528 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3774906494}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3907847320}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310844072}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3496755756}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3347278500}) 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.529 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2911226379}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1412465230}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=281945055}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=269100462}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.444+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=398594373}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795471373}) 2024-04-19T00:51:41.530 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1279997866}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1128961294}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2617717609}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1725097337}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=758536106}) 2024-04-19T00:51:41.531 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=396966178}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900081388}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.448+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2548239702}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138769457}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=20966250}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4179702570}) 2024-04-19T00:51:41.532 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3752067786}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=438893063}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=117833270}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1595234397}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836700}) 2024-04-19T00:51:41.533 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.452+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3940953854}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734293076}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64638821}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1330879102}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1349472131}) 2024-04-19T00:51:41.534 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2554451754}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2204373711}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1614235861}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658011208}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4270114834}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.535 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=535513534}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2786289064}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.456+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=194741171}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1481800689}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1871612564}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.536 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=776928472}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=405475635}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346248831}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=742829684}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2113307159}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4196336065}) 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.537 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=56471520}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2209555571}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=936726175}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3296464581}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.538 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4231616292}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190428769}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.460+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2284530001}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481705723}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597485870}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3248847969}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.539 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1161168508}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3380319996}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3016248272}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2981410210}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1170599939}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.464+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=609535245}) 2024-04-19T00:51:41.540 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=13} crc {logm=1147236757,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1702987011,mon_config_key=3982140354}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=2503628825,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2040647713}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=71,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=8} crc {logm=3609273835,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1702987011,mon_config_key=2845857477}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=71,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=10,mon_config_key=9} crc {logm=1478623714,mdsmap=1401145748,mgr=1871624564,mgr_command_descs=3636332126,mgrstat=1426396518,mon_config_key=635327454}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.541 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=78589953,mdsmap=2840952586,mgr=1318271769,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7} crc {logm=1685466571,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2746371886}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=66,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11} crc {logm=3459264204,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=433660206}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=2914197888,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.468+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=1284629214,mdsmap=2840952586,mgr=517009004,mgr_metadata=2750671369,mgrstat=1426396518,mon_config_key=2845857477}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.542 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=57} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=3112274954,osdmap=2495713731}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=59} crc {mon_config_key=1328169386,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=3235173061,osdmap=924176371}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=8,monmap=2,osd_metadata=6,osd_snap=22,osdmap=62} crc {mon_config_key=1328169386,monmap=3140514009,osd_metadata=3468572908,osd_snap=3022661404,osdmap=1289651231}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=60} crc {mon_config_key=2654898169,monmap=4080741109,osd_metadata=2350310201,osd_pg_creating=4056323874,osd_snap=374558040,osdmap=2287486810}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=20,osdmap=62} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3305883572,osd_pg_creating=4056323874,osd_snap=2106510206,osdmap=1666499964}) 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.543 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=57} crc {mon_config_key=2654898169,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=3112274954,osdmap=2181534200}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=23,osdmap=59} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3305883572,osd_pg_creating=4056323874,osd_snap=1532685383,osdmap=2600956860}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=58} crc {mon_config_key=2513593582,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=26235821,osdmap=2791803402}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.472+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=8,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=58} crc {mon_config_key=1035730443,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=4056323874,osd_snap=3112274954,osdmap=3189106954}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1672854064}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3394650554}) 2024-04-19T00:51:41.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2754907467}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=519504348}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2881494723}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3739574134}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3106607611}) 2024-04-19T00:51:41.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135887304}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4132812715}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2612992481}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.476+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1951379832}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=540746232}) 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3409594188}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=230045663}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=263486349}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=203722633}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1845086219}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.480+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=960965151}) 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3782241950}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1908299598}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3335428072}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1338941658}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=471102923}) 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3487956553}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4096507319}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=878841298}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.484+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4015749339}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=218630511}) 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=373922110}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2124983693}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=197375008}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1441463189}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=31357824}) 2024-04-19T00:51:41.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1019902786}) 2024-04-19T00:51:41.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2436492067}) 2024-04-19T00:51:41.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.488+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2078934711}) 2024-04-19T00:51:41.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3545624527}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=499886866}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=21} crc {osdmap=2540016138}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=21} crc {osdmap=270301354}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=21} crc {osdmap=3452213191}) 2024-04-19T00:51:41.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=21} crc {osdmap=2736267574}) 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=22} crc {osdmap=1195614215}) 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2847052774}) 2024-04-19T00:51:41.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:51:41.508+0000 7ff91ad57640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=22} crc {osdmap=3303457292}) 2024-04-19T00:51:42.339 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:51:42.339 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:51:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient: get_monmap_and_config 2024-04-19T00:51:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient: build_initial_monmap 2024-04-19T00:51:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient: monmap: 2024-04-19T00:51:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.473 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:42.475557+0000 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:42.475557+0000 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:42.474 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:42.475 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 Processor -- start 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 -- start start 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient: init 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient: _add_conns ranks=[9,7,5,0,6,10,4,8,1,2,3] 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe32c138cd0 0x7fe32c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): picked mon.noname-a con 0x7fe32c138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe32c0a5b80 0x7fe32c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): picked mon.noname-i con 0x7fe32c0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c1358b0 0x7fe32c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): picked mon.noname-h con 0x7fe32c1358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fe32c0a4410 con 0x7fe32c138cd0 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe32c139b30 con 0x7fe32c1358b0 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe32c0a3dc0 con 0x7fe32c0a5b80 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe331f2a640 10 monclient(hunting): authenticate will time out at 1372.602763s 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe32c138cd0 0x7fe32c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe32c138cd0 0x7fe32c1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39972/0 (socket says 172.21.15.43:39972) 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 -- 172.21.15.43:0/1909570037 learned_addr learned my addr 172.21.15.43:0/1909570037 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:42.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): get_auth_request con 0x7fe32c138cd0 auth_method 0 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32bfff640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c1358b0 0x7fe32c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32affd640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe32c0a5b80 0x7fe32c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32bfff640 10 monclient(hunting): get_auth_request con 0x7fe32c1358b0 auth_method 0 2024-04-19T00:51:42.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32affd640 10 monclient(hunting): get_auth_request con 0x7fe32c0a5b80 auth_method 0 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient(hunting): handle_auth_done global_id 64233 payload 274 2024-04-19T00:51:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 -- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe32c0a5b80 msgr2=0x7fe32c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe32c0a5b80 0x7fe32c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 -- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c1358b0 msgr2=0x7fe32c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c1358b0 0x7fe32c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:42.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 monclient: found mon.noname-a 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.472+0000 7fe32b7fe640 1 -- 172.21.15.43:0/1909570037 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c0a40a0 con 0x7fe32c138cd0 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32b7fe640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe32c138cd0 0x7fe32c1390b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe314009870 tx=0x7fe31401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847db4ba1fd3bf36 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 1 -- 172.21.15.43:0/1909570037 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3140b0070 con 0x7fe32c138cd0 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: dump: 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.491 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.494 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 1 -- 172.21.15.43:0/1909570037 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3140aa810 con 0x7fe32c138cd0 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 1 -- 172.21.15.43:0/1909570037 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe3140a9300 con 0x7fe32c138cd0 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 5 monclient: authenticate success, global_id 64233 2024-04-19T00:51:42.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 4 set_mon_vals no callback set 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe32a7fc640 10 monclient: dump: 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 10 monclient: shutdown 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 -- 172.21.15.43:0/1909570037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe32c138cd0 msgr2=0x7fe32c1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe32c138cd0 0x7fe32c1390b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe314009870 tx=0x7fe31401c3c0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 -- 172.21.15.43:0/1909570037 shutdown_connections 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c1358b0 0x7fe32c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe32c0a5b80 0x7fe32c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1909570037 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe32c138cd0 0x7fe32c1390b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 -- 172.21.15.43:0/1909570037 >> 172.21.15.43:0/1909570037 conn(0x7fe32c0ab2f0 msgr2=0x7fe32c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 -- 172.21.15.43:0/1909570037 shutdown_connections 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 -- 172.21.15.43:0/1909570037 wait complete. 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 10 monclient: build_initial_monmap 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 10 monclient: monmap: 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:42.498 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 librados: starting msgr at 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 librados: starting objecter 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 Processor -- start 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 -- start start 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 librados: setting wanted keys 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 1 librados: calling monclient init 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.476+0000 7fe331f2a640 10 monclient: init 2024-04-19T00:51:42.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _add_conns ranks=[3,8,5,6,7,1,2,4,0] 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe32c0a5b80 0x7fe32c1d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): picked mon.noname-g con 0x7fe32c0a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 0x7fe32c0a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): picked mon.noname-e con 0x7fe32c1358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c138cd0 0x7fe32c0a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): picked mon.noname-h con 0x7fe32c138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe32c0a2010 con 0x7fe32c0a5b80 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe32c0a1e90 con 0x7fe32c138cd0 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe32c142aa0 con 0x7fe32c1358b0 2024-04-19T00:51:42.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient(hunting): authenticate will time out at 1372.608494s 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 0x7fe32c0a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 0x7fe32c0a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52922/0 (socket says 172.21.15.43:52922) 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 -- 172.21.15.43:0/1037002701 learned_addr learned my addr 172.21.15.43:0/1037002701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): get_auth_request con 0x7fe32c1358b0 auth_method 0 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32bfff640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c138cd0 0x7fe32c0a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe32c0a5b80 0x7fe32c1d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 10 monclient(hunting): get_auth_request con 0x7fe32c0a5b80 auth_method 0 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient(hunting): handle_auth_done global_id 23093 payload 824 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 -- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c138cd0 msgr2=0x7fe32c0a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c138cd0 0x7fe32c0a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 -- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe32c0a5b80 msgr2=0x7fe32c1d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe32c0a5b80 0x7fe32c1d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient: _finish_hunting 0 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 monclient: found mon.noname-e 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 -- 172.21.15.43:0/1037002701 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32c0a3590 con 0x7fe32c1358b0 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32bfff640 10 monclient: get_auth_request con 0x7fe32c138cd0 auth_method 0 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32bfff640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c138cd0 0x7fe32c0a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32affd640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 0x7fe32c0a0e30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00d800 tx=0x7fe31c00dcc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 client.?.objecter ms_handle_connect 0x7fe32c1358b0 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe31c00bb30 con 0x7fe32c1358b0 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: dump: 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.503 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: _finish_auth 0 2024-04-19T00:51:42.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe31c00bcf0 con 0x7fe32c1358b0 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe31c0148a0 con 0x7fe32c1358b0 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 5 monclient: authenticate success, global_id 23093 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _renew_subs 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe32c09c370 con 0x7fe32c1358b0 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe3297fa640 4 set_mon_vals no callback set 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 monclient: dump: 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.505 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.506 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 client.23093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _renew_subs 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe32c09d3d0 con 0x7fe32c1358b0 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 librados: init done 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe32c0693b0 con 0x7fe32c1358b0 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe31c01b030 con 0x7fe32c1358b0 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe30c038e10 0x7fe30c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(319..319 src has 1..319) ==== 6154+0+0 (secure 0 0 0) 0x7fe31c019020 con 0x7fe32c1358b0 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 10 client.23093.objecter ms_dispatch 0x7fe32c000ba0 osd_map(319..319 src has 1..319) 2024-04-19T00:51:42.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 3 client.23093.objecter handle_osd_map got epochs [319,319] > 0 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 3 client.23093.objecter handle_osd_map decoding full epoch 319 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe323fff640 20 client.23093.objecter dump_active .. 0 homeless 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe30c038e10 0x7fe30c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.480+0000 7fe32b7fe640 10 monclient: get_auth_request con 0x7fe30c038e10 auth_method 0 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.484+0000 7fe323fff640 10 client.23093.objecter ms_handle_connect 0x7fe30c038e10 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.484+0000 7fe32b7fe640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe30c038e10 0x7fe30c03b2d0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7fe3140ad000 tx=0x7fe3140a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.484+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe31c022060 con 0x7fe32c1358b0 2024-04-19T00:51:42.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.484+0000 7fe323fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:42.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.484+0000 7fe323fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.716+0000 7fe331f2a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.716+0000 7fe331f2a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.716+0000 7fe331f2a640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.716+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe32c1355c0 con 0x7fe32c1358b0 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe323fff640 1 -- 172.21.15.43:0/1037002701 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fe32c0a3590 con 0x7fe32c1358b0 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe323fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:51:42.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe323fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:42.721 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":50,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:51:42.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 10 librados: watch_flush enter 2024-04-19T00:51:42.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 10 librados: watch_flush exit 2024-04-19T00:51:42.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 20 client.23093.objecter shutdown clearing up homeless session... 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 10 client.23093.objecter successfully canceled tick 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe30c038e10 msgr2=0x7fe30c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe30c038e10 0x7fe30c03b2d0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7fe3140ad000 tx=0x7fe3140a6000 comp rx=0 tx=0).stop 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 10 monclient: shutdown 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 msgr2=0x7fe32c0a0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 0x7fe32c0a0e30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00d800 tx=0x7fe31c00dcc0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 shutdown_connections 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe32c138cd0 0x7fe32c0a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe30c038e10 0x7fe30c03b2d0 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe32c1358b0 0x7fe32c0a0e30 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 --2- 172.21.15.43:0/1037002701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe32c0a5b80 0x7fe32c1d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 >> 172.21.15.43:0/1037002701 conn(0x7fe32c0ab2f0 msgr2=0x7fe32c0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:42.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 shutdown_connections 2024-04-19T00:51:42.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 -- 172.21.15.43:0/1037002701 wait complete. 2024-04-19T00:51:42.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.720+0000 7fe331f2a640 1 librados: shutdown 2024-04-19T00:51:42.740 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:51:42.740 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:51:42.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:51:42.740 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:51:42.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient: get_monmap_and_config 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient: build_initial_monmap 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient: monmap: 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:42.870979+0000 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:42.870979+0000 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:42.869 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:42.870 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:42.870 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:42.870 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:42.870 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:42.870 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:42.870 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 1 Processor -- start 2024-04-19T00:51:42.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 1 -- start start 2024-04-19T00:51:42.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient: init 2024-04-19T00:51:42.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient: _add_conns ranks=[0,4,1,7,10,5,3,8,2,6,9] 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6f00a3050 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f00a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f00a3430 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa6f013beb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f0076730 con 0x7fa6f00a3050 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f009b0e0 con 0x7fa6f013beb0 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa6f0076480 con 0x7fa6f00a3430 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f7aa4640 10 monclient(hunting): authenticate will time out at 1372.998322s 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f5840640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fa6f00a3050 0x7fa6f0136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51332/0 (socket says 172.21.15.43:51332) 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f5840640 1 -- 172.21.15.43:0/949911216 learned_addr learned my addr 172.21.15.43:0/949911216 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:51:42.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f503f640 1 --2- 172.21.15.43:0/949911216 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f00a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f503f640 10 monclient(hunting): get_auth_request con 0x7fa6f00a3430 auth_method 0 2024-04-19T00:51:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f503f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f503f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.868+0000 7fa6f503f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 10 monclient(hunting): handle_auth_done global_id 64156 payload 274 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 1 -- 172.21.15.43:0/949911216 >> v1:172.21.15.161:6789/0 conn(0x7fa6f013beb0 legacy=0x7fa6f013e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 1 -- 172.21.15.43:0/949911216 >> v1:172.21.15.43:6789/0 conn(0x7fa6f00a3050 legacy=0x7fa6f0136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 10 monclient: _finish_hunting 0 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 1 monclient: found mon.noname-c 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:51:42.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 1 -- 172.21.15.43:0/949911216 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f009dd20 con 0x7fa6f00a3430 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f503f640 1 --2- 172.21.15.43:0/949911216 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f00a3810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00097f0 tx=0x7fa6e001c370 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2725cfe5aa51344b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 1 -- 172.21.15.43:0/949911216 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6e00b1070 con 0x7fa6f00a3430 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: dump: 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.876 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.877 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: _finish_auth 0 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: _check_auth_tickets 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 1 -- 172.21.15.43:0/949911216 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e00ab800 con 0x7fa6f00a3430 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 5 monclient: authenticate success, global_id 64156 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:42.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 4 set_mon_vals no callback set 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 1 -- 172.21.15.43:0/949911216 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6e00aa2f0 con 0x7fa6f00a3430 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f483e640 10 monclient: dump: 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.879 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 10 monclient: shutdown 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/949911216 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 msgr2=0x7fa6f00a3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.872+0000 7fa6f7aa4640 1 --2- 172.21.15.43:0/949911216 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f00a3810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00097f0 tx=0x7fa6e001c370 comp rx=0 tx=0).stop 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/949911216 shutdown_connections 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 --2- 172.21.15.43:0/949911216 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f00a3810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/949911216 >> 172.21.15.43:0/949911216 conn(0x7fa6f00a4140 msgr2=0x7fa6f00a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:42.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/949911216 shutdown_connections 2024-04-19T00:51:42.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/949911216 wait complete. 2024-04-19T00:51:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: build_initial_monmap 2024-04-19T00:51:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: monmap: 2024-04-19T00:51:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:42.888 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:42.889 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 librados: starting msgr at 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 librados: starting objecter 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 Processor -- start 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- start start 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 librados: setting wanted keys 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 librados: calling monclient init 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: init 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: _add_conns ranks=[8,4,1,6,5,2,7,3,0] 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6f00a3050 0x7fa6f01ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): picked mon.noname-e con 0x7fa6f00a3050 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:42.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f0135900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f00a3430 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6f013beb0 0x7fa6f0135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): picked mon.noname-f con 0x7fa6f013beb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f0142770 con 0x7fa6f013beb0 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa6f01425f0 con 0x7fa6f00a3430 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa6f01428f0 con 0x7fa6f00a3050 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): authenticate will time out at 1373.005761s 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f0135900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6f00a3050 0x7fa6f01ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f0135900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34198/0 (socket says 172.21.15.43:34198) 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 1 -- 172.21.15.43:0/3483020699 learned_addr learned my addr 172.21.15.43:0/3483020699 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): get_auth_request con 0x7fa6f00a3430 auth_method 0 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6f00a3050 0x7fa6f01ceae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52932/0 (socket says 172.21.15.43:52932) 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): get_auth_request con 0x7fa6f00a3050 auth_method 0 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6f013beb0 0x7fa6f0135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): get_auth_request con 0x7fa6f013beb0 auth_method 0 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f6041640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient(hunting): handle_auth_done global_id 23102 payload 824 2024-04-19T00:51:42.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 msgr2=0x7fa6f0135900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f0135900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6f013beb0 msgr2=0x7fa6f0135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6f013beb0 0x7fa6f0135e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient: _finish_hunting 0 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 monclient: found mon.noname-e 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f012cae0 con 0x7fa6f00a3050 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa6f00a3430 0x7fa6f0135900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f5840640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6f00a3050 0x7fa6f01ceae0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c420 tx=0x7fa6ec00c8e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 client.?.objecter ms_handle_connect 0x7fa6f00a3050 2024-04-19T00:51:42.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6ec017070 con 0x7fa6f00a3050 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: dump: 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.895 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0070d0 con 0x7fa6f00a3050 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 5 monclient: authenticate success, global_id 23102 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: _renew_subs 2024-04-19T00:51:42.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f012cdc0 con 0x7fa6f00a3050 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6ec007730 con 0x7fa6f00a3050 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e7fff640 4 set_mon_vals no callback set 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 client.23102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 monclient: dump: 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.897 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: _renew_subs 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f012c700 con 0x7fa6f00a3050 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 librados: init done 2024-04-19T00:51:42.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 0x7fa6f00714a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f006d140 con 0x7fa6f00710c0 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa6ec0072c0 con 0x7fa6f00a3050 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 0x7fa6f00714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient: get_auth_request con 0x7fa6f00710c0 auth_method 0 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6d4038e10 0x7fa6d403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(320..320 src has 1..320) ==== 5769+0+0 (secure 0 0 0) 0x7fa6ec0484f0 con 0x7fa6f00a3050 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 10 client.23102.objecter ms_dispatch 0x7fa6f0000ba0 osd_map(320..320 src has 1..320) 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 3 client.23102.objecter handle_osd_map got epochs [320,320] > 0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.876+0000 7fa6e67fc640 3 client.23102.objecter handle_osd_map decoding full epoch 320 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6f503f640 10 monclient(hunting): handle_auth_done global_id 64242 payload 824 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6f503f640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6f00008e0 con 0x7fa6f00710c0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 20 client.23102.objecter dump_active .. 0 homeless 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 client.23102.objecter ms_handle_connect 0x7fa6f00710c0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 client.23102.objecter resend_mon_ops 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6f503f640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 0x7fa6f00714a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009710 tx=0x7fa6e0019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6f6041640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6d4038e10 0x7fa6d403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6f6041640 10 monclient: get_auth_request con 0x7fa6d4038e10 auth_method 0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6e00b1040 con 0x7fa6f00710c0 2024-04-19T00:51:42.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 monclient: dump: 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.901 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6e00ab880 con 0x7fa6f00710c0 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 msgr2=0x7fa6f00714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 0x7fa6f00714a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009710 tx=0x7fa6e0019110 comp rx=0 tx=0).stop 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 msgr2=0x7fa6f00714a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00710c0 0x7fa6f00714a0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009710 tx=0x7fa6e0019110 comp rx=0 tx=0).stop 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6f6041640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6d4038e10 0x7fa6d403b2d0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fa6f012d850 tx=0x7fa6e8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.880+0000 7fa6e67fc640 10 client.23102.objecter ms_handle_connect 0x7fa6d4038e10 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f7aa4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f7aa4640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 0x7fa6f0138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f7aa4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f00a1b20 con 0x7fa6f00719e0 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 0x7fa6f0138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient: get_auth_request con 0x7fa6f00719e0 auth_method 0 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 10 monclient(hunting): handle_auth_done global_id 64251 payload 824 2024-04-19T00:51:42.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 1 -- 172.21.15.43:0/3483020699 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa6f0071df0 con 0x7fa6f00719e0 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 10 client.23102.objecter ms_handle_connect 0x7fa6f00719e0 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 10 client.23102.objecter resend_mon_ops 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6f503f640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 0x7fa6f0138910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00ad000 tx=0x7fa6e00a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa6e00b1040 con 0x7fa6f00719e0 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 10 monclient: dump: 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:42.904 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.884+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fa6e00b6450 con 0x7fa6f00719e0 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6e67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 msgr2=0x7fa6f0138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6e67fc640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 0x7fa6f0138910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00ad000 tx=0x7fa6e00a9000 comp rx=0 tx=0).stop 2024-04-19T00:51:42.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6e67fc640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 msgr2=0x7fa6f0138910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6e67fc640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa6f00719e0 0x7fa6f0138910 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa6e00ad000 tx=0x7fa6e00a9000 comp rx=0 tx=0).stop 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 29155, 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:42.906 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:42.907 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:42.908 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.909 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:42.910 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:42.911 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:42.912 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:42.914 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:42.916 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:42.919 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.920 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.921 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.922 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.923 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:42.924 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:42.925 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.926 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:42.927 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:42.928 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:42.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 10 librados: watch_flush enter 2024-04-19T00:51:42.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 10 librados: watch_flush exit 2024-04-19T00:51:42.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 20 client.23102.objecter shutdown clearing up homeless session... 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 10 client.23102.objecter successfully canceled tick 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6d4038e10 msgr2=0x7fa6d403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa6d4038e10 0x7fa6d403b2d0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7fa6f012d850 tx=0x7fa6e8009040 comp rx=0 tx=0).stop 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 10 monclient: shutdown 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6f00a3050 msgr2=0x7fa6f01ceae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 --2- 172.21.15.43:0/3483020699 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa6f00a3050 0x7fa6f01ceae0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c420 tx=0x7fa6ec00c8e0 comp rx=0 tx=0).stop 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f6041640 1 -- 172.21.15.43:0/3483020699 reap_dead start 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f6041640 1 -- 172.21.15.43:0/3483020699 reap_dead start 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 shutdown_connections 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 >> 172.21.15.43:0/3483020699 conn(0x7fa6f00a4140 msgr2=0x7fa6f00a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:42.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 shutdown_connections 2024-04-19T00:51:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 -- 172.21.15.43:0/3483020699 wait complete. 2024-04-19T00:51:42.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:42.888+0000 7fa6f7aa4640 1 librados: shutdown 2024-04-19T00:51:42.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:42.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:51:42.932 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient: get_monmap_and_config 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient: monmap: 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:43.062523+0000 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:43.062523+0000 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.061 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:43.062 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:43.062 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 Processor -- start 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 -- start start 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient: init 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient: _add_conns ranks=[0,2,8,7,9,1,5,3,10,6,4] 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda20138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda200a5b80 0x7fda2013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): picked mon.noname-b con 0x7fda200a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 0x7fda201407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): picked mon.noname-e con 0x7fda201397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda200a3dc0 con 0x7fda20138cd0 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fda200a4410 con 0x7fda200a5b80 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fda200a40a0 con 0x7fda201397c0 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda258f9640 10 monclient(hunting): authenticate will time out at 1373.189598s 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda200a5b80 0x7fda2013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 0x7fda201407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 0x7fda201407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52938/0 (socket says 172.21.15.43:52938) 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda200a5b80 0x7fda2013e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:53780/0 (socket says 172.21.15.43:53780) 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 -- 172.21.15.43:0/995814857 learned_addr learned my addr 172.21.15.43:0/995814857 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1effd640 1 --1- 172.21.15.43:0/995814857 >> v1:172.21.15.43:6789/0 conn(0x7fda20138cd0 0x7fda201390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51342/0 (socket says 172.21.15.43:51342) 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request con 0x7fda201397c0 auth_method 0 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): get_auth_request con 0x7fda200a5b80 auth_method 0 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_done global_id 23111 payload 274 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 -- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda200a5b80 msgr2=0x7fda2013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 --2- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda200a5b80 0x7fda2013e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 -- 172.21.15.43:0/995814857 >> v1:172.21.15.43:6789/0 conn(0x7fda20138cd0 legacy=0x7fda201390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 monclient: found mon.noname-e 2024-04-19T00:51:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:43.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.060+0000 7fda1f7fe640 1 -- 172.21.15.43:0/995814857 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda200a2ae0 con 0x7fda201397c0 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 --2- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 0x7fda201407c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e80 tx=0x7fda1801c910 comp rx=0 tx=0).ready entity=mon.8 client_cookie=4bc6e41cae39da10 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 1 -- 172.21.15.43:0/995814857 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda180090d0 con 0x7fda201397c0 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: dump: 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.074 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 1 -- 172.21.15.43:0/995814857 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda180a5b10 con 0x7fda201397c0 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 1 -- 172.21.15.43:0/995814857 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda180a5d00 con 0x7fda201397c0 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 5 monclient: authenticate success, global_id 23111 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 4 set_mon_vals no callback set 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1dffb640 10 monclient: dump: 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.076 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.077 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: shutdown 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 msgr2=0x7fda201407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 --2- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 0x7fda201407c0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fda18009e80 tx=0x7fda1801c910 comp rx=0 tx=0).stop 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- 172.21.15.43:0/995814857 shutdown_connections 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 --2- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda201397c0 0x7fda201407c0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 --2- 172.21.15.43:0/995814857 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda200a5b80 0x7fda2013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- 172.21.15.43:0/995814857 >> 172.21.15.43:0/995814857 conn(0x7fda200ab2f0 msgr2=0x7fda200ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- 172.21.15.43:0/995814857 shutdown_connections 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- 172.21.15.43:0/995814857 wait complete. 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: monmap: 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.079 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 librados: starting msgr at 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 librados: starting objecter 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 Processor -- start 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- start start 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 librados: setting wanted keys 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 librados: calling monclient init 2024-04-19T00:51:43.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: init 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient: _add_conns ranks=[1,8,4,7,3,5,2,0,6] 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda200a5b80 0x7fda201d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): picked mon.noname-f con 0x7fda200a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda20138cd0 0x7fda201369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): picked mon.noname-e con 0x7fda20138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fda201397c0 0x7fda2009d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): picked mon.noname-c con 0x7fda201397c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda20143730 con 0x7fda200a5b80 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fda200a20a0 con 0x7fda201397c0 2024-04-19T00:51:43.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fda201438b0 con 0x7fda20138cd0 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda258f9640 10 monclient(hunting): authenticate will time out at 1373.195256s 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda20138cd0 0x7fda201369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fda201397c0 0x7fda2009d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda20138cd0 0x7fda201369c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52940/0 (socket says 172.21.15.43:52940) 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 1 -- 172.21.15.43:0/2821863775 learned_addr learned my addr 172.21.15.43:0/2821863775 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request con 0x7fda201397c0 auth_method 0 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): get_auth_request con 0x7fda20138cd0 auth_method 0 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1effd640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda200a5b80 0x7fda201d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1effd640 10 monclient(hunting): get_auth_request con 0x7fda200a5b80 auth_method 0 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient(hunting): handle_auth_done global_id 64174 payload 824 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda20138cd0 msgr2=0x7fda201369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda20138cd0 0x7fda201369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda200a5b80 msgr2=0x7fda201d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda200a5b80 0x7fda201d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 monclient: found mon.noname-c 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1f7fe640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda200a3040 con 0x7fda201397c0 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1effd640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda200a5b80 0x7fda201d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.064+0000 7fda1e7fc640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fda20138cd0 0x7fda201369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1f7fe640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fda201397c0 0x7fda2009d8e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda200a4270 tx=0x7fda180196b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 client.?.objecter ms_handle_connect 0x7fda201397c0 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:43.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda180bc020 con 0x7fda201397c0 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: dump: 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.085 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda18003e10 con 0x7fda201397c0 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda180044b0 con 0x7fda201397c0 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 5 monclient: authenticate success, global_id 64174 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 monclient: _renew_subs 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2009f510 con 0x7fda201397c0 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1d7fa640 4 set_mon_vals no callback set 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: dump: 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.087 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 client.64174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 monclient: _renew_subs 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fda200a0f60 con 0x7fda201397c0 2024-04-19T00:51:43.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 1 librados: init done 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 0x7fda2002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fda200a3590 con 0x7fda2009e5e0 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fda180bf020 con 0x7fda201397c0 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 0x7fda2002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient: get_auth_request con 0x7fda2009e5e0 auth_method 0 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fda0c038e10 0x7fda0c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(320..320 src has 1..320) ==== 5769+0+0 (secure 0 0 0) 0x7fda180b3040 con 0x7fda201397c0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 client.64174.objecter ms_dispatch 0x7fda20000ba0 osd_map(320..320 src has 1..320) 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 3 client.64174.objecter handle_osd_map got epochs [320,320] > 0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 3 client.64174.objecter handle_osd_map decoding full epoch 320 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 20 client.64174.objecter dump_active .. 0 homeless 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 10 monclient(hunting): handle_auth_done global_id 23348 payload 824 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda200693b0 con 0x7fda2009e5e0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1e7fc640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fda0c038e10 0x7fda0c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1e7fc640 10 monclient: get_auth_request con 0x7fda0c038e10 auth_method 0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 client.64174.objecter ms_handle_connect 0x7fda2009e5e0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 client.64174.objecter resend_mon_ops 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1effd640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 0x7fda2002c450 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fda200a72e0 tx=0x7fda14007830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda14019070 con 0x7fda2009e5e0 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: dump: 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.091 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda14010510 con 0x7fda2009e5e0 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 msgr2=0x7fda2002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 0x7fda2002c450 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fda200a72e0 tx=0x7fda14007830 comp rx=0 tx=0).stop 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 msgr2=0x7fda2002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e5e0 0x7fda2002c450 secure :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fda200a72e0 tx=0x7fda14007830 comp rx=0 tx=0).stop 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda07fff640 10 client.64174.objecter ms_handle_connect 0x7fda0c038e10 2024-04-19T00:51:43.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.068+0000 7fda1e7fc640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fda0c038e10 0x7fda0c03b2d0 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fda2009f040 tx=0x7fda10009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 0x7fda2002c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fda2006d140 con 0x7fda2009e9c0 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 0x7fda2002c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient: get_auth_request con 0x7fda2009e9c0 auth_method 0 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 10 monclient(hunting): handle_auth_done global_id 23357 payload 824 2024-04-19T00:51:43.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 1 -- 172.21.15.43:0/2821863775 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda200a2e20 con 0x7fda2009e9c0 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1effd640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 0x7fda2002c990 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fda14001ec0 tx=0x7fda14000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 client.64174.objecter ms_handle_connect 0x7fda2009e9c0 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 client.64174.objecter resend_mon_ops 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fda14019040 con 0x7fda2009e9c0 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 monclient: dump: 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.096 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7fda14010d40 con 0x7fda2009e9c0 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:43.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 msgr2=0x7fda2002c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 0x7fda2002c990 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fda14001ec0 tx=0x7fda14000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 msgr2=0x7fda2002c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda07fff640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fda2009e9c0 0x7fda2002c990 secure :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fda14001ec0 tx=0x7fda14000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:43.098 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 446575, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.099 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.100 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.101 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:43.102 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:43.103 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:43.103 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:43.103 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:43.103 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:43.103 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:43.103 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.104 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.106 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:43.109 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:43.111 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:43.112 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.113 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.115 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.118 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.119 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.121 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.122 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.123 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.124 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.125 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.125 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:43.125 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:43.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 10 librados: watch_flush enter 2024-04-19T00:51:43.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 10 librados: watch_flush exit 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 20 client.64174.objecter shutdown clearing up homeless session... 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 10 client.64174.objecter successfully canceled tick 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fda0c038e10 msgr2=0x7fda0c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fda0c038e10 0x7fda0c03b2d0 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fda2009f040 tx=0x7fda10009190 comp rx=0 tx=0).stop 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 10 monclient: shutdown 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fda201397c0 msgr2=0x7fda2009d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fda201397c0 0x7fda2009d8e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda200a4270 tx=0x7fda180196b0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda1f7fe640 1 -- 172.21.15.43:0/2821863775 reap_dead start 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 shutdown_connections 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 --2- 172.21.15.43:0/2821863775 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fda201397c0 0x7fda2009d8e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 >> 172.21.15.43:0/2821863775 conn(0x7fda200ab2f0 msgr2=0x7fda200aa0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 shutdown_connections 2024-04-19T00:51:43.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 -- 172.21.15.43:0/2821863775 wait complete. 2024-04-19T00:51:43.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.076+0000 7fda258f9640 1 librados: shutdown 2024-04-19T00:51:43.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:43.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:51:43.128 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:51:43.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient: get_monmap_and_config 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient: monmap: 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:43.259109+0000 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:43.259109+0000 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:43.257 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:43.258 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 Processor -- start 2024-04-19T00:51:43.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 -- start start 2024-04-19T00:51:43.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient: init 2024-04-19T00:51:43.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient: _add_conns ranks=[4,5,8,9,0,10,3,6,7,2,1] 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb100a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): picked mon.noname-c con 0x7feb100a1ec0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a22a0 0x7feb100a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): picked mon.noname-h con 0x7feb100a22a0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7feb100a12e0 0x7feb100a4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): picked mon.noname-e con 0x7feb100a12e0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7feb1006c4b0 con 0x7feb100a1ec0 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7feb1006bfc0 con 0x7feb100a22a0 2024-04-19T00:51:43.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7feb100a0060 con 0x7feb100a12e0 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb18628640 10 monclient(hunting): authenticate will time out at 1373.386913s 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb100a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7feb100a12e0 0x7feb100a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb100a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34222/0 (socket says 172.21.15.43:34222) 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 1 -- 172.21.15.43:0/2783692392 learned_addr learned my addr 172.21.15.43:0/2783692392 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): get_auth_request con 0x7feb100a1ec0 auth_method 0 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): get_auth_request con 0x7feb100a12e0 auth_method 0 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb15bc3640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a22a0 0x7feb100a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb15bc3640 10 monclient(hunting): get_auth_request con 0x7feb100a22a0 auth_method 0 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb15bc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb15bc3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb15bc3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient(hunting): handle_auth_done global_id 23129 payload 274 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 -- 172.21.15.43:0/2783692392 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a22a0 msgr2=0x7feb100a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a22a0 0x7feb100a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 -- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 msgr2=0x7feb100a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb100a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 monclient: found mon.noname-e 2024-04-19T00:51:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb16bc5640 1 -- 172.21.15.43:0/2783692392 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb100a0340 con 0x7feb100a12e0 2024-04-19T00:51:43.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.256+0000 7feb163c4640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb100a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb16bc5640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7feb100a12e0 0x7feb100a4430 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7feb08009dc0 tx=0x7feb0801c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=deb99b7f5c3ca64e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 1 -- 172.21.15.43:0/2783692392 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb080b2070 con 0x7feb100a12e0 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: dump: 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.276 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 5 monclient: authenticate success, global_id 23129 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 1 -- 172.21.15.43:0/2783692392 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb080a59b0 con 0x7feb100a12e0 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 4 set_mon_vals no callback set 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 1 -- 172.21.15.43:0/2783692392 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb080a5be0 con 0x7feb100a12e0 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb153c2640 10 monclient: dump: 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.278 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 10 monclient: shutdown 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 -- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7feb100a12e0 msgr2=0x7feb100a4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7feb100a12e0 0x7feb100a4430 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7feb08009dc0 tx=0x7feb0801c850 comp rx=0 tx=0).stop 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 -- 172.21.15.43:0/2783692392 shutdown_connections 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7feb100a12e0 0x7feb100a4430 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a22a0 0x7feb100a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 --2- 172.21.15.43:0/2783692392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb100a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 -- 172.21.15.43:0/2783692392 >> 172.21.15.43:0/2783692392 conn(0x7feb100a8600 msgr2=0x7feb100a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 -- 172.21.15.43:0/2783692392 shutdown_connections 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 -- 172.21.15.43:0/2783692392 wait complete. 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 10 monclient: monmap: 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.280 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.281 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 librados: starting msgr at 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 librados: starting objecter 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 Processor -- start 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 -- start start 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 librados: setting wanted keys 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 1 librados: calling monclient init 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.260+0000 7feb18628640 10 monclient: init 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient: _add_conns ranks=[5,4,2,1,8,0,3,6,7] 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a12e0 0x7feb101c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): picked mon.noname-h con 0x7feb100a12e0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb1013e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): picked mon.noname-c con 0x7feb100a1ec0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb10126470 0x7feb101268f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): picked mon.noname-b con 0x7feb10126470 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7feb10142b60 con 0x7feb10126470 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7feb101429e0 con 0x7feb100a1ec0 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7feb10142ce0 con 0x7feb100a12e0 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): authenticate will time out at 1373.392252s 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb1013e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb10126470 0x7feb101268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb1013e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34236/0 (socket says 172.21.15.43:34236) 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 -- 172.21.15.43:0/3134865460 learned_addr learned my addr 172.21.15.43:0/3134865460 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): get_auth_request con 0x7feb100a1ec0 auth_method 0 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb10126470 0x7feb101268f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:53796/0 (socket says 172.21.15.43:53796) 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): get_auth_request con 0x7feb10126470 auth_method 0 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a12e0 0x7feb101c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): get_auth_request con 0x7feb100a12e0 auth_method 0 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient(hunting): handle_auth_done global_id 64192 payload 824 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a12e0 msgr2=0x7feb101c7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7feb100a12e0 0x7feb101c7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb10126470 msgr2=0x7feb101268f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb10126470 0x7feb101268f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 monclient: found mon.noname-c 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb101280a0 con 0x7feb100a1ec0 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7feb10126470 0x7feb101268f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb15bc3640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb1013e9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feb0000d5f0 tx=0x7feb0000dab0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 client.?.objecter ms_handle_connect 0x7feb100a1ec0 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb00012a60 con 0x7feb100a1ec0 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: dump: 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.287 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb00019070 con 0x7feb100a1ec0 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb00012c20 con 0x7feb100a1ec0 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: dump: 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.289 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb14bc1640 4 set_mon_vals no callback set 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 5 monclient: authenticate success, global_id 64192 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient: _renew_subs 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1009f4a0 con 0x7feb100a1ec0 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 client.64192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient: _renew_subs 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1012a390 con 0x7feb100a1ec0 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 librados: init done 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 0x7feb10126e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7feb1006d140 con 0x7feb100a22a0 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 0x7feb10126e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient: get_auth_request con 0x7feb100a22a0 auth_method 0 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7feb000125f0 con 0x7feb100a1ec0 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 10 monclient(hunting): handle_auth_done global_id 64201 payload 824 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb1009ef70 con 0x7feb100a22a0 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7feaf4038e40 0x7feaf403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(320..320 src has 1..320) ==== 5769+0+0 (secure 0 0 0) 0x7feb0000f040 con 0x7feb100a1ec0 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 client.64192.objecter ms_dispatch 0x7feb10000ba0 osd_map(320..320 src has 1..320) 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 3 client.64192.objecter handle_osd_map got epochs [320,320] > 0 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 3 client.64192.objecter handle_osd_map decoding full epoch 320 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb163c4640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 0x7feb10126e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb0c011600 tx=0x7feb0c011ac0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 20 client.64192.objecter dump_active .. 0 homeless 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 client.64192.objecter ms_handle_connect 0x7feb100a22a0 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 client.64192.objecter resend_mon_ops 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb0c00fba0 con 0x7feb100a22a0 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: dump: 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.293 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7feaf4038e40 0x7feaf403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 10 monclient: get_auth_request con 0x7feaf4038e40 auth_method 0 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb0c00f730 con 0x7feb100a22a0 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:43.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 msgr2=0x7feb10126e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 0x7feb10126e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb0c011600 tx=0x7feb0c011ac0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 msgr2=0x7feb10126e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a22a0 0x7feb10126e30 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feb0c011600 tx=0x7feb0c011ac0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb16bc5640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7feaf4038e40 0x7feaf403b300 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7feb080af000 tx=0x7feb080a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.264+0000 7feb06ffd640 10 client.64192.objecter ms_handle_connect 0x7feaf4038e40 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 0x7feb10122a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7feb10076500 con 0x7feb10127370 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 0x7feb10122a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient: get_auth_request con 0x7feb10127370 auth_method 0 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 10 monclient(hunting): handle_auth_done global_id 64210 payload 824 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 1 -- 172.21.15.43:0/3134865460 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb10127780 con 0x7feb10127370 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb163c4640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 0x7feb10122a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c10 tx=0x7feb0c0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 client.64192.objecter ms_handle_connect 0x7feb10127370 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 client.64192.objecter resend_mon_ops 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7feb0c0064b0 con 0x7feb10127370 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 monclient: dump: 2024-04-19T00:51:43.296 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.297 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7feb0c006040 con 0x7feb10127370 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 msgr2=0x7feb10122a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 0x7feb10122a00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c10 tx=0x7feb0c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 msgr2=0x7feb10122a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb06ffd640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb10127370 0x7feb10122a00 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feb0c000c10 tx=0x7feb0c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.298 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 29543, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:43.299 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.300 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:43.302 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.303 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.304 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.305 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.306 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.307 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:43.308 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:43.309 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.310 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.311 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.312 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:43.313 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.314 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.315 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.316 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.318 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.319 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.321 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:43.321 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:43.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 10 librados: watch_flush enter 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 10 librados: watch_flush exit 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 20 client.64192.objecter shutdown clearing up homeless session... 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 10 client.64192.objecter successfully canceled tick 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7feaf4038e40 msgr2=0x7feaf403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7feaf4038e40 0x7feaf403b300 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7feb080af000 tx=0x7feb080a7070 comp rx=0 tx=0).stop 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 10 monclient: shutdown 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 msgr2=0x7feb1013e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 --2- 172.21.15.43:0/3134865460 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7feb100a1ec0 0x7feb1013e9f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feb0000d5f0 tx=0x7feb0000dab0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb16bc5640 1 -- 172.21.15.43:0/3134865460 reap_dead start 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb16bc5640 1 -- 172.21.15.43:0/3134865460 reap_dead start 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 shutdown_connections 2024-04-19T00:51:43.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 >> 172.21.15.43:0/3134865460 conn(0x7feb100a8600 msgr2=0x7feb10071000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 shutdown_connections 2024-04-19T00:51:43.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 -- 172.21.15.43:0/3134865460 wait complete. 2024-04-19T00:51:43.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.272+0000 7feb18628640 1 librados: shutdown 2024-04-19T00:51:43.324 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:43.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:51:43.324 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:51:43.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient: get_monmap_and_config 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient: monmap: 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:43.446872+0000 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:43.446872+0000 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.445 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:43.446 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:43.447 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 Processor -- start 2024-04-19T00:51:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 -- start start 2024-04-19T00:51:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient: init 2024-04-19T00:51:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient: _add_conns ranks=[0,8,4,10,1,2,6,9,3,7,5] 2024-04-19T00:51:43.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dc8138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9dc8139540 0x7f9dc813dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): picked mon.noname-e con 0x7f9dc8139540 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9dc80a5bb0 0x7f9dc8140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): picked mon.noname-c con 0x7f9dc80a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc80a0d40 con 0x7f9dc8138a50 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9dc80a1020 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f9dc80a0780 con 0x7f9dc8139540 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dcd1be640 10 monclient(hunting): authenticate will time out at 1373.574020s 2024-04-19T00:51:43.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc6d9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9dc8138a50 0x7f9dc8138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51356/0 (socket says 172.21.15.43:51356) 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc6d9d640 1 -- 172.21.15.43:0/2063760255 learned_addr learned my addr 172.21.15.43:0/2063760255 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9dc80a5bb0 0x7f9dc8140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): get_auth_request con 0x7f9dc80a5bb0 auth_method 0 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9dc8139540 0x7f9dc813dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): get_auth_request con 0x7f9dc8139540 auth_method 0 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient(hunting): handle_auth_done global_id 64219 payload 274 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc5d9b640 1 -- 172.21.15.43:0/2063760255 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2090076545 0 0) 0x7f9dc80a0d40 con 0x7f9dc8138a50 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 1 -- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9dc8139540 msgr2=0x7f9dc813dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9dc8139540 0x7f9dc813dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 1 -- 172.21.15.43:0/2063760255 >> v1:172.21.15.43:6789/0 conn(0x7f9dc8138a50 legacy=0x7f9dc8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 1 monclient: found mon.noname-c 2024-04-19T00:51:43.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:51:43.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc759e640 1 -- 172.21.15.43:0/2063760255 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc80a0a60 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.444+0000 7f9dc5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc759e640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9dc80a5bb0 0x7f9dc8140530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009dc0 tx=0x7f9dc001c850 comp rx=0 tx=0).ready entity=mon.4 client_cookie=2212b0f7e9f6744a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 1 -- 172.21.15.43:0/2063760255 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9dc00b2070 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: dump: 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.459 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.460 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 1 -- 172.21.15.43:0/2063760255 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc00a5a50 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 1 -- 172.21.15.43:0/2063760255 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9dc00a5c40 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 5 monclient: authenticate success, global_id 64219 2024-04-19T00:51:43.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 4 set_mon_vals no callback set 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dc5d9b640 10 monclient: dump: 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.462 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.463 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.464 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: shutdown 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 -- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9dc80a5bb0 msgr2=0x7f9dc8140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9dc80a5bb0 0x7f9dc8140530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009dc0 tx=0x7f9dc001c850 comp rx=0 tx=0).stop 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 -- 172.21.15.43:0/2063760255 shutdown_connections 2024-04-19T00:51:43.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9dc80a5bb0 0x7f9dc8140530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/2063760255 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9dc8139540 0x7f9dc813dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 -- 172.21.15.43:0/2063760255 >> 172.21.15.43:0/2063760255 conn(0x7f9dc80a7cb0 msgr2=0x7f9dc80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 -- 172.21.15.43:0/2063760255 shutdown_connections 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 -- 172.21.15.43:0/2063760255 wait complete. 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: monmap: 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:43.466 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 librados: starting msgr at 2024-04-19T00:51:43.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 librados: starting objecter 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 Processor -- start 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 -- start start 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 librados: setting wanted keys 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 librados: calling monclient init 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: init 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient: _add_conns ranks=[0,5,7,2,8,3,6,4,1] 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9dc80a5bb0 0x7f9dc8137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient(hunting): picked mon.noname-a con 0x7f9dc80a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9dc8138a50 0x7f9dc813c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient(hunting): picked mon.noname-h con 0x7f9dc8138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9dc8139540 0x7f9dc813cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient(hunting): picked mon.noname-i con 0x7f9dc8139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.448+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc8143620 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f9dc81434a0 con 0x7f9dc8138a50 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9dc81437a0 con 0x7f9dc8139540 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient(hunting): authenticate will time out at 1373.580028s 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9dc80a5bb0 0x7f9dc8137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9dc80a5bb0 0x7f9dc8137820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:39994/0 (socket says 172.21.15.43:39994) 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 -- 172.21.15.43:0/1935673661 learned_addr learned my addr 172.21.15.43:0/1935673661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): get_auth_request con 0x7f9dc80a5bb0 auth_method 0 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient(hunting): handle_auth_done global_id 64278 payload 824 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9dc8138a50 0x7f9dc813c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc759e640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9dc8139540 0x7f9dc813cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9dc8139540 msgr2=0x7f9dc813cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9dc8139540 0x7f9dc813cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9dc8138a50 msgr2=0x7f9dc813c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9dc8138a50 0x7f9dc813c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 monclient: found mon.noname-a 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc81221d0 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc6d9d640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9dc80a5bb0 0x7f9dc8137820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9db000c450 tx=0x7f9db000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 client.?.objecter ms_handle_connect 0x7f9dc80a5bb0 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9db0018070 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: dump: 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.471 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db00047f0 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 5 monclient: authenticate success, global_id 64278 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient: _renew_subs 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc809fc90 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9db0004e50 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc559a640 4 set_mon_vals no callback set 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 client.64278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: dump: 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.473 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.474 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient: _renew_subs 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc8121ef0 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 librados: init done 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 0x7f9dc813d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f9dc806d140 con 0x7f9dc813cff0 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9db00049e0 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 0x7f9dc813d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient: get_auth_request con 0x7f9dc813cff0 auth_method 0 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9da8038fb0 0x7f9da803b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 10 monclient(hunting): handle_auth_done global_id 64194 payload 824 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9dc813d910 con 0x7f9dc813cff0 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(320..320 src has 1..320) ==== 5769+0+0 (secure 0 0 0) 0x7f9db0048470 con 0x7f9dc80a5bb0 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 client.64278.objecter ms_dispatch 0x7f9dc8000ba0 osd_map(320..320 src has 1..320) 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 3 client.64278.objecter handle_osd_map got epochs [320,320] > 0 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 3 client.64278.objecter handle_osd_map decoding full epoch 320 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 20 client.64278.objecter dump_active .. 0 homeless 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 client.64278.objecter ms_handle_connect 0x7f9dc813cff0 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc659c640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 0x7f9dc813d3d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9dc806e8c0 tx=0x7f9db8010c20 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 client.64278.objecter resend_mon_ops 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc759e640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9da8038fb0 0x7f9da803b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc759e640 10 monclient: get_auth_request con 0x7f9da8038fb0 auth_method 0 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9db800f040 con 0x7f9dc813cff0 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: dump: 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.477 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9db801c400 con 0x7f9dc813cff0 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 msgr2=0x7f9dc813d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 0x7f9dc813d3d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9dc806e8c0 tx=0x7f9db8010c20 comp rx=0 tx=0).stop 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 msgr2=0x7f9dc813d3d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc813cff0 0x7f9dc813d3d0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9dc806e8c0 tx=0x7f9db8010c20 comp rx=0 tx=0).stop 2024-04-19T00:51:43.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dbf7fe640 10 client.64278.objecter ms_handle_connect 0x7f9da8038fb0 2024-04-19T00:51:43.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.452+0000 7f9dc759e640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9da8038fb0 0x7f9da803b470 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00af000 tx=0x7f9dc00a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:43.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 0x7f9dc80a3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f9dc81d14c0 con 0x7f9dc80a3040 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 0x7f9dc80a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient: get_auth_request con 0x7f9dc80a3040 auth_method 0 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 10 monclient(hunting): handle_auth_done global_id 64203 payload 824 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 1 -- 172.21.15.43:0/1935673661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9dc813eaf0 con 0x7f9dc80a3040 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 client.64278.objecter ms_handle_connect 0x7f9dc80a3040 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 client.64278.objecter resend_mon_ops 2024-04-19T00:51:43.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc659c640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 0x7f9dc80a3450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9db8000c10 tx=0x7f9db80046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9db800ffc0 con 0x7f9dc80a3040 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 monclient: dump: 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.482 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9442 (secure 0 0 0) 0x7f9db800fb50 con 0x7f9dc80a3040 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 msgr2=0x7f9dc80a3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 0x7f9dc80a3450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9db8000c10 tx=0x7f9db80046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 msgr2=0x7f9dc80a3450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dbf7fe640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9dc80a3040 0x7f9dc80a3450 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9db8000c10 tx=0x7f9db80046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 29735, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:43.484 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.485 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.486 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:43.487 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.488 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.489 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.490 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.491 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.492 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:43.493 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.494 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.495 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.496 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:43.497 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.498 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.499 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.500 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.501 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.502 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.503 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.504 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.505 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.506 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:43.507 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 10 librados: watch_flush enter 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 10 librados: watch_flush exit 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 20 client.64278.objecter shutdown clearing up homeless session... 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 10 client.64278.objecter successfully canceled tick 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9da8038fb0 msgr2=0x7f9da803b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9da8038fb0 0x7f9da803b470 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00af000 tx=0x7f9dc00a7070 comp rx=0 tx=0).stop 2024-04-19T00:51:43.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 10 monclient: shutdown 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9dc80a5bb0 msgr2=0x7f9dc8137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 --2- 172.21.15.43:0/1935673661 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9dc80a5bb0 0x7f9dc8137820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9db000c450 tx=0x7f9db000c910 comp rx=0 tx=0).stop 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc759e640 1 -- 172.21.15.43:0/1935673661 reap_dead start 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dc759e640 1 -- 172.21.15.43:0/1935673661 reap_dead start 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 shutdown_connections 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 >> 172.21.15.43:0/1935673661 conn(0x7f9dc80a7cb0 msgr2=0x7f9dc80a9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 shutdown_connections 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 -- 172.21.15.43:0/1935673661 wait complete. 2024-04-19T00:51:43.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.460+0000 7f9dcd1be640 1 librados: shutdown 2024-04-19T00:51:43.511 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:43.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:51:43.511 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:51:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.624+0000 7f4df5dfc640 10 monclient: get_monmap_and_config 2024-04-19T00:51:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient: monmap: 2024-04-19T00:51:43.628 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:43.630592+0000 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:43.630592+0000 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.629 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:43.630 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:43.630 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 Processor -- start 2024-04-19T00:51:43.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 -- start start 2024-04-19T00:51:43.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient: init 2024-04-19T00:51:43.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient: _add_conns ranks=[3,9,0,1,10,2,7,4,8,6,5] 2024-04-19T00:51:43.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4df009de40 0x7f4df00a3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): picked mon.noname-g con 0x7f4df009de40 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4df009e220 0x7f4df00a3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): picked mon.noname-a con 0x7f4df009e220 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4df009ae90 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df009bb10 con 0x7f4df009ae90 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f4df009d020 con 0x7f4df009e220 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4df00766c0 con 0x7f4df009de40 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4df5dfc640 10 monclient(hunting): authenticate will time out at 1373.757863s 2024-04-19T00:51:43.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4df009e220 0x7f4df00a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deffff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4df009ae90 0x7f4df013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51364/0 (socket says 172.21.15.43:51364) 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deffff640 1 -- 172.21.15.43:0/75803122 learned_addr learned my addr 172.21.15.43:0/75803122 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): get_auth_request con 0x7f4df009e220 auth_method 0 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4def7fe640 1 --2- 172.21.15.43:0/75803122 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4df009de40 0x7f4df00a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4def7fe640 10 monclient(hunting): get_auth_request con 0x7f4df009de40 auth_method 0 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4def7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4def7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient(hunting): handle_auth_done global_id 64287 payload 274 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 1 -- 172.21.15.43:0/75803122 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4df009de40 msgr2=0x7f4df00a3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 1 --2- 172.21.15.43:0/75803122 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4df009de40 0x7f4df00a3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 1 -- 172.21.15.43:0/75803122 >> v1:172.21.15.43:6789/0 conn(0x7f4df009ae90 legacy=0x7f4df013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 1 monclient: found mon.noname-a 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4dee7fc640 1 -- 172.21.15.43:0/75803122 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786401123 0 0) 0x7f4df009bb10 con 0x7f4df009ae90 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4deeffd640 1 -- 172.21.15.43:0/75803122 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df009c6f0 con 0x7f4df009e220 2024-04-19T00:51:43.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.628+0000 7f4dee7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4deeffd640 1 --2- 172.21.15.43:0/75803122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df00a3590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097c0 tx=0x7f4dd801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93369d400043970c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 1 -- 172.21.15.43:0/75803122 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4dd80b0070 con 0x7f4df009e220 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: dump: 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.638 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 1 -- 172.21.15.43:0/75803122 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dd80aa810 con 0x7f4df009e220 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 5 monclient: authenticate success, global_id 64287 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 1 -- 172.21.15.43:0/75803122 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4dd80a9300 con 0x7f4df009e220 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4dee7fc640 10 monclient: dump: 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.640 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 4 set_mon_vals no callback set 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 10 monclient: shutdown 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 -- 172.21.15.43:0/75803122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 msgr2=0x7f4df00a3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/75803122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df00a3590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097c0 tx=0x7f4dd801c330 comp rx=0 tx=0).stop 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 -- 172.21.15.43:0/75803122 shutdown_connections 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/75803122 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df00a3590 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/75803122 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4df009de40 0x7f4df00a3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 -- 172.21.15.43:0/75803122 >> 172.21.15.43:0/75803122 conn(0x7f4df00a4140 msgr2=0x7f4df00a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 -- 172.21.15.43:0/75803122 shutdown_connections 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 -- 172.21.15.43:0/75803122 wait complete. 2024-04-19T00:51:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 10 monclient: monmap: 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 librados: starting msgr at 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 librados: starting objecter 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.632+0000 7f4df5dfc640 1 Processor -- start 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 -- start start 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 librados: setting wanted keys 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 librados: calling monclient init 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: init 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: _add_conns ranks=[4,8,0,5,2,6,1,7,3] 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4df009ae90 0x7f4df013e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): picked mon.noname-c con 0x7f4df009ae90 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df009de40 0x7f4df013e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): picked mon.noname-e con 0x7f4df009de40 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df013ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): picked mon.noname-a con 0x7f4df009e220 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4df01438c0 con 0x7f4df009e220 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4df0143740 con 0x7f4df009ae90 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4df0143a40 con 0x7f4df009de40 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient(hunting): authenticate will time out at 1373.764831s 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df013ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df013ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40006/0 (socket says 172.21.15.43:40006) 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 -- 172.21.15.43:0/1811885441 learned_addr learned my addr 172.21.15.43:0/1811885441 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): get_auth_request con 0x7f4df009e220 auth_method 0 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient(hunting): handle_auth_done global_id 64305 payload 824 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df009de40 msgr2=0x7f4df013e8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df009de40 0x7f4df013e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4df009ae90 msgr2=0x7f4df013e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4df009ae90 0x7f4df013e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 monclient: found mon.noname-a 2024-04-19T00:51:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df009e620 con 0x7f4df009e220 2024-04-19T00:51:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4deffff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df013ee30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c410 tx=0x7f4de800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 client.?.objecter ms_handle_connect 0x7f4df009e220 2024-04-19T00:51:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4de80047f0 con 0x7f4df009e220 2024-04-19T00:51:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:43.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: dump: 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.650 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de80049f0 con 0x7f4df009e220 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4de8004ba0 con 0x7f4df009e220 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4de7fff640 10 monclient: dump: 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.651 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 5 monclient: authenticate success, global_id 64305 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: _renew_subs 2024-04-19T00:51:43.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df013f6e0 con 0x7f4df009e220 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4dedffb640 4 set_mon_vals no callback set 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 client.64305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: _renew_subs 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df008de50 con 0x7f4df009e220 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.636+0000 7f4df5dfc640 1 librados: init done 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4df5dfc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 0x7f4df01d4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4df006d140 con 0x7f4df01d4a70 2024-04-19T00:51:43.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 0x7f4df01d4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient: get_auth_request con 0x7f4df01d4a70 auth_method 0 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4de8011070 con 0x7f4df009e220 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4dd0038e10 0x7f4dd003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7f4de8049010 con 0x7f4df009e220 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 10 client.64305.objecter ms_dispatch 0x7f4df0000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:43.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 3 client.64305.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 3 client.64305.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 20 client.64305.objecter dump_active .. 0 homeless 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4deeffd640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4dd0038e10 0x7f4dd003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4deeffd640 10 monclient: get_auth_request con 0x7f4dd0038e10 auth_method 0 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 10 monclient(hunting): handle_auth_done global_id 23147 payload 824 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4def7fe640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4df00008e0 con 0x7f4df01d4a70 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4de7fff640 10 client.64305.objecter ms_handle_connect 0x7f4dd0038e10 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.640+0000 7f4deeffd640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4dd0038e10 0x7f4dd003b2d0 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80ad000 tx=0x7f4dd80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4def7fe640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 0x7f4df01d4e50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4de0007d90 tx=0x7f4de000e4b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 client.64305.objecter ms_handle_connect 0x7f4df01d4a70 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 client.64305.objecter resend_mon_ops 2024-04-19T00:51:43.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4de000ede0 con 0x7f4df01d4a70 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 monclient: dump: 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.656 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4de000ede0 con 0x7f4df01d4a70 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 msgr2=0x7f4df01d4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 0x7f4df01d4e50 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4de0007d90 tx=0x7f4de000e4b0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 msgr2=0x7f4df01d4e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.644+0000 7f4de7fff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df01d4a70 0x7f4df01d4e50 secure :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f4de0007d90 tx=0x7f4de000e4b0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 0x7f4df01d5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4df01d58d0 con 0x7f4df008d5b0 2024-04-19T00:51:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 0x7f4df01d5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient: get_auth_request con 0x7f4df008d5b0 auth_method 0 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 10 monclient(hunting): handle_auth_done global_id 23156 payload 824 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 1 -- 172.21.15.43:0/1811885441 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4df0068240 con 0x7f4df008d5b0 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4def7fe640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 0x7f4df01d5390 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4de0007d90 tx=0x7f4de0004a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 client.64305.objecter ms_handle_connect 0x7f4df008d5b0 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 client.64305.objecter resend_mon_ops 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4de001d770 con 0x7f4df008d5b0 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:43.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 monclient: dump: 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.661 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f4de00045d0 con 0x7f4df008d5b0 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 msgr2=0x7f4df01d5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 0x7f4df01d5390 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4de0007d90 tx=0x7f4de0004a50 comp rx=0 tx=0).stop 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 msgr2=0x7f4df01d5390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4de7fff640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4df008d5b0 0x7f4df01d5390 secure :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f4de0007d90 tx=0x7f4de0004a50 comp rx=0 tx=0).stop 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:43.662 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 447163, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:43.663 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.664 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.665 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:43.666 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.667 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.670 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.675 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.680 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:43.681 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:43.682 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.683 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:43.684 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:43.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 10 librados: watch_flush enter 2024-04-19T00:51:43.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 10 librados: watch_flush exit 2024-04-19T00:51:43.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 20 client.64305.objecter shutdown clearing up homeless session... 2024-04-19T00:51:43.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 10 client.64305.objecter successfully canceled tick 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4dd0038e10 msgr2=0x7f4dd003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4dd0038e10 0x7f4dd003b2d0 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80ad000 tx=0x7f4dd80a6000 comp rx=0 tx=0).stop 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 10 monclient: shutdown 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 msgr2=0x7f4df013ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4deffff640 1 -- 172.21.15.43:0/1811885441 reap_dead start 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df013ee30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4de800c410 tx=0x7f4de800c8d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 shutdown_connections 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 --2- 172.21.15.43:0/1811885441 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4df009e220 0x7f4df013ee30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 >> 172.21.15.43:0/1811885441 conn(0x7f4df00a4140 msgr2=0x7f4df009f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 shutdown_connections 2024-04-19T00:51:43.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 -- 172.21.15.43:0/1811885441 wait complete. 2024-04-19T00:51:43.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.652+0000 7f4df5dfc640 1 librados: shutdown 2024-04-19T00:51:43.687 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:43.687 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:51:43.687 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:51:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient: get_monmap_and_config 2024-04-19T00:51:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient: monmap: 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:43.807018+0000 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:43.807018+0000 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:43.806 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 Processor -- start 2024-04-19T00:51:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 -- start start 2024-04-19T00:51:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient: init 2024-04-19T00:51:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient: _add_conns ranks=[5,8,3,10,0,4,6,2,1,9,7] 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a98136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): picked mon.noname-h con 0x7f0a9809e4c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a980a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): picked mon.noname-e con 0x7f0a9809e8a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a9809eca0 0x7f0a980a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): picked mon.noname-g con 0x7f0a9809eca0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0a980a1c50 con 0x7f0a9809eca0 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a980a1e20 con 0x7f0a9809e4c0 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a9806c760 con 0x7f0a9809e8a0 2024-04-19T00:51:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9d105640 10 monclient(hunting): authenticate will time out at 1373.933838s 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a980a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a980a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52976/0 (socket says 172.21.15.43:52976) 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 -- 172.21.15.43:0/1119273217 learned_addr learned my addr 172.21.15.43:0/1119273217 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): get_auth_request con 0x7f0a9809e8a0 auth_method 0 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9759e640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a9809eca0 0x7f0a980a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a98136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9759e640 10 monclient(hunting): get_auth_request con 0x7f0a9809eca0 auth_method 0 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a96d9d640 10 monclient(hunting): get_auth_request con 0x7f0a9809e4c0 auth_method 0 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient(hunting): handle_auth_done global_id 23165 payload 274 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 -- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 msgr2=0x7f0a98136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a98136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 -- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a9809eca0 msgr2=0x7f0a980a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a9809eca0 0x7f0a980a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 monclient: found mon.noname-e 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.804+0000 7f0a9659c640 1 -- 172.21.15.43:0/1119273217 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a980a5510 con 0x7f0a9809e8a0 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a980a3940 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009e80 tx=0x7f0a8801c9a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=e4f675a9cd777c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 1 -- 172.21.15.43:0/1119273217 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a880090d0 con 0x7f0a9809e8a0 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: dump: 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 1 -- 172.21.15.43:0/1119273217 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a880a59f0 con 0x7f0a9809e8a0 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 5 monclient: authenticate success, global_id 23165 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 4 set_mon_vals no callback set 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 1 -- 172.21.15.43:0/1119273217 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a880a5be0 con 0x7f0a9809e8a0 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a95d9b640 10 monclient: dump: 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: shutdown 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- 172.21.15.43:0/1119273217 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 msgr2=0x7f0a980a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a980a3940 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f0a88009e80 tx=0x7f0a8801c9a0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- 172.21.15.43:0/1119273217 shutdown_connections 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a9809eca0 0x7f0a980a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a980a3940 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- 172.21.15.43:0/1119273217 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a98136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- 172.21.15.43:0/1119273217 >> 172.21.15.43:0/1119273217 conn(0x7f0a980a85c0 msgr2=0x7f0a980a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- 172.21.15.43:0/1119273217 shutdown_connections 2024-04-19T00:51:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- 172.21.15.43:0/1119273217 wait complete. 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: monmap: 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 librados: starting msgr at 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 librados: starting objecter 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 Processor -- start 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- start start 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 librados: setting wanted keys 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 librados: calling monclient init 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: init 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient: _add_conns ranks=[5,8,4,2,6,7,0,1,3] 2024-04-19T00:51:43.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a9809d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): picked mon.noname-h con 0x7f0a9809e4c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a9809f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): picked mon.noname-e con 0x7f0a9809e8a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a9809eca0 0x7f0a9809f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): picked mon.noname-c con 0x7f0a9809eca0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0a981406e0 con 0x7f0a9809eca0 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0a980a47d0 con 0x7f0a9809e4c0 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a98140860 con 0x7f0a9809e8a0 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9d105640 10 monclient(hunting): authenticate will time out at 1373.939058s 2024-04-19T00:51:43.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a9809f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a9809f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52980/0 (socket says 172.21.15.43:52980) 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 -- 172.21.15.43:0/2572157476 learned_addr learned my addr 172.21.15.43:0/2572157476 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): get_auth_request con 0x7f0a9809e8a0 auth_method 0 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a9809eca0 0x7f0a9809f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): get_auth_request con 0x7f0a9809eca0 auth_method 0 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a9809d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient(hunting): handle_auth_done global_id 23174 payload 824 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 msgr2=0x7f0a9809d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a9809d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a9809eca0 msgr2=0x7f0a9809f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a9809eca0 0x7f0a9809f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 monclient: found mon.noname-e 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a980a0d00 con 0x7f0a9809e8a0 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a96d9d640 10 monclient: get_auth_request con 0x7f0a9809e4c0 auth_method 0 2024-04-19T00:51:43.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0a9809e4c0 0x7f0a9809d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a9659c640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a9809f430 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0a880af040 tx=0x7f0a88019740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a7f7fe640 10 client.?.objecter ms_handle_connect 0x7f0a9809e8a0 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.808+0000 7f0a7f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a880bc020 con 0x7f0a9809e8a0 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: dump: 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.830 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:43.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 5 monclient: authenticate success, global_id 23174 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a88003e10 con 0x7f0a9809e8a0 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 monclient: _renew_subs 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a880044b0 con 0x7f0a9809e8a0 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a98133350 con 0x7f0a9809e8a0 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a94d99640 4 set_mon_vals no callback set 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: dump: 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.832 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 client.23174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:43.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 monclient: _renew_subs 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9806b9e0 con 0x7f0a9809e8a0 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 1 librados: init done 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 0x7f0a9809feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9806d140 con 0x7f0a98132eb0 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0a880bf020 con 0x7f0a9809e8a0 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a84038e10 0x7f0a8403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7f0a880e44c0 con 0x7f0a9809e8a0 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 client.23174.objecter ms_dispatch 0x7f0a98000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 3 client.23174.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 3 client.23174.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:43.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 0x7f0a9809feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient: get_auth_request con 0x7f0a98132eb0 auth_method 0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 20 client.23174.objecter dump_active .. 0 homeless 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9759e640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a84038e10 0x7f0a8403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9759e640 10 monclient: get_auth_request con 0x7f0a84038e10 auth_method 0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_done global_id 64207 payload 824 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a980a05b0 con 0x7f0a98132eb0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 client.23174.objecter ms_handle_connect 0x7f0a98132eb0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 0x7f0a9809feb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00e740 tx=0x7f0a8c00ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 client.23174.objecter resend_mon_ops 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 client.23174.objecter ms_handle_connect 0x7f0a84038e10 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a9759e640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a84038e10 0x7f0a8403b2d0 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f0a9809df40 tx=0x7f0a90008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a8c017070 con 0x7f0a98132eb0 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: dump: 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.836 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a8c00ed30 con 0x7f0a98132eb0 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 msgr2=0x7f0a9809feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 0x7f0a9809feb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00e740 tx=0x7f0a8c00ec00 comp rx=0 tx=0).stop 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 msgr2=0x7f0a9809feb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.812+0000 7f0a7f7fe640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a98132eb0 0x7f0a9809feb0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c00e740 tx=0x7f0a8c00ec00 comp rx=0 tx=0).stop 2024-04-19T00:51:43.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a9d105640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:43.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a9d105640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 0x7f0a980a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a9d105640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a98132b40 con 0x7f0a980a5e10 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 0x7f0a980a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient: get_auth_request con 0x7f0a980a5e10 auth_method 0 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 10 monclient(hunting): handle_auth_done global_id 64216 payload 824 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 1 -- 172.21.15.43:0/2572157476 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a98073050 con 0x7f0a980a5e10 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 client.23174.objecter ms_handle_connect 0x7f0a980a5e10 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 client.23174.objecter resend_mon_ops 2024-04-19T00:51:43.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a96d9d640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 0x7f0a980a6220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c10 tx=0x7f0a8c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a8c00f040 con 0x7f0a980a5e10 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 monclient: dump: 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.841 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f0a8c003620 con 0x7f0a980a5e10 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 msgr2=0x7f0a980a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 0x7f0a980a6220 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c10 tx=0x7f0a8c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 msgr2=0x7f0a980a6220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:43.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.820+0000 7f0a7f7fe640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a980a5e10 0x7f0a980a6220 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c000c10 tx=0x7f0a8c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 30087, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.849 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:43.851 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.852 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.853 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.854 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.856 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.857 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.858 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.859 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.860 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.861 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:43.862 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:43.863 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:43.864 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:43.865 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:43.865 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:43.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:43.865 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:43.865 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:43.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 10 librados: watch_flush enter 2024-04-19T00:51:43.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 10 librados: watch_flush exit 2024-04-19T00:51:43.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 20 client.23174.objecter shutdown clearing up homeless session... 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 10 client.23174.objecter successfully canceled tick 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a84038e10 msgr2=0x7f0a8403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a84038e10 0x7f0a8403b2d0 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f0a9809df40 tx=0x7f0a90008040 comp rx=0 tx=0).stop 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 10 monclient: shutdown 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 msgr2=0x7f0a9809f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 --2- 172.21.15.43:0/2572157476 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a9809e8a0 0x7f0a9809f430 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0a880af040 tx=0x7f0a88019740 comp rx=0 tx=0).stop 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9759e640 1 -- 172.21.15.43:0/2572157476 reap_dead start 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9759e640 1 -- 172.21.15.43:0/2572157476 reap_dead start 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 shutdown_connections 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 >> 172.21.15.43:0/2572157476 conn(0x7f0a980a85c0 msgr2=0x7f0a98130dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 shutdown_connections 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 -- 172.21.15.43:0/2572157476 wait complete. 2024-04-19T00:51:43.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.824+0000 7f0a9d105640 1 librados: shutdown 2024-04-19T00:51:43.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:43.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:51:43.868 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:51:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient: get_monmap_and_config 2024-04-19T00:51:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient: build_initial_monmap 2024-04-19T00:51:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient: monmap: 2024-04-19T00:51:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:43.987645+0000 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:43.987645+0000 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:43.987 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 Processor -- start 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 -- start start 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient: init 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient: _add_conns ranks=[2,0,5,8,7,1,3,10,6,4,9] 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d4138cd0 0x7f65d41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): picked mon.noname-b con 0x7f65d4138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65d40a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d41395f0 0x7f65d4135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): picked mon.noname-h con 0x7f65d41395f0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65d40a3dc0 con 0x7f65d40a5b80 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f65d40a4410 con 0x7f65d4138cd0 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f65d40a40a0 con 0x7f65d41395f0 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65da103640 10 monclient(hunting): authenticate will time out at 1374.114801s 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d4138cd0 0x7f65d41390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d2ffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f65d40a5b80 0x7f65d41358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51378/0 (socket says 172.21.15.43:51378) 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d2ffd640 1 -- 172.21.15.43:0/2654510373 learned_addr learned my addr 172.21.15.43:0/2654510373 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): get_auth_request con 0x7f65d4138cd0 auth_method 0 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d3fff640 1 --2- 172.21.15.43:0/2654510373 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d41395f0 0x7f65d4135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient(hunting): handle_auth_done global_id 23375 payload 274 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d27fc640 1 -- 172.21.15.43:0/2654510373 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4120472014 0 0) 0x7f65d40a3dc0 con 0x7f65d40a5b80 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 1 -- 172.21.15.43:0/2654510373 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d41395f0 msgr2=0x7f65d4135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 1 --2- 172.21.15.43:0/2654510373 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d41395f0 0x7f65d4135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 1 -- 172.21.15.43:0/2654510373 >> v1:172.21.15.43:6789/0 conn(0x7f65d40a5b80 legacy=0x7f65d41358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 1 monclient: found mon.noname-b 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:51:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d37fe640 1 -- 172.21.15.43:0/2654510373 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d4139a80 con 0x7f65d4138cd0 2024-04-19T00:51:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.983+0000 7f65d27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:51:43.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d37fe640 1 --2- 172.21.15.43:0/2654510373 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d4138cd0 0x7f65d41390b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009dc0 tx=0x7f65cc01c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1115094ab3d8b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:43.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 1 -- 172.21.15.43:0/2654510373 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65cc0b2070 con 0x7f65d4138cd0 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: dump: 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 1 -- 172.21.15.43:0/2654510373 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc0a5b00 con 0x7f65d4138cd0 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 5 monclient: authenticate success, global_id 23375 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 1 -- 172.21.15.43:0/2654510373 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65cc0a5cf0 con 0x7f65d4138cd0 2024-04-19T00:51:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 4 set_mon_vals no callback set 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65d27fc640 10 monclient: dump: 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: shutdown 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 -- 172.21.15.43:0/2654510373 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d4138cd0 msgr2=0x7f65d41390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 --2- 172.21.15.43:0/2654510373 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d4138cd0 0x7f65d41390b0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009dc0 tx=0x7f65cc01c7f0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 -- 172.21.15.43:0/2654510373 shutdown_connections 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 --2- 172.21.15.43:0/2654510373 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d41395f0 0x7f65d4135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 --2- 172.21.15.43:0/2654510373 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d4138cd0 0x7f65d41390b0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 -- 172.21.15.43:0/2654510373 >> 172.21.15.43:0/2654510373 conn(0x7f65d40ab2f0 msgr2=0x7f65d40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 -- 172.21.15.43:0/2654510373 shutdown_connections 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 -- 172.21.15.43:0/2654510373 wait complete. 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: monmap: 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.003 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 librados: starting msgr at 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 librados: starting objecter 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 Processor -- start 2024-04-19T00:51:44.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 -- start start 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 librados: setting wanted keys 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 librados: calling monclient init 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: init 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient: _add_conns ranks=[2,5,4,1,0,3,7,8,6] 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 0x7f65d412e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.987+0000 7f65da103640 10 monclient(hunting): picked mon.noname-b con 0x7f65d40a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d4138cd0 0x7f65d412ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): picked mon.noname-h con 0x7f65d4138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f65d41395f0 0x7f65d412f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): picked mon.noname-c con 0x7f65d41395f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f65d4136500 con 0x7f65d40a5b80 2024-04-19T00:51:44.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f65d40a20a0 con 0x7f65d41395f0 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f65d4136680 con 0x7f65d4138cd0 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): authenticate will time out at 1374.120069s 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 0x7f65d412e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f65d41395f0 0x7f65d412f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 0x7f65d412e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:53824/0 (socket says 172.21.15.43:53824) 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 -- 172.21.15.43:0/3421927083 learned_addr learned my addr 172.21.15.43:0/3421927083 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): get_auth_request con 0x7f65d40a5b80 auth_method 0 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): get_auth_request con 0x7f65d41395f0 auth_method 0 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d4138cd0 0x7f65d412ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): get_auth_request con 0x7f65d4138cd0 auth_method 0 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient(hunting): handle_auth_done global_id 23384 payload 824 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d4138cd0 msgr2=0x7f65d412ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f65d4138cd0 0x7f65d412ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f65d41395f0 msgr2=0x7f65d412f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f65d41395f0 0x7f65d412f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 monclient: found mon.noname-b 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d409e480 con 0x7f65d40a5b80 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f65d41395f0 0x7f65d412f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d37fe640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 0x7f65d412e840 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f65cc01d090 tx=0x7f65cc0195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 client.?.objecter ms_handle_connect 0x7f65d40a5b80 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65cc0b2050 con 0x7f65d40a5b80 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: dump: 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.009 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65cc0be070 con 0x7f65d40a5b80 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65cc003cf0 con 0x7f65d40a5b80 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 5 monclient: authenticate success, global_id 23384 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient: _renew_subs 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d40a3590 con 0x7f65d40a5b80 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d1ffb640 4 set_mon_vals no callback set 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:44.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: dump: 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.011 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 client.23384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient: _renew_subs 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d40a02b0 con 0x7f65d40a5b80 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 librados: init done 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 0x7f65d409c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f65d409fa10 con 0x7f65d40a0a90 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f65cc004040 con 0x7f65d40a5b80 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65c0038e10 0x7f65c003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 0x7f65d409c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7f65cc0e45d0 con 0x7f65d40a5b80 2024-04-19T00:51:44.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 client.23384.objecter ms_dispatch 0x7f65d4000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 3 client.23384.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 3 client.23384.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient: get_auth_request con 0x7f65d40a0a90 auth_method 0 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 20 client.23384.objecter dump_active .. 0 homeless 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65c0038e10 0x7f65c003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 10 monclient: get_auth_request con 0x7f65c0038e10 auth_method 0 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_done global_id 19083 payload 824 2024-04-19T00:51:44.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65d40693b0 con 0x7f65d40a0a90 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d2ffd640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 0x7f65d409c470 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f65d409db50 tx=0x7f65c800e960 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 client.23384.objecter ms_handle_connect 0x7f65d40a0a90 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 client.23384.objecter resend_mon_ops 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 client.23384.objecter ms_handle_connect 0x7f65c0038e10 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65d3fff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65c0038e10 0x7f65c003b2d0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7f65c400ada0 tx=0x7f65c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65c80070d0 con 0x7f65d40a0a90 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.991+0000 7f65bbfff640 10 monclient: dump: 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.014 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f65c8018070 con 0x7f65d40a0a90 2024-04-19T00:51:44.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:44.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 msgr2=0x7f65d409c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 0x7f65d409c470 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f65d409db50 tx=0x7f65c800e960 comp rx=0 tx=0).stop 2024-04-19T00:51:44.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 msgr2=0x7f65d409c470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.995+0000 7f65bbfff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d40a0a90 0x7f65d409c470 secure :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f65d409db50 tx=0x7f65c800e960 comp rx=0 tx=0).stop 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65da103640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65da103640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 0x7f65d409cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65da103640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f65d406d140 con 0x7f65d409c9b0 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 0x7f65d409cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient: get_auth_request con 0x7f65d409c9b0 auth_method 0 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 10 monclient(hunting): handle_auth_done global_id 19092 payload 824 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 1 -- 172.21.15.43:0/3421927083 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f65d409d300 con 0x7f65d409c9b0 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65d2ffd640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 0x7f65d409cdc0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f65c8014000 tx=0x7f65c8000ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65bbfff640 10 client.23384.objecter ms_handle_connect 0x7f65d409c9b0 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:43.999+0000 7f65bbfff640 10 client.23384.objecter resend_mon_ops 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f65c8007540 con 0x7f65d409c9b0 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 10 monclient: dump: 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.019 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f65c80070d0 con 0x7f65d409c9b0 2024-04-19T00:51:44.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 msgr2=0x7f65d409cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 0x7f65d409cdc0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f65c8014000 tx=0x7f65c8000ee0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 msgr2=0x7f65d409cdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65bbfff640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f65d409c9b0 0x7f65d409cdc0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 447484, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:51:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:44.025 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.026 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.027 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:44.028 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.032 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.034 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.037 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.038 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:44.039 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.040 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.041 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.042 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.043 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:44.044 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 10 librados: watch_flush enter 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 10 librados: watch_flush exit 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 20 client.23384.objecter shutdown clearing up homeless session... 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 10 client.23384.objecter successfully canceled tick 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65c0038e10 msgr2=0x7f65c003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f65c0038e10 0x7f65c003b2d0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7f65c400ada0 tx=0x7f65c4008040 comp rx=0 tx=0).stop 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 10 monclient: shutdown 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 msgr2=0x7f65d412e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 0x7f65d412e840 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f65cc01d090 tx=0x7f65cc0195f0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65d3fff640 1 -- 172.21.15.43:0/3421927083 reap_dead start 2024-04-19T00:51:44.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 shutdown_connections 2024-04-19T00:51:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 --2- 172.21.15.43:0/3421927083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f65d40a5b80 0x7f65d412e840 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 >> 172.21.15.43:0/3421927083 conn(0x7f65d40ab2f0 msgr2=0x7f65d4131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 shutdown_connections 2024-04-19T00:51:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 -- 172.21.15.43:0/3421927083 wait complete. 2024-04-19T00:51:44.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.003+0000 7f65da103640 1 librados: shutdown 2024-04-19T00:51:44.047 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:44.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:51:44.047 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:51:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient: get_monmap_and_config 2024-04-19T00:51:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient: monmap: 2024-04-19T00:51:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:44.175544+0000 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:44.175544+0000 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:44.175 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 Processor -- start 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 -- start start 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient: init 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient: _add_conns ranks=[5,7,0,10,1,6,2,9,4,3,8] 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9334138cd0 0x7f93341390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): picked mon.noname-h con 0x7f9334138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93340a5b80 0x7f93341395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): picked mon.noname-i con 0x7f93340a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f93341358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93340a3dc0 con 0x7f93341358b0 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f93340a4410 con 0x7f9334138cd0 2024-04-19T00:51:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9334139b30 con 0x7f93340a5b80 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 10 monclient(hunting): authenticate will time out at 1374.302836s 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93337fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f93341358b0 0x7f9334135d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:51386/0 (socket says 172.21.15.43:51386) 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93337fe640 1 -- 172.21.15.43:0/3413536625 learned_addr learned my addr 172.21.15.43:0/3413536625 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9334138cd0 0x7f93341390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161463655 0 0) 0x7f93340a3dc0 con 0x7f93341358b0 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): my global_id is 64323 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93340a5b80 0x7f93341395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931c003530 con 0x7f93341358b0 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): get_auth_request con 0x7f9334138cd0 auth_method 0 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): get_auth_request con 0x7f93340a5b80 auth_method 0 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2163265287 0 0) 0x7f931c003530 con 0x7f93341358b0 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93340a5b80 msgr2=0x7f93341395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f93340a5b80 0x7f93341395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9334138cd0 msgr2=0x7f93341390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9334138cd0 0x7f93341390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93340a40a0 con 0x7f93341358b0 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f932c001970 con 0x7f93341358b0 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 5 monclient: authenticate success, global_id 64323 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: dump: 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:51:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 >> v1:172.21.15.43:6789/0 conn(0x7f93341358b0 legacy=0x7f9334135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient: _add_conns ranks=[6,2,4,8,7,1,5,0,3] 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f931c004210 0x7f931c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): picked mon.d con 0x7f931c004210 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f931c004da0 0x7f931c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): picked mon.b con 0x7f931c004da0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f931c009ab0 0x7f931c00bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): picked mon.c con 0x7f931c009ab0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f93340a4410 con 0x7f931c004da0 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9334139b30 con 0x7f931c009ab0 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f931c00c850 con 0x7f931c004210 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9331ffb640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9339644640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f931c004210 0x7f931c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f931c004da0 0x7f931c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93337fe640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f931c009ab0 0x7f931c00bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): get_auth_request con 0x7f931c004210 auth_method 0 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f9332ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:44.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): get_auth_request con 0x7f931c004da0 auth_method 0 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.171+0000 7f93337fe640 10 monclient(hunting): get_auth_request con 0x7f931c009ab0 auth_method 0 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:51:44.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient(hunting): handle_auth_done global_id 64323 payload 293 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 -- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f931c009ab0 msgr2=0x7f931c00bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f931c009ab0 0x7f931c00bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 -- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f931c004da0 msgr2=0x7f931c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f931c004da0 0x7f931c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 monclient: found mon.d 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 -- 172.21.15.43:0/3413536625 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931c003530 con 0x7f931c004210 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93327fc640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f931c004da0 0x7f931c005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:51:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f93337fe640 1 -- 172.21.15.43:0/3413536625 reap_dead start 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9332ffd640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f931c004210 0x7f931c004810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9324009ab0 tx=0x7f932401c9d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=db5efe428594f83b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93240096b0 con 0x7f931c004210 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: dump: 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93240a6930 con 0x7f931c004210 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 1 -- 172.21.15.43:0/3413536625 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93240a6ae0 con 0x7f931c004210 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 4 set_mon_vals no callback set 2024-04-19T00:51:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9331ffb640 10 monclient: dump: 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 10 monclient: shutdown 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 -- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f931c004210 msgr2=0x7f931c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f931c004210 0x7f931c004810 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9324009ab0 tx=0x7f932401c9d0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 -- 172.21.15.43:0/3413536625 shutdown_connections 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 --2- 172.21.15.43:0/3413536625 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f931c004210 0x7f931c004810 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 -- 172.21.15.43:0/3413536625 >> 172.21.15.43:0/3413536625 conn(0x7f93340ab2f0 msgr2=0x7f93340ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 -- 172.21.15.43:0/3413536625 shutdown_connections 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 1 -- 172.21.15.43:0/3413536625 wait complete. 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.175+0000 7f9339644640 10 monclient: monmap: 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.201 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 librados: starting msgr at 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 librados: starting objecter 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 Processor -- start 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- start start 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 librados: setting wanted keys 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 librados: calling monclient init 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: init 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: _add_conns ranks=[5,6,2,7,4,3,0,8,1] 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f931c004210 0x7f9334134cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): picked mon.noname-h con 0x7f931c004210 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 0x7f93341d0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): picked mon.noname-d con 0x7f9334135210 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93341d1360 0x7f93341d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): picked mon.noname-b con 0x7f93341d1360 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9334136270 con 0x7f93341d1360 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f93341360f0 con 0x7f931c004210 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f93340a1e90 con 0x7f9334135210 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): authenticate will time out at 1374.308744s 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 0x7f93341d0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 0x7f93341d0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59064/0 (socket says 172.21.15.43:59064) 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 -- 172.21.15.43:0/146961024 learned_addr learned my addr 172.21.15.43:0/146961024 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): get_auth_request con 0x7f9334135210 auth_method 0 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93341d1360 0x7f93341d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): get_auth_request con 0x7f93341d1360 auth_method 0 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f931c004210 0x7f9334134cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): get_auth_request con 0x7f931c004210 auth_method 0 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient(hunting): handle_auth_done global_id 64212 payload 824 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f931c004210 msgr2=0x7f9334134cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f931c004210 0x7f9334134cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93341d1360 msgr2=0x7f93341d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f93341d1360 0x7f93341d17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 monclient: found mon.noname-d 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93340a3590 con 0x7f9334135210 2024-04-19T00:51:44.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93327fc640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 0x7f93341d0e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9328007e70 tx=0x7f93280115a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 client.?.objecter ms_handle_connect 0x7f9334135210 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f932801a020 con 0x7f9334135210 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: dump: 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.206 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93280168e0 con 0x7f9334135210 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f93280153d0 con 0x7f9334135210 2024-04-19T00:51:44.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 monclient: dump: 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.208 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93317fa640 4 set_mon_vals no callback set 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 5 monclient: authenticate success, global_id 64212 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: _renew_subs 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93340a3040 con 0x7f9334135210 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 client.64212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: _renew_subs 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f93341223b0 con 0x7f9334135210 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 librados: init done 2024-04-19T00:51:44.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 0x7f9334136ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f933406d140 con 0x7f93341368c0 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f932801d020 con 0x7f9334135210 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9320038e40 0x7f932003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7f9328012040 con 0x7f9334135210 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 client.64212.objecter ms_dispatch 0x7f9334000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 3 client.64212.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 3 client.64212.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 20 client.64212.objecter dump_active .. 0 homeless 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 0x7f9334136ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient: get_auth_request con 0x7f93341368c0 auth_method 0 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9320038e40 0x7f932003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f93337fe640 10 monclient: get_auth_request con 0x7f9320038e40 auth_method 0 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 10 monclient(hunting): handle_auth_done global_id 44996 payload 824 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93340693b0 con 0x7f93341368c0 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 client.64212.objecter ms_handle_connect 0x7f93341368c0 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f9332ffd640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 0x7f9334136ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f931c004790 tx=0x7f931c005840 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.179+0000 7f931b7fe640 10 client.64212.objecter resend_mon_ops 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 10 client.64212.objecter ms_handle_connect 0x7f9320038e40 2024-04-19T00:51:44.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f93337fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9320038e40 0x7f932003b300 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7f93340a4e40 tx=0x7f932c001750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9324009770 con 0x7f93341368c0 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 10 monclient: dump: 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.212 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9324009770 con 0x7f93341368c0 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 msgr2=0x7f9334136ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 0x7f9334136ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f931c004790 tx=0x7f931c005840 comp rx=0 tx=0).stop 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 msgr2=0x7f9334136ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.183+0000 7f931b7fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341368c0 0x7f9334136ca0 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f931c004790 tx=0x7f931c005840 comp rx=0 tx=0).stop 2024-04-19T00:51:44.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9339644640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:44.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9339644640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 0x7f93341dc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9339644640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f9334068240 con 0x7f93341371e0 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9332ffd640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 0x7f93341dc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9332ffd640 10 monclient: get_auth_request con 0x7f93341371e0 auth_method 0 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9332ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9332ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.187+0000 7f9332ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9332ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9332ffd640 10 monclient(hunting): handle_auth_done global_id 45005 payload 824 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9332ffd640 1 -- 172.21.15.43:0/146961024 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f93340a9b60 con 0x7f93341371e0 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9332ffd640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 0x7f93341dc1d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324009ab0 tx=0x7f93240a7000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 client.64212.objecter ms_handle_connect 0x7f93341371e0 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 client.64212.objecter resend_mon_ops 2024-04-19T00:51:44.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9324003ca0 con 0x7f93341371e0 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 monclient: dump: 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.217 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f9324003830 con 0x7f93341371e0 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 msgr2=0x7f93341dc1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 0x7f93341dc1d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324009ab0 tx=0x7f93240a7000 comp rx=0 tx=0).stop 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 msgr2=0x7f93341dc1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f931b7fe640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f93341371e0 0x7f93341dc1d0 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9324009ab0 tx=0x7f93240a7000 comp rx=0 tx=0).stop 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 303922, 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:44.236 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.238 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.240 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.241 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.242 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:44.243 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:44.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 10 librados: watch_flush enter 2024-04-19T00:51:44.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 10 librados: watch_flush exit 2024-04-19T00:51:44.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 20 client.64212.objecter shutdown clearing up homeless session... 2024-04-19T00:51:44.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 10 client.64212.objecter successfully canceled tick 2024-04-19T00:51:44.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9320038e40 msgr2=0x7f932003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9320038e40 0x7f932003b300 secure :-1 s=READY pgs=3472 cs=0 l=1 rev1=1 crypto rx=0x7f93340a4e40 tx=0x7f932c001750 comp rx=0 tx=0).stop 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 10 monclient: shutdown 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 msgr2=0x7f93341d0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f93337fe640 1 -- 172.21.15.43:0/146961024 reap_dead start 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 0x7f93341d0e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9328007e70 tx=0x7f93280115a0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 shutdown_connections 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 --2- 172.21.15.43:0/146961024 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f9334135210 0x7f93341d0e20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 >> 172.21.15.43:0/146961024 conn(0x7f93340ab2f0 msgr2=0x7f93340a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 shutdown_connections 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 -- 172.21.15.43:0/146961024 wait complete. 2024-04-19T00:51:44.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.191+0000 7f9339644640 1 librados: shutdown 2024-04-19T00:51:44.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:44.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:51:44.246 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.359+0000 7f446b187640 10 monclient: get_monmap_and_config 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.359+0000 7f446b187640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.359+0000 7f446b187640 10 monclient: monmap: 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:44.366252+0000 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:44.366252+0000 2024-04-19T00:51:44.364 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:44.365 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 Processor -- start 2024-04-19T00:51:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 -- start start 2024-04-19T00:51:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient: init 2024-04-19T00:51:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient: _add_conns ranks=[3,1,6,2,8,9,10,5,4,0,7] 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4464138a50 0x7f4464138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): picked mon.noname-g con 0x7f4464138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f44640a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f4464138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): picked mon.noname-d con 0x7f4464139370 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44640a0d40 con 0x7f44640a5bb0 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f44640a1020 con 0x7f4464138a50 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f44640a0780 con 0x7f4464139370 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f446b187640 10 monclient(hunting): authenticate will time out at 1374.493484s 2024-04-19T00:51:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f4464138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f4464138070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59074/0 (socket says 172.21.15.43:59074) 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 -- 172.21.15.43:0/3859333087 learned_addr learned my addr 172.21.15.43:0/3859333087 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): get_auth_request con 0x7f4464139370 auth_method 0 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4468f23640 1 --2- 172.21.15.43:0/3859333087 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4464138a50 0x7f4464138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4468f23640 10 monclient(hunting): get_auth_request con 0x7f4464138a50 auth_method 0 2024-04-19T00:51:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4468f23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4468f23640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4468f23640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient(hunting): handle_auth_done global_id 64221 payload 274 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 -- 172.21.15.43:0/3859333087 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4464138a50 msgr2=0x7f4464138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 --2- 172.21.15.43:0/3859333087 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4464138a50 0x7f4464138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 -- 172.21.15.43:0/3859333087 >> v1:172.21.15.161:6789/0 conn(0x7f44640a5bb0 legacy=0x7f4464137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 monclient: found mon.noname-d 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.363+0000 7f4469724640 1 -- 172.21.15.43:0/3859333087 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44640a0a60 con 0x7f4464139370 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 1 --2- 172.21.15.43:0/3859333087 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f4464138070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f445c009e80 tx=0x7f445c01c8b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=a608f2ad92c7a627 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 1 -- 172.21.15.43:0/3859333087 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f445c0090d0 con 0x7f4464139370 2024-04-19T00:51:44.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: dump: 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 1 -- 172.21.15.43:0/3859333087 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f445c0a5ac0 con 0x7f4464139370 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 1 -- 172.21.15.43:0/3859333087 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f445c0a5cb0 con 0x7f4464139370 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 5 monclient: authenticate success, global_id 64221 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 4 set_mon_vals no callback set 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445b7fe640 10 monclient: dump: 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.375 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.375 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.375 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: shutdown 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- 172.21.15.43:0/3859333087 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 msgr2=0x7f4464138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 --2- 172.21.15.43:0/3859333087 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f4464138070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f445c009e80 tx=0x7f445c01c8b0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- 172.21.15.43:0/3859333087 shutdown_connections 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 --2- 172.21.15.43:0/3859333087 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f4464138070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 --2- 172.21.15.43:0/3859333087 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4464138a50 0x7f4464138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- 172.21.15.43:0/3859333087 >> 172.21.15.43:0/3859333087 conn(0x7f44640a7cb0 msgr2=0x7f44640a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- 172.21.15.43:0/3859333087 shutdown_connections 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- 172.21.15.43:0/3859333087 wait complete. 2024-04-19T00:51:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: monmap: 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 librados: starting msgr at 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 librados: starting objecter 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 Processor -- start 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- start start 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 librados: setting wanted keys 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 librados: calling monclient init 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: init 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient: _add_conns ranks=[5,7,6,4,3,1,8,2,0] 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f44640a5bb0 0x7f446409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): picked mon.noname-h con 0x7f44640a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464138a50 0x7f446409bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): picked mon.noname-i con 0x7f4464138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f446409c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): picked mon.noname-d con 0x7f4464139370 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4464138730 con 0x7f44640a5bb0 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f44641385b0 con 0x7f4464139370 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f4464143820 con 0x7f4464138a50 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f446b187640 10 monclient(hunting): authenticate will time out at 1374.499531s 2024-04-19T00:51:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f446409c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f446409c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59076/0 (socket says 172.21.15.43:59076) 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 1 -- 172.21.15.43:0/2944951589 learned_addr learned my addr 172.21.15.43:0/2944951589 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): get_auth_request con 0x7f4464139370 auth_method 0 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4469724640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4468f23640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f44640a5bb0 0x7f446409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4468f23640 10 monclient(hunting): get_auth_request con 0x7f44640a5bb0 auth_method 0 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4468f23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4468f23640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f4468f23640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445bfff640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464138a50 0x7f446409bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445bfff640 10 monclient(hunting): get_auth_request con 0x7f4464138a50 auth_method 0 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.367+0000 7f445bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 10 monclient(hunting): handle_auth_done global_id 64230 payload 824 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464138a50 msgr2=0x7f446409bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464138a50 0x7f446409bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f44640a5bb0 msgr2=0x7f446409ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f44640a5bb0 0x7f446409ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 monclient: found mon.noname-d 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446409ee60 con 0x7f4464139370 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4469724640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f446409c4f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f445c009040 tx=0x7f445c0196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 client.?.objecter ms_handle_connect 0x7f4464139370 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f445c0bc020 con 0x7f4464139370 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: dump: 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f445c009250 con 0x7f4464139370 2024-04-19T00:51:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f445c003bb0 con 0x7f4464139370 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: dump: 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f445affd640 4 set_mon_vals no callback set 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 5 monclient: authenticate success, global_id 64230 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 monclient: _renew_subs 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446409d6d0 con 0x7f4464139370 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 client.64230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 monclient: _renew_subs 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f446409e230 con 0x7f4464139370 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 1 librados: init done 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 0x7f44641d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f446406d140 con 0x7f4464135ce0 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f445c0bf020 con 0x7f4464139370 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4450038e10 0x7f445003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 0x7f44641d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7f445c0b3040 con 0x7f4464139370 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 client.64230.objecter ms_dispatch 0x7f4464000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 3 client.64230.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 3 client.64230.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient: get_auth_request con 0x7f4464135ce0 auth_method 0 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 20 client.64230.objecter dump_active .. 0 homeless 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f445bfff640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4450038e10 0x7f445003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f445bfff640 10 monclient: get_auth_request con 0x7f4450038e10 auth_method 0 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 10 monclient(hunting): handle_auth_done global_id 19663 payload 824 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4464134ed0 con 0x7f4464135ce0 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 client.64230.objecter ms_handle_connect 0x7f4464135ce0 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 client.64230.objecter resend_mon_ops 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f4468f23640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 0x7f44641d09b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f446000f4b0 tx=0x7f446000f970 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 client.64230.objecter ms_handle_connect 0x7f4450038e10 2024-04-19T00:51:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f445bfff640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4450038e10 0x7f445003b2d0 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7f44640aa4d0 tx=0x7f4454009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4460019020 con 0x7f4464135ce0 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: dump: 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4460014670 con 0x7f4464135ce0 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 msgr2=0x7f44641d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 0x7f44641d09b0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f446000f4b0 tx=0x7f446000f970 comp rx=0 tx=0).stop 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 msgr2=0x7f44641d09b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.371+0000 7f44597fa640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4464135ce0 0x7f44641d09b0 secure :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f446000f4b0 tx=0x7f446000f970 comp rx=0 tx=0).stop 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f446b187640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f446b187640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 0x7f44641d1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f446b187640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f44641d1840 con 0x7f44641d0ef0 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 0x7f44641d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient: get_auth_request con 0x7f44641d0ef0 auth_method 0 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 10 monclient(hunting): handle_auth_done global_id 19672 payload 824 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 1 -- 172.21.15.43:0/2944951589 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4464076190 con 0x7f44641d0ef0 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f4468f23640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 0x7f44641d1300 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f446000f4b0 tx=0x7f44600046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f44597fa640 10 client.64230.objecter ms_handle_connect 0x7f44641d0ef0 2024-04-19T00:51:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f44597fa640 10 client.64230.objecter resend_mon_ops 2024-04-19T00:51:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4460019040 con 0x7f44641d0ef0 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f44597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f44597fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.379+0000 7f44597fa640 10 monclient: dump: 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f4460004050 con 0x7f44641d0ef0 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 msgr2=0x7f44641d1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 0x7f44641d1300 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f446000f4b0 tx=0x7f44600046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 msgr2=0x7f44641d1300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f44597fa640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f44641d0ef0 0x7f44641d1300 secure :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f446000f4b0 tx=0x7f44600046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 50, 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 447840, 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:51:44.401 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.402 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.403 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:51:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:51:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:51:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:51:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:51:44.423 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 10 librados: watch_flush enter 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 10 librados: watch_flush exit 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 20 client.64230.objecter shutdown clearing up homeless session... 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 10 client.64230.objecter successfully canceled tick 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4450038e10 msgr2=0x7f445003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4450038e10 0x7f445003b2d0 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7f44640aa4d0 tx=0x7f4454009040 comp rx=0 tx=0).stop 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 10 monclient: shutdown 2024-04-19T00:51:44.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 msgr2=0x7f446409c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 --2- 172.21.15.43:0/2944951589 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4464139370 0x7f446409c4f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f445c009040 tx=0x7f445c0196b0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f4469724640 1 -- 172.21.15.43:0/2944951589 reap_dead start 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f4469724640 1 -- 172.21.15.43:0/2944951589 reap_dead start 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 shutdown_connections 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 >> 172.21.15.43:0/2944951589 conn(0x7f44640a7cb0 msgr2=0x7f4464134000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 shutdown_connections 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 -- 172.21.15.43:0/2944951589 wait complete. 2024-04-19T00:51:44.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.383+0000 7f446b187640 1 librados: shutdown 2024-04-19T00:51:44.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:51:44.427 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b', 'd', 'h', 'i'] 2024-04-19T00:51:44.427 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-04-19T00:51:44.427 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-19T00:51:44.427 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b store 2024-04-19T00:51:44.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b sync_force --yes-i-really-mean-it 2024-04-19T00:51:44.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.543+0000 7fca88c81640 10 monclient: get_monmap_and_config 2024-04-19T00:51:44.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.543+0000 7fca88c81640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.543+0000 7fca88c81640 10 monclient: monmap: 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:44.548832+0000 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:44.548832+0000 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.548 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.548 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.548 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:44.548 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:44.548 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.543+0000 7fca88c81640 1 Processor -- start 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 -- start start 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient: init 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient: _add_conns ranks=[3,6,8,5,4,10,9,7,2,1,0] 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840a2180 0x7fca84135f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): picked mon.noname-g con 0x7fca840a2180 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fca84073410 0x7fca841364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): picked mon.noname-d con 0x7fca84073410 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 0x7fca840a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): picked mon.noname-e con 0x7fca8409ff10 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fca840a2560 con 0x7fca840a2180 2024-04-19T00:51:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fca8409fcd0 con 0x7fca84073410 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fca84135b30 con 0x7fca8409ff10 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient(hunting): authenticate will time out at 1374.676885s 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 0x7fca840a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 0x7fca840a3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52996/0 (socket says 172.21.15.43:52996) 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 1 -- 172.21.15.43:0/2934525227 learned_addr learned my addr 172.21.15.43:0/2934525227 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): get_auth_request con 0x7fca8409ff10 auth_method 0 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fca84073410 0x7fca841364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): get_auth_request con 0x7fca84073410 auth_method 0 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca8259c640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840a2180 0x7fca84135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca8259c640 10 monclient(hunting): get_auth_request con 0x7fca840a2180 auth_method 0 2024-04-19T00:51:44.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca8259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca8259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient(hunting): handle_auth_done global_id 64239 payload 274 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 -- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 msgr2=0x7fca840a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 0x7fca840a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 -- 172.21.15.43:0/2934525227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840a2180 msgr2=0x7fca84135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840a2180 0x7fca84135f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 monclient: found mon.noname-d 2024-04-19T00:51:44.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:51:44.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 -- 172.21.15.43:0/2934525227 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca840a13c0 con 0x7fca84073410 2024-04-19T00:51:44.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca82d9d640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 0x7fca840a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca7bfff640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fca84073410 0x7fca841364c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009820 tx=0x7fca6c01c150 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7064f914a8a738f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 1 -- 172.21.15.43:0/2934525227 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca6c0b0070 con 0x7fca84073410 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: dump: 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 1 -- 172.21.15.43:0/2934525227 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca6c0aa810 con 0x7fca84073410 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 5 monclient: authenticate success, global_id 64239 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 1 -- 172.21.15.43:0/2934525227 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca6c0a9300 con 0x7fca84073410 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca88c81640 4 set_mon_vals no callback set 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.547+0000 7fca81d9b640 10 monclient: dump: 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: shutdown 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fca84073410 msgr2=0x7fca841364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fca84073410 0x7fca841364c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fca6c009820 tx=0x7fca6c01c150 comp rx=0 tx=0).stop 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/2934525227 shutdown_connections 2024-04-19T00:51:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fca8409ff10 0x7fca840a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fca84073410 0x7fca841364c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- 172.21.15.43:0/2934525227 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fca840a2180 0x7fca84135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/2934525227 >> 172.21.15.43:0/2934525227 conn(0x7fca840a8870 msgr2=0x7fca840a4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/2934525227 shutdown_connections 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/2934525227 wait complete. 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: monmap: 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 librados: starting msgr at 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 librados: starting objecter 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 Processor -- start 2024-04-19T00:51:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- start start 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 librados: setting wanted keys 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 librados: calling monclient init 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: init 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: _add_conns ranks=[5,1,0,8,3,2,4,6,7] 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fca84073410 0x7fca841cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): picked mon.noname-h con 0x7fca84073410 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8409ff10 0x7fca84131800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): picked mon.noname-f con 0x7fca8409ff10 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 0x7fca84138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): picked mon.noname-a con 0x7fca840a2180 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fca8413f890 con 0x7fca840a2180 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fca8413f710 con 0x7fca8409ff10 2024-04-19T00:51:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fca8413fa10 con 0x7fca84073410 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient(hunting): authenticate will time out at 1374.682604s 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 0x7fca84138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 0x7fca84138790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40016/0 (socket says 172.21.15.43:40016) 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 -- 172.21.15.43:0/1174894459 learned_addr learned my addr 172.21.15.43:0/1174894459 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): get_auth_request con 0x7fca840a2180 auth_method 0 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca8259c640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fca84073410 0x7fca841cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7bfff640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8409ff10 0x7fca84131800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7bfff640 10 monclient(hunting): get_auth_request con 0x7fca8409ff10 auth_method 0 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca8259c640 10 monclient(hunting): get_auth_request con 0x7fca84073410 auth_method 0 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca8259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca8259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient(hunting): handle_auth_done global_id 64332 payload 824 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fca84073410 msgr2=0x7fca841cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fca84073410 0x7fca841cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8409ff10 msgr2=0x7fca84131800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca8409ff10 0x7fca84131800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 monclient: found mon.noname-a 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8413c700 con 0x7fca840a2180 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca82d9d640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 0x7fca84138790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00efc0 tx=0x7fca7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 client.?.objecter ms_handle_connect 0x7fca840a2180 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca7c00cbe0 con 0x7fca840a2180 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: dump: 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca7c00cd80 con 0x7fca840a2180 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 5 monclient: authenticate success, global_id 64332 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: _renew_subs 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca84073860 con 0x7fca840a2180 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca7c014d60 con 0x7fca840a2180 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca8159a640 4 set_mon_vals no callback set 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 client.64332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca7affd640 10 monclient: dump: 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: _renew_subs 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8413c420 con 0x7fca840a2180 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.551+0000 7fca88c81640 1 librados: init done 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca88c81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca88c81640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 0x7fca8402c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fca84074f70 con 0x7fca84131d40 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fca7c01b070 con 0x7fca840a2180 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 0x7fca8402c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient: get_auth_request con 0x7fca84131d40 auth_method 0 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fca64038e70 0x7fca6403b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7fca7c019020 con 0x7fca840a2180 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 client.64332.objecter ms_dispatch 0x7fca84000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 3 client.64332.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 3 client.64332.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 10 monclient(hunting): handle_auth_done global_id 23402 payload 824 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca840693b0 con 0x7fca84131d40 2024-04-19T00:51:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 20 client.64332.objecter dump_active .. 0 homeless 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7bfff640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fca64038e70 0x7fca6403b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 client.64332.objecter ms_handle_connect 0x7fca84131d40 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 client.64332.objecter resend_mon_ops 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca8259c640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 0x7fca8402c450 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fca7400f440 tx=0x7fca7400f900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7bfff640 10 monclient: get_auth_request con 0x7fca64038e70 auth_method 0 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca74019030 con 0x7fca84131d40 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 monclient: dump: 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca74014670 con 0x7fca84131d40 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 msgr2=0x7fca8402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 0x7fca8402c450 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fca7400f440 tx=0x7fca7400f900 comp rx=0 tx=0).stop 2024-04-19T00:51:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 msgr2=0x7fca8402c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca84131d40 0x7fca8402c450 secure :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7fca7400f440 tx=0x7fca7400f900 comp rx=0 tx=0).stop 2024-04-19T00:51:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7affd640 10 client.64332.objecter ms_handle_connect 0x7fca64038e70 2024-04-19T00:51:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.555+0000 7fca7bfff640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fca64038e70 0x7fca6403b330 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0ad000 tx=0x7fca6c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 0x7fca8402cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fca840a08f0 con 0x7fca8402c990 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 0x7fca8402cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient: get_auth_request con 0x7fca8402c990 auth_method 0 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 10 monclient(hunting): handle_auth_done global_id 23411 payload 824 2024-04-19T00:51:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 1 -- 172.21.15.43:0/1174894459 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fca84071840 con 0x7fca8402c990 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 client.64332.objecter ms_handle_connect 0x7fca8402c990 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 client.64332.objecter resend_mon_ops 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca8259c640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 0x7fca8402cd70 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fca7400f440 tx=0x7fca740046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fca7401f240 con 0x7fca8402c990 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 monclient: dump: 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.582 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fca7401f240 con 0x7fca8402c990 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:44.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 msgr2=0x7fca8402cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 0x7fca8402cd70 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fca7400f440 tx=0x7fca740046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 msgr2=0x7fca8402cd70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca7affd640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fca8402c990 0x7fca8402cd70 secure :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fca7400f440 tx=0x7fca740046d0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:51:44.584 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 10 librados: watch_flush enter 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 10 librados: watch_flush exit 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 20 client.64332.objecter shutdown clearing up homeless session... 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 10 client.64332.objecter successfully canceled tick 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fca64038e70 msgr2=0x7fca6403b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fca64038e70 0x7fca6403b330 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0ad000 tx=0x7fca6c0a6000 comp rx=0 tx=0).stop 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 10 monclient: shutdown 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca82d9d640 1 -- 172.21.15.43:0/1174894459 reap_dead start 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 msgr2=0x7fca84138790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 0x7fca84138790 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00efc0 tx=0x7fca7c00c490 comp rx=0 tx=0).stop 2024-04-19T00:51:44.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 shutdown_connections 2024-04-19T00:51:44.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 --2- 172.21.15.43:0/1174894459 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fca840a2180 0x7fca84138790 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 >> 172.21.15.43:0/1174894459 conn(0x7fca840a8870 msgr2=0x7fca840a6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 shutdown_connections 2024-04-19T00:51:44.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 -- 172.21.15.43:0/1174894459 wait complete. 2024-04-19T00:51:44.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.563+0000 7fca88c81640 1 librados: shutdown 2024-04-19T00:51:44.587 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-19T00:51:44.587 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T00:51:44.587 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:51:44.603 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T00:51:44.603 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-04-19T00:51:44.603 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T00:51:44.603 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-04-19T00:51:44.604 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:51:44.655 INFO:tasks.ceph.mon.d:Stopped 2024-04-19T00:51:44.656 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.h 2024-04-19T00:51:44.656 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-04-19T00:51:44.656 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-04-19T00:51:44.656 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:51:44.688 INFO:tasks.ceph.mon.h:Stopped 2024-04-19T00:51:44.688 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-04-19T00:51:44.688 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i store 2024-04-19T00:51:44.688 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i sync_force --yes-i-really-mean-it 2024-04-19T00:51:44.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient: get_monmap_and_config 2024-04-19T00:51:44.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient: monmap: 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:51:44.824859+0000 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:51:44.824859+0000 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.824 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.824 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:51:44.824 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:51:44.824 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 1 Processor -- start 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 1 -- start start 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient: init 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient: _add_conns ranks=[10,9,7,1,0,5,4,8,6,2,3] 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4154138d40 0x7f41540a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 10 monclient(hunting): picked mon.noname-f con 0x7f4154138d40 addr v2:172.21.15.161:3300/0 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.819+0000 7f415a655640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4154139120 0x7f4154139590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): picked mon.noname-a con 0x7f4154139120 addr v2:172.21.15.43:3300/0 2024-04-19T00:51:44.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41541358b0 0x7f4154135ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): picked mon.noname-i con 0x7f41541358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f41540a2720 con 0x7f4154139120 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f41540a3e30 con 0x7f4154138d40 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f41540a2b50 con 0x7f41541358b0 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient(hunting): authenticate will time out at 1374.952065s 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4154139120 0x7f4154139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4154139120 0x7f4154139590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40032/0 (socket says 172.21.15.43:40032) 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 -- 172.21.15.43:0/4051943773 learned_addr learned my addr 172.21.15.43:0/4051943773 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): get_auth_request con 0x7f4154139120 auth_method 0 2024-04-19T00:51:44.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4153fff640 1 --2- 172.21.15.43:0/4051943773 >> v2:172.21.15.161:3300/0 conn(0x7f4154138d40 0x7f41540a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4158bf2640 1 --2- 172.21.15.43:0/4051943773 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41541358b0 0x7f4154135ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4153fff640 10 monclient(hunting): get_auth_request con 0x7f4154138d40 auth_method 0 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4153fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4153fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4158bf2640 10 monclient(hunting): get_auth_request con 0x7f41541358b0 auth_method 0 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4158bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4158bf2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4158bf2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient(hunting): handle_auth_done global_id 64341 payload 274 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 -- 172.21.15.43:0/4051943773 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41541358b0 msgr2=0x7f4154135ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 --2- 172.21.15.43:0/4051943773 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41541358b0 0x7f4154135ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 -- 172.21.15.43:0/4051943773 >> v2:172.21.15.161:3300/0 conn(0x7f4154138d40 msgr2=0x7f41540a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 --2- 172.21.15.43:0/4051943773 >> v2:172.21.15.161:3300/0 conn(0x7f4154138d40 0x7f41540a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 monclient: found mon.noname-a 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 -- 172.21.15.43:0/4051943773 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4154139ad0 con 0x7f4154139120 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f41537fe640 1 --2- 172.21.15.43:0/4051943773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4154139120 0x7f4154139590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41540ac140 tx=0x7f413c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b03d83286e8a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 1 -- 172.21.15.43:0/4051943773 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f413c0b0070 con 0x7f4154139120 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: dump: 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.831 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 1 -- 172.21.15.43:0/4051943773 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413c0aa810 con 0x7f4154139120 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 5 monclient: authenticate success, global_id 64341 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient: get_monmap_and_config success 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 4 set_mon_vals no callback set 2024-04-19T00:51:44.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 1 -- 172.21.15.43:0/4051943773 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f413c0a9300 con 0x7f4154139120 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f4152ffd640 10 monclient: dump: 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.833 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.834 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.834 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.834 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.834 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 10 monclient: shutdown 2024-04-19T00:51:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- 172.21.15.43:0/4051943773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4154139120 msgr2=0x7f4154139590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 --2- 172.21.15.43:0/4051943773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4154139120 0x7f4154139590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f41540ac140 tx=0x7f413c01c3e0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- 172.21.15.43:0/4051943773 shutdown_connections 2024-04-19T00:51:44.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 --2- 172.21.15.43:0/4051943773 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41541358b0 0x7f4154135ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 --2- 172.21.15.43:0/4051943773 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4154139120 0x7f4154139590 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 --2- 172.21.15.43:0/4051943773 >> v2:172.21.15.161:3300/0 conn(0x7f4154138d40 0x7f41540a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- 172.21.15.43:0/4051943773 >> 172.21.15.43:0/4051943773 conn(0x7f41540ab2f0 msgr2=0x7f41540ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- 172.21.15.43:0/4051943773 shutdown_connections 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.823+0000 7f415a655640 1 -- 172.21.15.43:0/4051943773 wait complete. 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: build_initial_monmap 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: monmap: 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:51:44.840 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 librados: starting msgr at 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 librados: starting objecter 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 Processor -- start 2024-04-19T00:51:44.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 -- start start 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 librados: setting wanted keys 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 librados: calling monclient init 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: init 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: _reopen_session rank -1 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: _add_conns ranks=[5,6,3,0,4,7,8,2,1] 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41541358b0 0x7f41541d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): picked mon.noname-h con 0x7f41541358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4154138d40 0x7f415413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): picked mon.noname-d con 0x7f4154138d40 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4154139120 0x7f415413f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): picked mon.noname-g con 0x7f4154139120 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4154142c20 con 0x7f4154139120 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4154142aa0 con 0x7f41541358b0 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f4154142da0 con 0x7f4154138d40 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): _renew_subs 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient(hunting): authenticate will time out at 1374.957426s 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f41537fe640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4154138d40 msgr2=0x7f415413ed00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f41537fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4154138d40 0x7f415413ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4153fff640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41541358b0 msgr2=0x7f41541d0da0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4153fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41541358b0 0x7f41541d0da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4154139120 0x7f415413f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4154139120 0x7f415413f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:54858/0 (socket says 172.21.15.43:54858) 2024-04-19T00:51:44.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 -- 172.21.15.43:0/1149653378 learned_addr learned my addr 172.21.15.43:0/1149653378 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): get_auth_request con 0x7f4154139120 auth_method 0 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient(hunting): handle_auth_done global_id 19119 payload 824 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4154138d40 msgr2=0x7f415413ed00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4154138d40 0x7f415413ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41541358b0 msgr2=0x7f41541d0da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41541358b0 0x7f41541d0da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient: _finish_hunting 0 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 monclient: found mon.noname-g 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41540a3140 con 0x7f4154139120 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4158bf2640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4154139120 0x7f415413f240 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f414c00d800 tx=0x7f414c00dcc0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 client.?.objecter ms_handle_connect 0x7f4154139120 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f414c00bb30 con 0x7f4154139120 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: dump: 2024-04-19T00:51:44.845 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.846 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: _finish_auth 0 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c019070 con 0x7f4154139120 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f414c00bd10 con 0x7f4154139120 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 5 monclient: authenticate success, global_id 19119 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: _renew_subs 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415413af10 con 0x7f4154139120 2024-04-19T00:51:44.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4151ffb640 4 set_mon_vals no callback set 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f4150ff9640 10 monclient: dump: 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.848 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 client.19119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: _renew_subs 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f415413f780 con 0x7f4154139120 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.827+0000 7f415a655640 1 librados: init done 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f415a655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f415a655640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 0x7f4154133a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f415406d140 con 0x7f4154133650 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 0x7f4154133a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient: get_auth_request con 0x7f4154133650 auth_method 0 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f414c00b6c0 con 0x7f4154139120 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4134038e10 0x7f413403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(321..321 src has 1..321) ==== 6141+0+0 (secure 0 0 0) 0x7f414c00f040 con 0x7f4154139120 2024-04-19T00:51:44.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 client.19119.objecter ms_dispatch 0x7f4154000ba0 osd_map(321..321 src has 1..321) 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 3 client.19119.objecter handle_osd_map got epochs [321,321] > 0 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 3 client.19119.objecter handle_osd_map decoding full epoch 321 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 20 client.19119.objecter dump_active .. 0 homeless 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 10 monclient(hunting): handle_auth_done global_id 19690 payload 824 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41540693b0 con 0x7f4154133650 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f41537fe640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4134038e10 0x7f413403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f41537fe640 10 monclient: get_auth_request con 0x7f4134038e10 auth_method 0 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 client.19119.objecter ms_handle_connect 0x7f4154133650 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4153fff640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 0x7f4154133a30 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4144007d60 tx=0x7f414400e4b0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 client.19119.objecter resend_mon_ops 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4144019070 con 0x7f4154133650 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 monclient: dump: 2024-04-19T00:51:44.851 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.852 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 client.19119.objecter ms_handle_connect 0x7f4134038e10 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f41537fe640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4134038e10 0x7f413403b2d0 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7f413c0ad000 tx=0x7f413c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f414400ede0 con 0x7f4154133650 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 msgr2=0x7f4154133a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 0x7f4154133a30 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4144007d60 tx=0x7f414400e4b0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 msgr2=0x7f4154133a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.831+0000 7f4150ff9640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133650 0x7f4154133a30 secure :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4144007d60 tx=0x7f414400e4b0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f415a655640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f415a655640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 0x7f4154137a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f415a655640 10 monclient(hunting): start opening mon connection 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f4154067760 con 0x7f4154133f70 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 0x7f4154137a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient: get_auth_request con 0x7f4154133f70 auth_method 0 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:51:44.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 10 monclient(hunting): handle_auth_done global_id 19699 payload 824 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 1 -- 172.21.15.43:0/1149653378 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f415413b430 con 0x7f4154133f70 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4150ff9640 10 client.19119.objecter ms_handle_connect 0x7f4154133f70 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4153fff640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 0x7f4154137a00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f4144007d60 tx=0x7f4144004ac0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.839+0000 7f4150ff9640 10 client.19119.objecter resend_mon_ops 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4144019070 con 0x7f4154133f70 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 10 monclient: dump: 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:51:44.856 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:51:44.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f414401a040 con 0x7f4154133f70 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 msgr2=0x7f4154137a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 0x7f4154137a00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f4144007d60 tx=0x7f4144004ac0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 msgr2=0x7f4154137a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4150ff9640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4154133f70 0x7f4154137a00 secure :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f4144007d60 tx=0x7f4144004ac0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:51:44.858 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:51:44.859 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 10 librados: watch_flush enter 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 10 librados: watch_flush exit 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 20 client.19119.objecter shutdown clearing up homeless session... 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 10 client.19119.objecter successfully canceled tick 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4134038e10 msgr2=0x7f413403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4134038e10 0x7f413403b2d0 secure :-1 s=READY pgs=3475 cs=0 l=1 rev1=1 crypto rx=0x7f413c0ad000 tx=0x7f413c008040 comp rx=0 tx=0).stop 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 10 monclient: shutdown 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4154139120 msgr2=0x7f415413f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:51:44.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 --2- 172.21.15.43:0/1149653378 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4154139120 0x7f415413f240 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f414c00d800 tx=0x7f414c00dcc0 comp rx=0 tx=0).stop 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4158bf2640 1 -- 172.21.15.43:0/1149653378 reap_dead start 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f4158bf2640 1 -- 172.21.15.43:0/1149653378 reap_dead start 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 shutdown_connections 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 >> 172.21.15.43:0/1149653378 conn(0x7f41540ab2f0 msgr2=0x7f41540a8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 shutdown_connections 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 -- 172.21.15.43:0/1149653378 wait complete. 2024-04-19T00:51:44.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:51:44.843+0000 7f415a655640 1 librados: shutdown 2024-04-19T00:51:44.862 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-04-19T00:51:44.862 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-04-19T00:51:44.862 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:51:44.895 INFO:tasks.ceph.mon.i:Stopped 2024-04-19T00:51:44.895 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-04-19T00:51:44.895 INFO:tasks.ceph.mon.a:Sent signal 19 2024-04-19T00:51:44.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:51:59.896 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-04-19T00:51:59.896 INFO:tasks.ceph.mon.a:Sent signal 18 2024-04-19T00:51:59.897 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-04-19T00:51:59.897 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:52:00.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient: get_monmap_and_config 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient: build_initial_monmap 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient: monmap: 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:52:00.032084+0000 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:52:00.032084+0000 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:52:00.031 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 Processor -- start 2024-04-19T00:52:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 -- start start 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient: init 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient: _add_conns ranks=[3,5,9,4,2,7,0,10,6,8,1] 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): picked mon.noname-g con 0x7f50440a1f20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 0x7f50440a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): picked mon.noname-h con 0x7f50440a1270 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): picked mon.noname-a con 0x7f50440a4430 addr v2:172.21.15.43:3300/0 2024-04-19T00:52:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f50440a0920 con 0x7f50440a4430 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f504406bf40 con 0x7f50440a1f20 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f504409fff0 con 0x7f50440a1270 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.027+0000 7f504c355640 10 monclient(hunting): authenticate will time out at 1390.159826s 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f50498f0640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 msgr2=0x7f50440a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:52:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f50498f0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 0x7f50440a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:43378/0 (socket says 172.21.15.43:43378) 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 learned_addr learned my addr 172.21.15.43:0/256962762 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 10 monclient(hunting): get_auth_request con 0x7f50440a4430 auth_method 0 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f50440a1f20 auth_method 0 2024-04-19T00:52:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 msgr2=0x7f50440a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:52:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 10 monclient(hunting): get_auth_request con 0x7f50440a4430 auth_method 0 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 0x7f50440a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f50440a1f20 auth_method 0 2024-04-19T00:52:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 msgr2=0x7f50440a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 0x7f50440a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 10 monclient(hunting): get_auth_request con 0x7f50440a4430 auth_method 0 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f50440a1f20 auth_method 0 2024-04-19T00:52:00.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:00.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:00.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 msgr2=0x7f50440a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 10 monclient(hunting): get_auth_request con 0x7f50440a4430 auth_method 0 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:01.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 0x7f50440a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f50440a1f20 auth_method 0 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a8f2640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:01.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:01.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:03.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 10 monclient(hunting): tick 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 monclient(hunting): continuing hunt 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 msgr2=0x7f50440a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f50440a1270 0x7f50440a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 msgr2=0x7f50440a39b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f50440a1f20 0x7f50440a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 msgr2=0x7f504413fba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.43:3300/0 conn(0x7f50440a4430 0x7f504413fba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.027+0000 7f503bfff640 10 monclient: _add_conns ranks=[7,6,0,10,4,8,2,1] 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 0x7f5028005c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-i con 0x7f5028003830 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 0x7f50280086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-d con 0x7f5028006260 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:52:03.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5028008dc0 addr v1:172.21.15.43:6789/0 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50440a02d0 con 0x7f5028008dc0 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f50440a0920 con 0x7f5028006260 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f504406bf40 con 0x7f5028003830 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f503bfff640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 msgr2=0x7f50280086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 0x7f50280086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 msgr2=0x7f5028005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:03.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 0x7f5028005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:03.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.235+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 msgr2=0x7f50280086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:52:03.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.235+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 0x7f50280086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:03.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.235+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 msgr2=0x7f5028005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:03.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 0x7f5028005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:03.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.639+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 msgr2=0x7f50280086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:52:03.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.639+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 0x7f50280086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:03.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.639+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 msgr2=0x7f5028005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:03.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:03.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 0x7f5028005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:04.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:04.443+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 msgr2=0x7f50280086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:52:04.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:04.443+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 0x7f50280086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:04.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:04.443+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 msgr2=0x7f5028005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:04.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:04.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 0x7f5028005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): tick 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 monclient(hunting): continuing hunt 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 msgr2=0x7f5028005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5028003830 0x7f5028005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 msgr2=0x7f50280086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5028006260 0x7f50280086f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> v1:172.21.15.43:6789/0 conn(0x7f5028008dc0 legacy=0x7f502800b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:06.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient: _add_conns ranks=[10,2,1,8,4] 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 reap_dead start 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-f con 0x7f502800cd90 addr v2:172.21.15.161:3300/0 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 reap_dead start 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 0x7f50280066e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f5028003c50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5028009270 addr v1:172.21.15.161:6789/0 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f502800dc90 con 0x7f5028009270 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f50440a0920 con 0x7f502800cd90 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f504406bf40 con 0x7f5028003c50 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f503bfff640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:06.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 msgr2=0x7f50280066e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 0x7f50280066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:06.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f502800cd90 auth_method 0 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:06.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:06.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 msgr2=0x7f50280066e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:06.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 0x7f50280066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f502800cd90 auth_method 0 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:06.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.235+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 msgr2=0x7f50280066e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 0x7f50280066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f502800cd90 auth_method 0 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:06.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:06.639+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 msgr2=0x7f50280066e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 0x7f50280066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f502800cd90 auth_method 0 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:52:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:07.443+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:52:08.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:08.743+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v1:172.21.15.161:6789/0 conn(0x7f5028009270 legacy=0x7f502800ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-19T00:52:08.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:08.743+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 >> v1:172.21.15.161:6789/0 conn(0x7f5028009270 legacy=0x7f502800ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:52:08.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:08.743+0000 7f504a8f2640 1 --1- 172.21.15.43:0/256962762 >> v1:172.21.15.161:6789/0 conn(0x7f5028009270 0x7f502800ba50 :-1 s=OPENED pgs=90 cs=1 l=1).handle_message read tag failed 2024-04-19T00:52:08.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:08.743+0000 7f504a8f2640 1 --1- 172.21.15.43:0/256962762 >> v1:172.21.15.161:6789/0 conn(0x7f5028009270 0x7f502800ba50 :-1 s=OPENED pgs=90 cs=1 l=1).fault on lossy channel, failing 2024-04-19T00:52:08.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:08.743+0000 7f50490ef640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.161:6789/0 2024-04-19T00:52:09.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): tick 2024-04-19T00:52:09.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 monclient(hunting): continuing hunt 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 msgr2=0x7f50280066e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5028003c50 0x7f50280066e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 msgr2=0x7f502800d170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> v2:172.21.15.161:3300/0 conn(0x7f502800cd90 0x7f502800d170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 >> v1:172.21.15.161:6789/0 conn(0x7f5028009270 legacy=0x7f502800ba50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient: _add_conns ranks=[8,4] 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f502800c160 0x7f502800c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-e con 0x7f502800c160 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f502800a660 0x7f502800aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f502800a660 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:52:09.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f50440a0920 con 0x7f502800a660 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 1 -- 172.21.15.43:0/256962762 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f504406bf40 con 0x7f502800c160 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f503bfff640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f502800a660 0x7f502800aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f502800c160 0x7f502800c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): get_auth_request con 0x7f502800a660 auth_method 0 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): get_auth_request con 0x7f502800c160 auth_method 0 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f504a0f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient(hunting): handle_auth_done global_id 64246 payload 274 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f502800c160 msgr2=0x7f502800c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f502800c160 0x7f502800c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 1 monclient: found mon.noname-c 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.031+0000 7f50498f0640 1 -- 172.21.15.43:0/256962762 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502800b670 con 0x7f502800a660 2024-04-19T00:52:09.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.039+0000 7f50498f0640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f502800a660 0x7f502800aa80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f50440a5b20 tx=0x7f503401ca20 comp rx=0 tx=0).ready entity=mon.4 client_cookie=28db3c74ff500607 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 1 -- 172.21.15.43:0/256962762 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50340a5c30 con 0x7f502800a660 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: dump: 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.047 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 1 -- 172.21.15.43:0/256962762 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50340a9070 con 0x7f502800a660 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 1 -- 172.21.15.43:0/256962762 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f50340a5df0 con 0x7f502800a660 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f504c355640 5 monclient: authenticate success, global_id 64246 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f504c355640 10 monclient: get_monmap_and_config success 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f504c355640 4 set_mon_vals no callback set 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.043+0000 7f50490ef640 10 monclient: dump: 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.049 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.050 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient: shutdown 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f502800a660 msgr2=0x7f502800aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 --2- 172.21.15.43:0/256962762 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f502800a660 0x7f502800aa80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f50440a5b20 tx=0x7f503401ca20 comp rx=0 tx=0).stop 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504a8f2640 1 -- 172.21.15.43:0/256962762 reap_dead start 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- 172.21.15.43:0/256962762 shutdown_connections 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- 172.21.15.43:0/256962762 >> 172.21.15.43:0/256962762 conn(0x7f50440a85a0 msgr2=0x7f50440a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- 172.21.15.43:0/256962762 shutdown_connections 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- 172.21.15.43:0/256962762 wait complete. 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient: monmap: 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 librados: starting msgr at 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 librados: starting objecter 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 Processor -- start 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- start start 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 librados: setting wanted keys 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 librados: calling monclient init 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient: init 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient: _add_conns ranks=[2,8,0,4,6,1,5,3,7] 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5044139060 0x7f5044139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): picked mon.noname-b con 0x7f5044139060 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f50441d0a10 0x7f5044122a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): picked mon.noname-e con 0x7f50441d0a10 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 0x7f50441233c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): picked mon.noname-a con 0x7f5044122f40 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f50440a2300 con 0x7f5044122f40 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f50440a4840 con 0x7f5044139060 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f50440a2480 con 0x7f50441d0a10 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504c355640 10 monclient(hunting): authenticate will time out at 1399.179793s 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504a0f1640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5044139060 msgr2=0x7f5044139440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504a0f1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5044139060 0x7f5044139440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.047+0000 7f504a8f2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 0x7f50441233c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 0x7f50441233c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60846/0 (socket says 172.21.15.43:60846) 2024-04-19T00:52:09.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 -- 172.21.15.43:0/1611184942 learned_addr learned my addr 172.21.15.43:0/1611184942 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): get_auth_request con 0x7f5044122f40 auth_method 0 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50498f0640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f50441d0a10 0x7f5044122a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50498f0640 10 monclient(hunting): get_auth_request con 0x7f50441d0a10 auth_method 0 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50498f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50498f0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50498f0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient(hunting): handle_auth_done global_id 64350 payload 824 2024-04-19T00:52:09.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 -- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f50441d0a10 msgr2=0x7f5044122a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f50441d0a10 0x7f5044122a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 -- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5044139060 msgr2=0x7f5044139440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5044139060 0x7f5044139440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 monclient: found mon.noname-a 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 -- 172.21.15.43:0/1611184942 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50441267f0 con 0x7f5044122f40 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50498f0640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f50441d0a10 0x7f5044122a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504a8f2640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 0x7f50441233c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f503c0096d0 tx=0x7f503c008db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 client.?.objecter ms_handle_connect 0x7f5044122f40 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f503c007d60 con 0x7f5044122f40 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: dump: 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.060 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c01b070 con 0x7f5044122f40 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 5 monclient: authenticate success, global_id 64350 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 monclient: _renew_subs 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5044125060 con 0x7f5044122f40 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f503c003bb0 con 0x7f5044122f40 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 monclient: dump: 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.062 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 client.64350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 monclient: _renew_subs 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5044126510 con 0x7f5044122f40 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f50488ee640 4 set_mon_vals no callback set 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 1 librados: init done 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f503c0078f0 con 0x7f5044122f40 2024-04-19T00:52:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f502803aed0 0x7f502803d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(322..322 src has 1..322) ==== 6154+0+0 (secure 0 0 0) 0x7f503c012040 con 0x7f5044122f40 2024-04-19T00:52:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 10 client.64350.objecter ms_dispatch 0x7f5044000ba0 osd_map(322..322 src has 1..322) 2024-04-19T00:52:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 3 client.64350.objecter handle_osd_map got epochs [322,322] > 0 2024-04-19T00:52:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 3 client.64350.objecter handle_osd_map decoding full epoch 322 2024-04-19T00:52:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f503affd640 20 client.64350.objecter dump_active .. 0 homeless 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.051+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f504409ef70 con 0x7f5044122f40 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.055+0000 7f504a0f1640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f502803aed0 0x7f502803d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.055+0000 7f504a0f1640 10 monclient: get_auth_request con 0x7f502803aed0 auth_method 0 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.059+0000 7f504a0f1640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f502803aed0 0x7f502803d390 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 crypto rx=0x7f5040000eb0 tx=0x7f5040019680 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.059+0000 7f503affd640 10 client.64350.objecter ms_handle_connect 0x7f502803aed0 2024-04-19T00:52:09.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.067+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f503c01d310 con 0x7f5044122f40 2024-04-19T00:52:09.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.067+0000 7f503affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.067+0000 7f503affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f50440766e0 con 0x7f5044122f40 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f503affd640 1 -- 172.21.15.43:0/1611184942 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3168 (secure 0 0 0) 0x7f503c0047e0 con 0x7f5044122f40 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f503affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:52:09.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f503affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:52:09.319 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":60,"quorum":[0,1,3,4,8],"quorum_names":["a","f","g","c","e"],"quorum_leader_name":"e","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 librados: watch_flush enter 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 librados: watch_flush exit 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 20 client.64350.objecter shutdown clearing up homeless session... 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 client.64350.objecter successfully canceled tick 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f502803aed0 msgr2=0x7f502803d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f502803aed0 0x7f502803d390 secure :-1 s=READY pgs=3476 cs=0 l=1 rev1=1 crypto rx=0x7f5040000eb0 tx=0x7f5040019680 comp rx=0 tx=0).stop 2024-04-19T00:52:09.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 10 monclient: shutdown 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 msgr2=0x7f50441233c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 0x7f50441233c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f503c0096d0 tx=0x7f503c008db0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 shutdown_connections 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5044122f40 0x7f50441233c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f502803aed0 0x7f502803d390 unknown :-1 s=CLOSED pgs=3476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f50441d0a10 0x7f5044122a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 --2- 172.21.15.43:0/1611184942 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5044139060 0x7f5044139440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.315+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 >> 172.21.15.43:0/1611184942 conn(0x7f50440a85a0 msgr2=0x7f50440a66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.319+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 shutdown_connections 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.319+0000 7f504c355640 1 -- 172.21.15.43:0/1611184942 wait complete. 2024-04-19T00:52:09.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.319+0000 7f504c355640 1 librados: shutdown 2024-04-19T00:52:09.338 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-04-19T00:52:09.338 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-04-19T00:52:09.338 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient: get_monmap_and_config 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient: monmap: 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:52:09.466586+0000 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:52:09.466586+0000 2024-04-19T00:52:09.464 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:52:09.465 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 Processor -- start 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 -- start start 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient: init 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient: _add_conns ranks=[3,1,7,10,2,4,0,5,6,9,8] 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 0x7ff63813d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): picked mon.noname-g con 0x7ff638076190 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff638072cb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff638073110 0x7ff6380a2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): picked mon.noname-i con 0x7ff638073110 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6380a41f0 con 0x7ff638072cb0 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff638076570 con 0x7ff638076190 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff63813da90 con 0x7ff638073110 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63ea9a640 10 monclient(hunting): authenticate will time out at 1399.593758s 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63d037640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff638073110 msgr2=0x7ff6380a2fd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63d037640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff638073110 0x7ff6380a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 0x7ff63813d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 0x7ff63813d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:42916/0 (socket says 172.21.15.43:42916) 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 -- 172.21.15.43:0/3299862742 learned_addr learned my addr 172.21.15.43:0/3299862742 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): get_auth_request con 0x7ff638076190 auth_method 0 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff6377fe640 1 -- 172.21.15.43:0/3299862742 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1765479267 0 0) 0x7ff6380a41f0 con 0x7ff638072cb0 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff6377fe640 10 monclient(hunting): _negotiate 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff6377fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff6377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff6377fe640 10 monclient(hunting): my global_id is 64243 2024-04-19T00:52:09.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff6377fe640 1 -- 172.21.15.43:0/3299862742 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff618003530 con 0x7ff638072cb0 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient(hunting): handle_auth_done global_id 19128 payload 274 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 -- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff638073110 msgr2=0x7ff6380a2fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 --2- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff638073110 0x7ff6380a2fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 -- 172.21.15.43:0/3299862742 >> v1:172.21.15.161:6789/0 conn(0x7ff638072cb0 legacy=0x7ff6380a28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 monclient: found mon.noname-g 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:52:09.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.463+0000 7ff63c836640 1 -- 172.21.15.43:0/3299862742 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff638070cb0 con 0x7ff638076190 2024-04-19T00:52:09.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63c836640 1 --2- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 0x7ff63813d480 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ff6280098d0 tx=0x7ff62801c460 comp rx=0 tx=0).ready entity=mon.3 client_cookie=444a6e3dd3eea2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 1 -- 172.21.15.43:0/3299862742 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6280b0070 con 0x7ff638076190 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: dump: 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.479 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 5 monclient: authenticate success, global_id 19128 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:52:09.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 1 -- 172.21.15.43:0/3299862742 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6280aa810 con 0x7ff638076190 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: get_monmap_and_config success 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 4 set_mon_vals no callback set 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 1 -- 172.21.15.43:0/3299862742 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6280aae30 con 0x7ff638076190 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff6377fe640 10 monclient: dump: 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: shutdown 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 msgr2=0x7ff63813d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 0x7ff63813d480 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7ff6280098d0 tx=0x7ff62801c460 comp rx=0 tx=0).stop 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- 172.21.15.43:0/3299862742 shutdown_connections 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff638073110 0x7ff6380a2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/3299862742 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638076190 0x7ff63813d480 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- 172.21.15.43:0/3299862742 >> 172.21.15.43:0/3299862742 conn(0x7ff6380a7fb0 msgr2=0x7ff6380a83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- 172.21.15.43:0/3299862742 shutdown_connections 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- 172.21.15.43:0/3299862742 wait complete. 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: monmap: 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:52:09.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 librados: starting msgr at 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 librados: starting objecter 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 Processor -- start 2024-04-19T00:52:09.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- start start 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 librados: setting wanted keys 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 librados: calling monclient init 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: init 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient: _add_conns ranks=[4,3,1,8,7,2,5,0,6] 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 0x7ff6381d5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient(hunting): picked mon.noname-c con 0x7ff638072cb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638073110 0x7ff6380a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient(hunting): picked mon.noname-g con 0x7ff638073110 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638076190 0x7ff6380a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient(hunting): picked mon.noname-f con 0x7ff638076190 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.467+0000 7ff63ea9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff638147b30 con 0x7ff638076190 2024-04-19T00:52:09.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff6380a3720 con 0x7ff638073110 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff638147cb0 con 0x7ff638072cb0 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient(hunting): authenticate will time out at 1399.600021s 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 0x7ff6381d5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 0x7ff6381d5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:39852/0 (socket says 172.21.15.43:39852) 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 -- 172.21.15.43:0/169776949 learned_addr learned my addr 172.21.15.43:0/169776949 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): get_auth_request con 0x7ff638072cb0 auth_method 0 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638073110 0x7ff6380a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): get_auth_request con 0x7ff638073110 auth_method 0 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638076190 0x7ff6380a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): get_auth_request con 0x7ff638076190 auth_method 0 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient(hunting): handle_auth_done global_id 64255 payload 824 2024-04-19T00:52:09.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638073110 msgr2=0x7ff6380a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff638073110 0x7ff6380a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638076190 msgr2=0x7ff6380a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff638076190 0x7ff6380a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 monclient: found mon.noname-c 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff638070510 con 0x7ff638072cb0 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63c836640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 0x7ff6381d5000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff6280ae040 tx=0x7ff628019200 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 client.?.objecter ms_handle_connect 0x7ff638072cb0 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:52:09.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6280b9070 con 0x7ff638072cb0 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: dump: 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.490 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6280044b0 con 0x7ff638072cb0 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 5 monclient: authenticate success, global_id 64255 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient: _renew_subs 2024-04-19T00:52:09.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63809bff0 con 0x7ff638072cb0 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6280b0040 con 0x7ff638072cb0 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 client.64255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff636ffd640 4 set_mon_vals no callback set 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: dump: 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.492 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient: _renew_subs 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63813a550 con 0x7ff638072cb0 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 librados: init done 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 0x7ff638138060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff6381d5d10 con 0x7ff638137c80 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 0x7ff638138060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient: get_auth_request con 0x7ff638137c80 auth_method 0 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff6280aacd0 con 0x7ff638072cb0 2024-04-19T00:52:09.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff61c038e10 0x7ff61c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 10 monclient(hunting): handle_auth_done global_id 64359 payload 824 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff63806d120 con 0x7ff638137c80 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(322..322 src has 1..322) ==== 6154+0+0 (secure 0 0 0) 0x7ff6280e2450 con 0x7ff638072cb0 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 client.64255.objecter ms_dispatch 0x7ff638000ba0 osd_map(322..322 src has 1..322) 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 3 client.64255.objecter handle_osd_map got epochs [322,322] > 0 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 3 client.64255.objecter handle_osd_map decoding full epoch 322 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 20 client.64255.objecter dump_active .. 0 homeless 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff637fff640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 0x7ff638138060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff63806d0f0 tx=0x7ff62000ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 client.64255.objecter ms_handle_connect 0x7ff638137c80 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 client.64255.objecter resend_mon_ops 2024-04-19T00:52:09.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff61c038e10 0x7ff61c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff620003c50 con 0x7ff638137c80 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: dump: 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 10 monclient: get_auth_request con 0x7ff61c038e10 auth_method 0 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff620018070 con 0x7ff638137c80 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 msgr2=0x7ff638138060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 0x7ff638138060 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff63806d0f0 tx=0x7ff62000ff70 comp rx=0 tx=0).stop 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 msgr2=0x7ff638138060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff638137c80 0x7ff638138060 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff63806d0f0 tx=0x7ff62000ff70 comp rx=0 tx=0).stop 2024-04-19T00:52:09.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff6357fa640 10 client.64255.objecter ms_handle_connect 0x7ff61c038e10 2024-04-19T00:52:09.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.471+0000 7ff63d037640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff61c038e10 0x7ff61c03b2d0 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7ff6381d55d0 tx=0x7ff630006f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:52:09.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 0x7ff63813bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7ff63806ffc0 con 0x7ff6381385a0 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 0x7ff63813bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient: get_auth_request con 0x7ff6381385a0 auth_method 0 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 10 monclient(hunting): handle_auth_done global_id 64368 payload 824 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 1 -- 172.21.15.43:0/169776949 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff63813a310 con 0x7ff6381385a0 2024-04-19T00:52:09.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff637fff640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 0x7ff63813bd20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff620017000 tx=0x7ff620001370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 client.64255.objecter ms_handle_connect 0x7ff6381385a0 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 client.64255.objecter resend_mon_ops 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff62000d630 con 0x7ff6381385a0 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 monclient: dump: 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.501 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9398 (secure 0 0 0) 0x7ff620005420 con 0x7ff6381385a0 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 msgr2=0x7ff63813bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 0x7ff63813bd20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff620017000 tx=0x7ff620001370 comp rx=0 tx=0).stop 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 msgr2=0x7ff63813bd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff6357fa640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff6381385a0 0x7ff63813bd20 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff620017000 tx=0x7ff620001370 comp rx=0 tx=0).stop 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 60, 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 55751, 2024-04-19T00:52:09.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.504 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.505 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:52:09.506 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:52:09.508 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.511 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:52:09.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:52:09.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.516 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:52:09.520 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:09.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:52:09.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.525 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:52:09.526 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:52:09.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 10 librados: watch_flush enter 2024-04-19T00:52:09.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 10 librados: watch_flush exit 2024-04-19T00:52:09.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 20 client.64255.objecter shutdown clearing up homeless session... 2024-04-19T00:52:09.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 10 client.64255.objecter successfully canceled tick 2024-04-19T00:52:09.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff61c038e10 msgr2=0x7ff61c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff61c038e10 0x7ff61c03b2d0 secure :-1 s=READY pgs=3477 cs=0 l=1 rev1=1 crypto rx=0x7ff6381d55d0 tx=0x7ff630006f90 comp rx=0 tx=0).stop 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 10 monclient: shutdown 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 msgr2=0x7ff6381d5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 0x7ff6381d5000 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff6280ae040 tx=0x7ff628019200 comp rx=0 tx=0).stop 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63d037640 1 -- 172.21.15.43:0/169776949 reap_dead start 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 shutdown_connections 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 --2- 172.21.15.43:0/169776949 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff638072cb0 0x7ff6381d5000 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 >> 172.21.15.43:0/169776949 conn(0x7ff6380a7fb0 msgr2=0x7ff638071a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 shutdown_connections 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 -- 172.21.15.43:0/169776949 wait complete. 2024-04-19T00:52:09.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.479+0000 7ff63ea9a640 1 librados: shutdown 2024-04-19T00:52:09.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:52:09.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-04-19T00:52:09.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.647+0000 7f29c23b3640 10 monclient: get_monmap_and_config 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.647+0000 7f29c23b3640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.647+0000 7f29c23b3640 10 monclient: monmap: 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:52:09.654185+0000 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:52:09.654185+0000 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:52:09.652 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:52:09.653 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 1 Processor -- start 2024-04-19T00:52:09.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 1 -- start start 2024-04-19T00:52:09.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient: init 2024-04-19T00:52:09.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient: _add_conns ranks=[10,1,0,5,6,2,8,3,4,7,9] 2024-04-19T00:52:09.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f29bc138cd0 0x7f29bc1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient(hunting): picked mon.noname-f con 0x7f29bc138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f29bc0a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29bc1395f0 addr v1:172.21.15.43:6789/0 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29bc1366d0 con 0x7f29bc1395f0 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29bc0a3dc0 con 0x7f29bc0a5b80 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f29bc0a4410 con 0x7f29bc138cd0 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 10 monclient(hunting): authenticate will time out at 1399.781339s 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f29bc1395f0 0x7f29bc135fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49756/0 (socket says 172.21.15.43:49756) 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 -- 172.21.15.43:0/1358321493 learned_addr learned my addr 172.21.15.43:0/1358321493 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 1 --2- 172.21.15.43:0/1358321493 >> v2:172.21.15.161:3300/0 conn(0x7f29bc138cd0 0x7f29bc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): get_auth_request con 0x7f29bc138cd0 auth_method 0 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378438295 0 0) 0x7f29bc1366d0 con 0x7f29bc1395f0 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): _negotiate 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): my global_id is 64377 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29a8003530 con 0x7f29bc1395f0 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3651244584 0 0) 0x7f29a8003530 con 0x7f29bc1395f0 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 >> v2:172.21.15.161:3300/0 conn(0x7f29bc138cd0 msgr2=0x7f29bc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 --2- 172.21.15.43:0/1358321493 >> v2:172.21.15.161:3300/0 conn(0x7f29bc138cd0 0x7f29bc1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 >> v1:172.21.15.161:6789/0 conn(0x7f29bc0a5b80 legacy=0x7f29bc1358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc0a40a0 con 0x7f29bc1395f0 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f29b4001cf0 con 0x7f29bc1395f0 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 5 monclient: authenticate success, global_id 64377 2024-04-19T00:52:09.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: dump: 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.668 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 >> v1:172.21.15.43:6789/0 conn(0x7f29bc1395f0 legacy=0x7f29bc135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient: _add_conns ranks=[7,2,4,8,5,3,6,1,0] 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8004210 0x7f29a8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): picked mon.i con 0x7f29a8004210 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29a8004da0 0x7f29a8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): picked mon.b con 0x7f29a8004da0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:52:09.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29a8009c40 0x7f29a800c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): picked mon.c con 0x7f29a8009c40 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f29bc0a4410 con 0x7f29a8004da0 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f29a800c9b0 con 0x7f29a8009c40 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f29a800cb60 con 0x7f29a8004210 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bb7fe640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c23b3640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29b3fff640 1 -- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29a8004da0 msgr2=0x7f29a8005220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 1 -- 172.21.15.43:0/1358321493 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8004210 msgr2=0x7f29a8004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:09.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29b3fff640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29a8004da0 0x7f29a8005220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29bbfff640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8004210 0x7f29a8004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29a8009c40 0x7f29a800c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): get_auth_request con 0x7f29a8009c40 auth_method 0 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient(hunting): handle_auth_done global_id 64377 payload 293 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 -- 172.21.15.43:0/1358321493 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8004210 msgr2=0x7f29a8004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8004210 0x7f29a8004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 -- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29a8004da0 msgr2=0x7f29a8005220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:09.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29a8004da0 0x7f29a8005220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 monclient: found mon.c 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 1 -- 172.21.15.43:0/1358321493 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29a8003530 con 0x7f29a8009c40 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.651+0000 7f29c0950640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 1 -- 172.21.15.43:0/1358321493 reap_dead start 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29a8009c40 0x7f29a800c0d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29b400a610 tx=0x7f29b401d020 comp rx=0 tx=0).ready entity=mon.4 client_cookie=90e265993b710962 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b400a260 con 0x7f29a8009c40 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: dump: 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.673 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b40a9070 con 0x7f29a8009c40 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: get_monmap_and_config success 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 4 set_mon_vals no callback set 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 1 -- 172.21.15.43:0/1358321493 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b40079b0 con 0x7f29a8009c40 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bb7fe640 10 monclient: dump: 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.674 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.675 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: shutdown 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29a8009c40 msgr2=0x7f29a800c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29a8009c40 0x7f29a800c0d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f29b400a610 tx=0x7f29b401d020 comp rx=0 tx=0).stop 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- 172.21.15.43:0/1358321493 shutdown_connections 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1358321493 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29a8009c40 0x7f29a800c0d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- 172.21.15.43:0/1358321493 >> 172.21.15.43:0/1358321493 conn(0x7f29bc0ab2f0 msgr2=0x7f29bc0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- 172.21.15.43:0/1358321493 shutdown_connections 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- 172.21.15.43:0/1358321493 wait complete. 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: monmap: 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:52:09.676 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 librados: starting msgr at 2024-04-19T00:52:09.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 librados: starting objecter 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 Processor -- start 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- start start 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 librados: setting wanted keys 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 librados: calling monclient init 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: init 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient: _add_conns ranks=[7,2,4,3,8,5,6,0,1] 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8009c40 0x7f29bc1d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): picked mon.noname-i con 0x7f29a8009c40 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29bc1d0fe0 0x7f29bc1331b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): picked mon.noname-b con 0x7f29bc1d0fe0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 0x7f29bc1336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): picked mon.noname-c con 0x7f29bc1d1400 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f29bc0a2010 con 0x7f29bc1d0fe0 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f29bc0a1e90 con 0x7f29bc1d1400 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f29bc1390b0 con 0x7f29a8009c40 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c23b3640 10 monclient(hunting): authenticate will time out at 1399.787591s 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29b3fff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29bc1d0fe0 msgr2=0x7f29bc1331b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29b3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29bc1d0fe0 0x7f29bc1331b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bbfff640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8009c40 msgr2=0x7f29bc1d0aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29bbfff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8009c40 0x7f29bc1d0aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 0x7f29bc1336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 0x7f29bc1336f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:39860/0 (socket says 172.21.15.43:39860) 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 1 -- 172.21.15.43:0/1159285514 learned_addr learned my addr 172.21.15.43:0/1159285514 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): get_auth_request con 0x7f29bc1d1400 auth_method 0 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.655+0000 7f29c0950640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 10 monclient(hunting): handle_auth_done global_id 64264 payload 824 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8009c40 msgr2=0x7f29bc1d0aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f29a8009c40 0x7f29bc1d0aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29bc1d0fe0 msgr2=0x7f29bc1331b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f29bc1d0fe0 0x7f29bc1331b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 monclient: found mon.noname-c 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29bc09c720 con 0x7f29bc1d1400 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c0950640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 0x7f29bc1336f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f90 tx=0x7f29a8005840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 client.?.objecter ms_handle_connect 0x7f29bc1d1400 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b400a260 con 0x7f29bc1d1400 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: dump: 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.681 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b4001cb0 con 0x7f29bc1d1400 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 5 monclient: authenticate success, global_id 64264 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 monclient: _renew_subs 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29bc09cce0 con 0x7f29bc1d1400 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29b4007ae0 con 0x7f29bc1d1400 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29baffd640 4 set_mon_vals no callback set 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: dump: 2024-04-19T00:52:09.683 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.684 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.696 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 client.64264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 monclient: _renew_subs 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f29bc09ca00 con 0x7f29bc1d1400 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 1 librados: init done 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 0x7f29bc134010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f29bc06d140 con 0x7f29bc133c30 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 0x7f29bc134010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient: get_auth_request con 0x7f29bc133c30 auth_method 0 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f29b4007670 con 0x7f29bc1d1400 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 10 monclient(hunting): handle_auth_done global_id 64273 payload 824 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29bc0693b0 con 0x7f29bc133c30 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f299c038e10 0x7f299c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(322..322 src has 1..322) ==== 6154+0+0 (secure 0 0 0) 0x7f29b40e2570 con 0x7f29bc1d1400 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 client.64264.objecter ms_dispatch 0x7f29bc000ba0 osd_map(322..322 src has 1..322) 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 3 client.64264.objecter handle_osd_map got epochs [322,322] > 0 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 3 client.64264.objecter handle_osd_map decoding full epoch 322 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29bbfff640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 0x7f29bc134010 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e750 tx=0x7f29a400ec10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 20 client.64264.objecter dump_active .. 0 homeless 2024-04-19T00:52:09.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 client.64264.objecter ms_handle_connect 0x7f29bc133c30 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 client.64264.objecter resend_mon_ops 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29a4017070 con 0x7f29bc133c30 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: dump: 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.699 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b3fff640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f299c038e10 0x7f299c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b3fff640 10 monclient: get_auth_request con 0x7f299c038e10 auth_method 0 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f29a400ee60 con 0x7f29bc133c30 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 msgr2=0x7f29bc134010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 0x7f29bc134010 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e750 tx=0x7f29a400ec10 comp rx=0 tx=0).stop 2024-04-19T00:52:09.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 msgr2=0x7f29bc134010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc133c30 0x7f29bc134010 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b97fa640 10 client.64264.objecter ms_handle_connect 0x7f299c038e10 2024-04-19T00:52:09.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.659+0000 7f29b3fff640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f299c038e10 0x7f299c03b2d0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 crypto rx=0x7f29bc134890 tx=0x7f29ac007a00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 0x7f29bc137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f29bc068240 con 0x7f29bc1376c0 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 0x7f29bc137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient: get_auth_request con 0x7f29bc1376c0 auth_method 0 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 10 monclient(hunting): handle_auth_done global_id 64282 payload 824 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 1 -- 172.21.15.43:0/1159285514 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29bc134550 con 0x7f29bc1376c0 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 client.64264.objecter ms_handle_connect 0x7f29bc1376c0 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 client.64264.objecter resend_mon_ops 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29bbfff640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 0x7f29bc137ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c10 tx=0x7f29a4004740 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f29a4017030 con 0x7f29bc1376c0 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 monclient: dump: 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.704 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9228 (secure 0 0 0) 0x7f29a40066d0 con 0x7f29bc1376c0 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 msgr2=0x7f29bc137ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 0x7f29bc137ad0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c10 tx=0x7f29a4004740 comp rx=0 tx=0).stop 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 msgr2=0x7f29bc137ad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29b97fa640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1376c0 0x7f29bc137ad0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c10 tx=0x7f29a4004740 comp rx=0 tx=0).stop 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 60, 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 55939, 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:52:09.706 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:52:09.709 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.710 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.711 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:52:09.713 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.718 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.723 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.725 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:52:09.726 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 10 librados: watch_flush enter 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 10 librados: watch_flush exit 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 20 client.64264.objecter shutdown clearing up homeless session... 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 10 client.64264.objecter successfully canceled tick 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f299c038e10 msgr2=0x7f299c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f299c038e10 0x7f299c03b2d0 secure :-1 s=READY pgs=3478 cs=0 l=1 rev1=1 crypto rx=0x7f29bc134890 tx=0x7f29ac007a00 comp rx=0 tx=0).stop 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 10 monclient: shutdown 2024-04-19T00:52:09.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 msgr2=0x7f29bc1336f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 0x7f29bc1336f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f90 tx=0x7f29a8005840 comp rx=0 tx=0).stop 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c0950640 1 -- 172.21.15.43:0/1159285514 reap_dead start 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 shutdown_connections 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 --2- 172.21.15.43:0/1159285514 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f29bc1d1400 0x7f29bc1336f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 >> 172.21.15.43:0/1159285514 conn(0x7f29bc0ab2f0 msgr2=0x7f29bc0a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 shutdown_connections 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 -- 172.21.15.43:0/1159285514 wait complete. 2024-04-19T00:52:09.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.667+0000 7f29c23b3640 1 librados: shutdown 2024-04-19T00:52:09.732 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:52:09.732 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon e 2024-04-19T00:52:09.733 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:52:09.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.855+0000 7efdf8daa640 10 monclient: get_monmap_and_config 2024-04-19T00:52:09.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.855+0000 7efdf8daa640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient: monmap: 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:52:09.862531+0000 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:52:09.862531+0000 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.861 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:52:09.862 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 Processor -- start 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 -- start start 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient: init 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient: _add_conns ranks=[1,4,5,7,9,3,10,0,8,6,2] 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7efdf40a3050 addr v1:172.21.15.161:6789/0 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 0x7efdf40a3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): picked mon.noname-c con 0x7efdf40a3430 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf413bf20 0x7efdf413e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): picked mon.noname-h con 0x7efdf413bf20 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdf409dd20 con 0x7efdf40a3050 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7efdf409ec70 con 0x7efdf40a3430 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7efdf409ef50 con 0x7efdf413bf20 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf8daa640 10 monclient(hunting): authenticate will time out at 1399.989629s 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf2d9d640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf413bf20 msgr2=0x7efdf413e310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 0x7efdf40a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf2d9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf413bf20 0x7efdf413e310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 0x7efdf40a3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:39880/0 (socket says 172.21.15.43:39880) 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 -- 172.21.15.43:0/3551895800 learned_addr learned my addr 172.21.15.43:0/3551895800 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): get_auth_request con 0x7efdf40a3430 auth_method 0 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient(hunting): handle_auth_done global_id 64291 payload 274 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 -- 172.21.15.43:0/3551895800 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf413bf20 msgr2=0x7efdf413e310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 --2- 172.21.15.43:0/3551895800 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf413bf20 0x7efdf413e310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 -- 172.21.15.43:0/3551895800 >> v1:172.21.15.161:6789/0 conn(0x7efdf40a3050 legacy=0x7efdf4136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 monclient: found mon.noname-c 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.859+0000 7efdf1d9b640 1 -- 172.21.15.43:0/3551895800 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf409f230 con 0x7efdf40a3430 2024-04-19T00:52:09.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf1d9b640 1 --2- 172.21.15.43:0/3551895800 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 0x7efdf40a3870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efddc0098a0 tx=0x7efddc01c460 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d8be4d35888820c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 1 -- 172.21.15.43:0/3551895800 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efddc0b0070 con 0x7efdf40a3430 2024-04-19T00:52:09.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: dump: 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.874 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 1 -- 172.21.15.43:0/3551895800 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efddc0aa810 con 0x7efdf40a3430 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 1 -- 172.21.15.43:0/3551895800 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efddc0a9300 con 0x7efdf40a3430 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 5 monclient: authenticate success, global_id 64291 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: get_monmap_and_config success 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 4 set_mon_vals no callback set 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf159a640 10 monclient: dump: 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.876 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: shutdown 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- 172.21.15.43:0/3551895800 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 msgr2=0x7efdf40a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 --2- 172.21.15.43:0/3551895800 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 0x7efdf40a3870 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efddc0098a0 tx=0x7efddc01c460 comp rx=0 tx=0).stop 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- 172.21.15.43:0/3551895800 shutdown_connections 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 --2- 172.21.15.43:0/3551895800 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf413bf20 0x7efdf413e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 --2- 172.21.15.43:0/3551895800 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efdf40a3430 0x7efdf40a3870 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- 172.21.15.43:0/3551895800 >> 172.21.15.43:0/3551895800 conn(0x7efdf40a4140 msgr2=0x7efdf40a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- 172.21.15.43:0/3551895800 shutdown_connections 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- 172.21.15.43:0/3551895800 wait complete. 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: build_initial_monmap 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: monmap: 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:52:09.878 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 librados: starting msgr at 2024-04-19T00:52:09.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 librados: starting objecter 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 Processor -- start 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- start start 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 librados: setting wanted keys 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 librados: calling monclient init 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: init 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient: _add_conns ranks=[5,7,0,8,2,4,6,1,3] 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf40a3050 0x7efdf41d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): picked mon.noname-h con 0x7efdf40a3050 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efdf40a3430 0x7efdf41d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): picked mon.noname-i con 0x7efdf40a3430 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdf413bf20 0x7efdf41d1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): picked mon.noname-a con 0x7efdf413bf20 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efdf4141400 con 0x7efdf413bf20 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7efdf4141280 con 0x7efdf40a3050 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7efdf4141580 con 0x7efdf40a3430 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.863+0000 7efdf8daa640 10 monclient(hunting): authenticate will time out at 1399.995965s 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdf413bf20 0x7efdf41d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf1d9b640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efdf40a3430 msgr2=0x7efdf41d0bd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 1 -- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf40a3050 msgr2=0x7efdf41d0690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3302/0 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf1d9b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efdf40a3430 0x7efdf41d0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf40a3050 0x7efdf41d0690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdf413bf20 0x7efdf41d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60858/0 (socket says 172.21.15.43:60858) 2024-04-19T00:52:09.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 -- 172.21.15.43:0/2304138387 learned_addr learned my addr 172.21.15.43:0/2304138387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): get_auth_request con 0x7efdf413bf20 auth_method 0 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient(hunting): handle_auth_done global_id 64386 payload 824 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efdf40a3430 msgr2=0x7efdf41d0bd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efdf40a3430 0x7efdf41d0bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf40a3050 msgr2=0x7efdf41d0690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:09.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efdf40a3050 0x7efdf41d0690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 monclient: found mon.noname-a 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf412dd70 con 0x7efdf413bf20 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf2d9d640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdf413bf20 0x7efdf41d1110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efde400b440 tx=0x7efde400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 client.?.objecter ms_handle_connect 0x7efdf413bf20 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efde4017070 con 0x7efdf413bf20 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: dump: 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.883 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: _finish_auth 0 2024-04-19T00:52:09.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde40060d0 con 0x7efdf413bf20 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 5 monclient: authenticate success, global_id 64386 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 monclient: _renew_subs 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf412e330 con 0x7efdf413bf20 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efde40066f0 con 0x7efdf413bf20 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdebfff640 4 set_mon_vals no callback set 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 client.64386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: dump: 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.885 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.886 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 monclient: _renew_subs 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf412c990 con 0x7efdf413bf20 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 1 librados: init done 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 0x7efdf4072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7efdf406d140 con 0x7efdf4072650 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7efde4006280 con 0x7efdf413bf20 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 0x7efdf4072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient: get_auth_request con 0x7efdf4072650 auth_method 0 2024-04-19T00:52:09.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efdd4038e10 0x7efdd403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(322..322 src has 1..322) ==== 6154+0+0 (secure 0 0 0) 0x7efde4048560 con 0x7efdf413bf20 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 client.64386.objecter ms_dispatch 0x7efdf4000ba0 osd_map(322..322 src has 1..322) 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 3 client.64386.objecter handle_osd_map got epochs [322,322] > 0 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 3 client.64386.objecter handle_osd_map decoding full epoch 322 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 10 monclient(hunting): handle_auth_done global_id 23210 payload 824 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdf40008e0 con 0x7efdf4072650 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 20 client.64386.objecter dump_active .. 0 homeless 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 client.64386.objecter ms_handle_connect 0x7efdf4072650 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 client.64386.objecter resend_mon_ops 2024-04-19T00:52:09.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf259c640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 0x7efdf4072a30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7efdec000f50 tx=0x7efdec00cb80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdec00a5b0 con 0x7efdf4072650 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf1d9b640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efdd4038e10 0x7efdd403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: dump: 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.889 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf1d9b640 10 monclient: get_auth_request con 0x7efdd4038e10 auth_method 0 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efdec005b50 con 0x7efdf4072650 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 msgr2=0x7efdf4072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 0x7efdf4072a30 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7efdec000f50 tx=0x7efdec00cb80 comp rx=0 tx=0).stop 2024-04-19T00:52:09.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 msgr2=0x7efdf4072a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072650 0x7efdf4072a30 secure :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7efdec000f50 tx=0x7efdec00cb80 comp rx=0 tx=0).stop 2024-04-19T00:52:09.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdeaffd640 10 client.64386.objecter ms_handle_connect 0x7efdd4038e10 2024-04-19T00:52:09.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.867+0000 7efdf1d9b640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efdd4038e10 0x7efdd403b2d0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7efddc0ad000 tx=0x7efddc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:52:09.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 0x7efdf409b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7efdf412df40 con 0x7efdf4072f70 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 0x7efdf409b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient: get_auth_request con 0x7efdf4072f70 auth_method 0 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 10 monclient(hunting): handle_auth_done global_id 23219 payload 824 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 1 -- 172.21.15.43:0/2304138387 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efdf4136330 con 0x7efdf4072f70 2024-04-19T00:52:09.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 client.64386.objecter ms_handle_connect 0x7efdf4072f70 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 client.64386.objecter resend_mon_ops 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf259c640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 0x7efdf409b640 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7efdec013000 tx=0x7efdec001260 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efdec00ac30 con 0x7efdf4072f70 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 monclient: dump: 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:09.894 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9401 (secure 0 0 0) 0x7efdec00a7c0 con 0x7efdf4072f70 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 msgr2=0x7efdf409b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 0x7efdf409b640 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7efdec013000 tx=0x7efdec001260 comp rx=0 tx=0).stop 2024-04-19T00:52:09.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 msgr2=0x7efdf409b640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdeaffd640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efdf4072f70 0x7efdf409b640 secure :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7efdec013000 tx=0x7efdec001260 comp rx=0 tx=0).stop 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 60, 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 473387, 2024-04-19T00:52:09.896 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:52:09.897 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.898 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.899 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:52:09.900 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:09.901 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:52:09.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:52:09.905 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:52:09.906 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:52:09.910 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.912 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.915 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:09.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:52:09.919 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 10 librados: watch_flush enter 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 10 librados: watch_flush exit 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 20 client.64386.objecter shutdown clearing up homeless session... 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 10 client.64386.objecter successfully canceled tick 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efdd4038e10 msgr2=0x7efdd403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efdd4038e10 0x7efdd403b2d0 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 crypto rx=0x7efddc0ad000 tx=0x7efddc008040 comp rx=0 tx=0).stop 2024-04-19T00:52:09.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 10 monclient: shutdown 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdf413bf20 msgr2=0x7efdf41d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 --2- 172.21.15.43:0/2304138387 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efdf413bf20 0x7efdf41d1110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efde400b440 tx=0x7efde400b900 comp rx=0 tx=0).stop 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf2d9d640 1 -- 172.21.15.43:0/2304138387 reap_dead start 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf2d9d640 1 -- 172.21.15.43:0/2304138387 reap_dead start 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 shutdown_connections 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 >> 172.21.15.43:0/2304138387 conn(0x7efdf40a4140 msgr2=0x7efdf40992d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 shutdown_connections 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 -- 172.21.15.43:0/2304138387 wait complete. 2024-04-19T00:52:09.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:09.875+0000 7efdf8daa640 1 librados: shutdown 2024-04-19T00:52:09.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:52:09.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-04-19T00:52:09.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient: get_monmap_and_config 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient: build_initial_monmap 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient: monmap: 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:52:10.042712+0000 2024-04-19T00:52:10.040 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:52:10.042712+0000 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:52:10.041 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 Processor -- start 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 -- start start 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient: init 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient: _add_conns ranks=[1,3,7,9,6,4,2,10,8,0,5] 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f4660072810 addr v1:172.21.15.161:6789/0 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f46601390a0 0x7f466013df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): picked mon.noname-g con 0x7f46601390a0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f466013e450 0x7f4660140840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): picked mon.noname-i con 0x7f466013e450 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f466006db70 con 0x7f4660072810 2024-04-19T00:52:10.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f466006f870 con 0x7f46601390a0 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f46601394c0 con 0x7f466013e450 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f4665550640 10 monclient(hunting): authenticate will time out at 1400.169665s 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465f7fe640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f466013e450 msgr2=0x7f4660140840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465f7fe640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f466013e450 0x7f4660140840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4660072810 0x7f4660072bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:55398/0 (socket says 172.21.15.43:55398) 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465effd640 1 -- 172.21.15.43:0/2546576908 learned_addr learned my addr 172.21.15.43:0/2546576908 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 1 --2- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f46601390a0 0x7f466013df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): get_auth_request con 0x7f46601390a0 auth_method 0 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465dffb640 1 -- 172.21.15.43:0/2546576908 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2251812 0 0) 0x7f466006db70 con 0x7f4660072810 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465dffb640 10 monclient(hunting): _negotiate 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465dffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465dffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465dffb640 10 monclient(hunting): my global_id is 64279 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465dffb640 1 -- 172.21.15.43:0/2546576908 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4648003530 con 0x7f4660072810 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient(hunting): handle_auth_done global_id 19146 payload 274 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 1 -- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f466013e450 msgr2=0x7f4660140840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 1 --2- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f466013e450 0x7f4660140840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 1 -- 172.21.15.43:0/2546576908 >> v1:172.21.15.161:6789/0 conn(0x7f4660072810 legacy=0x7f4660072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:52:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 1 monclient: found mon.noname-g 2024-04-19T00:52:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:52:10.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.039+0000 7f465e7fc640 1 -- 172.21.15.43:0/2546576908 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466006f2b0 con 0x7f46601390a0 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 1 --2- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f46601390a0 0x7f466013df10 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f46540097f0 tx=0x7f465401c390 comp rx=0 tx=0).ready entity=mon.3 client_cookie=1d7649ce0bf66111 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 1 -- 172.21.15.43:0/2546576908 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46540b0070 con 0x7f46601390a0 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: dump: 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.053 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: _finish_auth 0 2024-04-19T00:52:10.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 1 -- 172.21.15.43:0/2546576908 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46540aa810 con 0x7f46601390a0 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 1 -- 172.21.15.43:0/2546576908 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46540aae30 con 0x7f46601390a0 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465dffb640 10 monclient: dump: 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.055 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 5 monclient: authenticate success, global_id 19146 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: get_monmap_and_config success 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 4 set_mon_vals no callback set 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: shutdown 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:10.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f46601390a0 msgr2=0x7f466013df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 --2- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f46601390a0 0x7f466013df10 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f46540097f0 tx=0x7f465401c390 comp rx=0 tx=0).stop 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- 172.21.15.43:0/2546576908 shutdown_connections 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 --2- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f466013e450 0x7f4660140840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 --2- 172.21.15.43:0/2546576908 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f46601390a0 0x7f466013df10 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- 172.21.15.43:0/2546576908 >> 172.21.15.43:0/2546576908 conn(0x7f46600a62e0 msgr2=0x7f466006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- 172.21.15.43:0/2546576908 shutdown_connections 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- 172.21.15.43:0/2546576908 wait complete. 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: build_initial_monmap 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: monmap: 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:52:10.057 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 librados: starting msgr at 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 librados: starting objecter 2024-04-19T00:52:10.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 Processor -- start 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- start start 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 librados: setting wanted keys 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 librados: calling monclient init 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: init 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient: _add_conns ranks=[6,0,3,4,8,2,5,7,1] 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4660072810 0x7f46601d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): picked mon.noname-d con 0x7f4660072810 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46601390a0 0x7f4660135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): picked mon.noname-a con 0x7f46601390a0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f466013e450 0x7f466013a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): picked mon.noname-g con 0x7f466013e450 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4660073650 con 0x7f46601390a0 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f46601435a0 con 0x7f466013e450 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f466006e5a0 con 0x7f4660072810 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f4665550640 10 monclient(hunting): authenticate will time out at 1400.175500s 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465effd640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4660072810 msgr2=0x7f46601d0590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4660072810 0x7f46601d0590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46601390a0 0x7f4660135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46601390a0 0x7f4660135920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:60862/0 (socket says 172.21.15.43:60862) 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 1 -- 172.21.15.43:0/3858932609 learned_addr learned my addr 172.21.15.43:0/3858932609 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): get_auth_request con 0x7f46601390a0 auth_method 0 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465f7fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f466013e450 0x7f466013a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465f7fe640 10 monclient(hunting): get_auth_request con 0x7f466013e450 auth_method 0 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 10 monclient(hunting): handle_auth_done global_id 64395 payload 824 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4660072810 msgr2=0x7f46601d0590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.043+0000 7f465e7fc640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4660072810 0x7f46601d0590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f466013e450 msgr2=0x7f466013a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f466013e450 0x7f466013a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 1 monclient: found mon.noname-a 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466013c6a0 con 0x7f46601390a0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465e7fc640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46601390a0 0x7f4660135920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4654009710 tx=0x7f4654019130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 client.?.objecter ms_handle_connect 0x7f46601390a0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46540b0040 con 0x7f46601390a0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: dump: 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.062 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: _finish_auth 0 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: _check_auth_tickets 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46540b9070 con 0x7f46601390a0 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 5 monclient: authenticate success, global_id 64395 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 monclient: _renew_subs 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466011fa10 con 0x7f46601390a0 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f46540044b0 con 0x7f46601390a0 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 client.64395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 monclient: dump: 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.064 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465cff9640 4 set_mon_vals no callback set 2024-04-19T00:52:10.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 monclient: _renew_subs 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46601d1410 con 0x7f46601390a0 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 1 librados: init done 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 0x7f466013b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4660120210 con 0x7f466013aee0 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 0x7f466013b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient: get_auth_request con 0x7f466013aee0 auth_method 0 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f46540aac80 con 0x7f46601390a0 2024-04-19T00:52:10.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f464c038e10 0x7f464c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(322..322 src has 1..322) ==== 6154+0+0 (secure 0 0 0) 0x7f46540e2f60 con 0x7f46601390a0 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 client.64395.objecter ms_dispatch 0x7f4660000ba0 osd_map(322..322 src has 1..322) 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 3 client.64395.objecter handle_osd_map got epochs [322,322] > 0 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 3 client.64395.objecter handle_osd_map decoding full epoch 322 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 20 client.64395.objecter dump_active .. 0 homeless 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 10 monclient(hunting): handle_auth_done global_id 64288 payload 824 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f466013b800 con 0x7f466013aee0 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465f7fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f464c038e10 0x7f464c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465f7fe640 10 monclient: get_auth_request con 0x7f464c038e10 auth_method 0 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 client.64395.objecter ms_handle_connect 0x7f466013aee0 2024-04-19T00:52:10.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 client.64395.objecter resend_mon_ops 2024-04-19T00:52:10.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465effd640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 0x7f466013b2c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4650004820 tx=0x7f465000cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f46477fe640 10 client.64395.objecter ms_handle_connect 0x7f464c038e10 2024-04-19T00:52:10.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.047+0000 7f465f7fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f464c038e10 0x7f464c03b2d0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7f466013cf60 tx=0x7f4658009450 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4650017070 con 0x7f466013aee0 2024-04-19T00:52:10.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 10 monclient: dump: 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.070 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f465000ce50 con 0x7f466013aee0 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 msgr2=0x7f466013b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 0x7f466013b2c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4650004820 tx=0x7f465000cb10 comp rx=0 tx=0).stop 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 msgr2=0x7f466013b2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.059+0000 7f46477fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f466013aee0 0x7f466013b2c0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4650004820 tx=0x7f465000cb10 comp rx=0 tx=0).stop 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f4665550640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:52:10.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f4665550640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 0x7f46600a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f4665550640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46600737d0 con 0x7f46600a0420 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 0x7f46600a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient: get_auth_request con 0x7f46600a0420 auth_method 0 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 10 monclient(hunting): handle_auth_done global_id 64297 payload 824 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 1 -- 172.21.15.43:0/3858932609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4660076440 con 0x7f46600a0420 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f465effd640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 0x7f46600a0830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4650014000 tx=0x7f465000f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f46477fe640 10 client.64395.objecter ms_handle_connect 0x7f46600a0420 2024-04-19T00:52:10.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.067+0000 7f46477fe640 10 client.64395.objecter resend_mon_ops 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f465000a5e0 con 0x7f46600a0420 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 10 monclient: dump: 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.077 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.078 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9228 (secure 0 0 0) 0x7f46500073a0 con 0x7f46600a0420 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 msgr2=0x7f46600a0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 0x7f46600a0830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4650014000 tx=0x7f465000f000 comp rx=0 tx=0).stop 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 msgr2=0x7f46600a0830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f46477fe640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46600a0420 0x7f46600a0830 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4650014000 tx=0x7f465000f000 comp rx=0 tx=0).stop 2024-04-19T00:52:10.079 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 60, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 56343, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:10.080 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.081 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:52:10.083 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:10.084 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.086 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.087 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.088 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:52:10.089 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.090 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.091 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.092 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:52:10.093 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.094 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:52:10.095 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.096 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:52:10.097 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.099 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:10.100 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.101 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:52:10.101 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:52:10.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 10 librados: watch_flush enter 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 10 librados: watch_flush exit 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 20 client.64395.objecter shutdown clearing up homeless session... 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 10 client.64395.objecter successfully canceled tick 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f464c038e10 msgr2=0x7f464c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f464c038e10 0x7f464c03b2d0 secure :-1 s=READY pgs=3480 cs=0 l=1 rev1=1 crypto rx=0x7f466013cf60 tx=0x7f4658009450 comp rx=0 tx=0).stop 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 10 monclient: shutdown 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46601390a0 msgr2=0x7f4660135920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 --2- 172.21.15.43:0/3858932609 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f46601390a0 0x7f4660135920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4654009710 tx=0x7f4654019130 comp rx=0 tx=0).stop 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f465f7fe640 1 -- 172.21.15.43:0/3858932609 reap_dead start 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f465f7fe640 1 -- 172.21.15.43:0/3858932609 reap_dead start 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 shutdown_connections 2024-04-19T00:52:10.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 >> 172.21.15.43:0/3858932609 conn(0x7f46600a62e0 msgr2=0x7f46600a3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:10.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 shutdown_connections 2024-04-19T00:52:10.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 -- 172.21.15.43:0/3858932609 wait complete. 2024-04-19T00:52:10.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.075+0000 7f4665550640 1 librados: shutdown 2024-04-19T00:52:10.104 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:52:10.104 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-04-19T00:52:10.104 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:52:10.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.223+0000 7ff4d5fd8640 10 monclient: get_monmap_and_config 2024-04-19T00:52:10.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.223+0000 7ff4d5fd8640 10 monclient: build_initial_monmap 2024-04-19T00:52:10.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.223+0000 7ff4d5fd8640 10 monclient: monmap: 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:52:10.228960+0000 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:52:10.228960+0000 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:10.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:10.228 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:10.228 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:52:10.228 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:52:10.228 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.223+0000 7ff4d5fd8640 1 Processor -- start 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.223+0000 7ff4d5fd8640 1 -- start start 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.223+0000 7ff4d5fd8640 10 monclient: init 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient: _add_conns ranks=[9,1,8,0,3,4,5,7,2,10,6] 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff4d0138cd0 0x7ff4d01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d0138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:52:10.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff4d00a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 0x7ff4d0135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): picked mon.noname-e con 0x7ff4d01395f0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d00a3dc0 con 0x7ff4d00a5b80 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d00a4410 con 0x7ff4d0138cd0 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ff4d00a40a0 con 0x7ff4d01395f0 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient(hunting): authenticate will time out at 1400.356216s 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 0x7ff4d0135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 0x7ff4d0135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:40726/0 (socket says 172.21.15.43:40726) 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 -- 172.21.15.43:0/1101486979 learned_addr learned my addr 172.21.15.43:0/1101486979 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): get_auth_request con 0x7ff4d01395f0 auth_method 0 2024-04-19T00:52:10.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 1 --2- 172.21.15.43:0/1101486979 >> v2:172.21.15.43:3300/0 conn(0x7ff4d0138cd0 0x7ff4d01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request con 0x7ff4d0138cd0 auth_method 0 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient(hunting): handle_auth_done global_id 23228 payload 274 2024-04-19T00:52:10.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 -- 172.21.15.43:0/1101486979 >> v2:172.21.15.43:3300/0 conn(0x7ff4d0138cd0 msgr2=0x7ff4d01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 --2- 172.21.15.43:0/1101486979 >> v2:172.21.15.43:3300/0 conn(0x7ff4d0138cd0 0x7ff4d01390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 -- 172.21.15.43:0/1101486979 >> v1:172.21.15.161:6789/0 conn(0x7ff4d00a5b80 legacy=0x7ff4d01358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient: _finish_hunting 0 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 monclient: found mon.noname-e 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:52:10.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 -- 172.21.15.43:0/1101486979 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d00a2ae0 con 0x7ff4d01395f0 2024-04-19T00:52:10.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4cffff640 1 --2- 172.21.15.43:0/1101486979 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 0x7ff4d0135fc0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00a7450 tx=0x7ff4c801c8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=21dcfa8f5e18405e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 1 -- 172.21.15.43:0/1101486979 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4c80090d0 con 0x7ff4d01395f0 2024-04-19T00:52:10.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: dump: 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.240 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: _finish_auth 0 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 5 monclient: authenticate success, global_id 23228 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 1 -- 172.21.15.43:0/1101486979 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c80a5b80 con 0x7ff4d01395f0 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient: get_monmap_and_config success 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 4 set_mon_vals no callback set 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 1 -- 172.21.15.43:0/1101486979 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4c80a5d70 con 0x7ff4d01395f0 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4ce7fc640 10 monclient: dump: 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 10 monclient: shutdown 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/1101486979 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 msgr2=0x7ff4d0135fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/1101486979 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 0x7ff4d0135fc0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7ff4d00a7450 tx=0x7ff4c801c8b0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/1101486979 shutdown_connections 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/1101486979 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff4d01395f0 0x7ff4d0135fc0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/1101486979 >> v2:172.21.15.43:3300/0 conn(0x7ff4d0138cd0 0x7ff4d01390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/1101486979 >> 172.21.15.43:0/1101486979 conn(0x7ff4d00ab2f0 msgr2=0x7ff4d00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/1101486979 shutdown_connections 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.227+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/1101486979 wait complete. 2024-04-19T00:52:10.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: build_initial_monmap 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: monmap: 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:52:10.244 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 librados: starting msgr at 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 librados: starting objecter 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 Processor -- start 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- start start 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 librados: setting wanted keys 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 librados: calling monclient init 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: init 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: _reopen_session rank -1 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: _add_conns ranks=[7,3,4,2,8,6,5,1,0] 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff4d00a5b80 0x7ff4d00a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): picked mon.noname-i con 0x7ff4d00a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:52:10.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d0138cd0 0x7ff4d012e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): picked mon.noname-g con 0x7ff4d0138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff4d01395f0 0x7ff4d012ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d01395f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d0136500 con 0x7ff4d0138cd0 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff4d00a20a0 con 0x7ff4d01395f0 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff4d0136680 con 0x7ff4d00a5b80 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): _renew_subs 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): authenticate will time out at 1400.361366s 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff4d00a5b80 msgr2=0x7ff4d00a0d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T00:52:10.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff4d00a5b80 0x7ff4d00a0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff4d01395f0 0x7ff4d012ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff4d01395f0 0x7ff4d012ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:39884/0 (socket says 172.21.15.43:39884) 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 -- 172.21.15.43:0/2324552896 learned_addr learned my addr 172.21.15.43:0/2324552896 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): get_auth_request con 0x7ff4d01395f0 auth_method 0 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d0138cd0 0x7ff4d012e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request con 0x7ff4d0138cd0 auth_method 0 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient(hunting): handle_auth_done global_id 64300 payload 824 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff4d00a5b80 msgr2=0x7ff4d00a0d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff4d00a5b80 0x7ff4d00a0d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d0138cd0 msgr2=0x7ff4d012e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d0138cd0 0x7ff4d012e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient: _finish_hunting 0 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 monclient: found mon.noname-c 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d00a0250 con 0x7ff4d01395f0 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cffff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff4d01395f0 0x7ff4d012ee10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009040 tx=0x7ff4c8019650 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 client.?.objecter ms_handle_connect 0x7ff4d01395f0 2024-04-19T00:52:10.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4c80bc020 con 0x7ff4d01395f0 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: dump: 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.249 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: _finish_auth 0 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: _check_auth_tickets 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c8003e10 con 0x7ff4d01395f0 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 5 monclient: authenticate success, global_id 64300 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: _renew_subs 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d009eda0 con 0x7ff4d01395f0 2024-04-19T00:52:10.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4c80044b0 con 0x7ff4d01395f0 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 client.64300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cdffb640 4 set_mon_vals no callback set 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 monclient: dump: 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.251 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: _renew_subs 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d009ff70 con 0x7ff4d01395f0 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 librados: init done 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff4c80bf020 con 0x7ff4d01395f0 2024-04-19T00:52:10.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 0x7ff4d009c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d006d140 con 0x7ff4d012f350 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff4b0038e10 0x7ff4b003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(323..323 src has 1..323) ==== 5769+0+0 (secure 0 0 0) 0x7ff4c80b3040 con 0x7ff4d01395f0 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 10 client.64300.objecter ms_dispatch 0x7ff4d0000ba0 osd_map(323..323 src has 1..323) 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 3 client.64300.objecter handle_osd_map got epochs [323,323] > 0 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 3 client.64300.objecter handle_osd_map decoding full epoch 323 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4c7fff640 20 client.64300.objecter dump_active .. 0 homeless 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 0x7ff4d009c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 10 monclient: get_auth_request con 0x7ff4d012f350 auth_method 0 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff4b0038e10 0x7ff4b003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.231+0000 7ff4ceffd640 10 monclient: get_auth_request con 0x7ff4b0038e10 auth_method 0 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_done global_id 19173 payload 824 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4cf7fe640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4d00a3590 con 0x7ff4d012f350 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4ceffd640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff4b0038e10 0x7ff4b003b2d0 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7ff4d009dcd0 tx=0x7ff4c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 client.64300.objecter ms_handle_connect 0x7ff4b0038e10 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 client.64300.objecter ms_handle_connect 0x7ff4d012f350 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 client.64300.objecter resend_mon_ops 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4cf7fe640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 0x7ff4d009c570 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80094c0 tx=0x7ff4b800ec90 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4b8017070 con 0x7ff4d012f350 2024-04-19T00:52:10.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 monclient: dump: 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.255 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4b800c9b0 con 0x7ff4d012f350 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 msgr2=0x7ff4d009c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 0x7ff4d009c570 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80094c0 tx=0x7ff4b800ec90 comp rx=0 tx=0).stop 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 msgr2=0x7ff4d009c570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:10.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.235+0000 7ff4c7fff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d012f350 0x7ff4d009c570 secure :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80094c0 tx=0x7ff4b800ec90 comp rx=0 tx=0).stop 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 0x7ff4d009cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d0068240 con 0x7ff4d009cab0 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 0x7ff4d009cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient: get_auth_request con 0x7ff4d009cab0 auth_method 0 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:52:10.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 10 monclient(hunting): handle_auth_done global_id 19182 payload 824 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 1 -- 172.21.15.43:0/2324552896 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4d012f730 con 0x7ff4d009cab0 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cf7fe640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 0x7ff4d009cec0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800ecd0 tx=0x7ff4b8004740 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 client.64300.objecter ms_handle_connect 0x7ff4d009cab0 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 client.64300.objecter resend_mon_ops 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4b8017030 con 0x7ff4d009cab0 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 monclient: dump: 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:52:10.259 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9569 (secure 0 0 0) 0x7ff4b8005bb0 con 0x7ff4d009cab0 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:52:10.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 msgr2=0x7ff4d009cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 0x7ff4d009cec0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800ecd0 tx=0x7ff4b8004740 comp rx=0 tx=0).stop 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 msgr2=0x7ff4d009cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4c7fff640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7ff4d009cab0 0x7ff4d009cec0 secure :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800ecd0 tx=0x7ff4b8004740 comp rx=0 tx=0).stop 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 60, 2024-04-19T00:52:10.261 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 473727, 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:10.262 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:10.263 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:52:10.264 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:52:10.265 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.266 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:52:10.268 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:52:10.270 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:52:10.272 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:52:10.274 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:52:10.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:52:10.279 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:52:10.281 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.282 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:52:10.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:52:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:52:10.284 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:52:10.284 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:52:10.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 10 librados: watch_flush enter 2024-04-19T00:52:10.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 10 librados: watch_flush exit 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 20 client.64300.objecter shutdown clearing up homeless session... 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 10 client.64300.objecter successfully canceled tick 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff4b0038e10 msgr2=0x7ff4b003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff4b0038e10 0x7ff4b003b2d0 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7ff4d009dcd0 tx=0x7ff4c0009040 comp rx=0 tx=0).stop 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 10 monclient: shutdown 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff4d01395f0 msgr2=0x7ff4d012ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 --2- 172.21.15.43:0/2324552896 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff4d01395f0 0x7ff4d012ee10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009040 tx=0x7ff4c8019650 comp rx=0 tx=0).stop 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cffff640 1 -- 172.21.15.43:0/2324552896 reap_dead start 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4cffff640 1 -- 172.21.15.43:0/2324552896 reap_dead start 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 shutdown_connections 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 >> 172.21.15.43:0/2324552896 conn(0x7ff4d00ab2f0 msgr2=0x7ff4d0131da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:52:10.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 shutdown_connections 2024-04-19T00:52:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 -- 172.21.15.43:0/2324552896 wait complete. 2024-04-19T00:52:10.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:52:10.243+0000 7ff4d5fd8640 1 librados: shutdown 2024-04-19T00:52:10.287 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:52:10.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T00:53:40.288 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-19T00:53:40.288 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-19T00:53:40.288 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T00:53:40.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T00:53:40.292 INFO:tasks.ceph.mon.b:Started 2024-04-19T00:53:40.292 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T00:53:40.292 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-04-19T00:53:40.293 INFO:tasks.ceph.mon.d:Restarting daemon 2024-04-19T00:53:40.293 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-04-19T00:53:40.295 INFO:tasks.ceph.mon.d:Started 2024-04-19T00:53:40.295 INFO:tasks.mon_thrash.mon_thrasher:killing mon.h 2024-04-19T00:53:40.295 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.h 2024-04-19T00:53:40.295 INFO:tasks.ceph.mon.h:Restarting daemon 2024-04-19T00:53:40.295 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i h 2024-04-19T00:53:40.298 INFO:tasks.ceph.mon.h:Started 2024-04-19T00:53:40.298 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-04-19T00:53:40.298 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-04-19T00:53:40.298 INFO:tasks.ceph.mon.i:Restarting daemon 2024-04-19T00:53:40.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-04-19T00:53:40.301 INFO:tasks.ceph.mon.i:Started 2024-04-19T00:53:40.301 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-04-19T00:53:40.301 INFO:tasks.ceph.mon.a:Sent signal 19 2024-04-19T00:53:40.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:53:40.402 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T00:53:40.398+0000 7f0d87277c80 -1 mon.b@-1(???).log v0 log_external_backlog rewinding external_log_to from 319 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:53:40.404 INFO:tasks.ceph.mon.i.smithi161.stderr:2024-04-19T00:53:40.404+0000 7f1d414c3c80 -1 mon.i@-1(???).log v0 log_external_backlog rewinding external_log_to from 319 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:53:55.302 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-04-19T00:53:55.303 INFO:tasks.ceph.mon.a:Sent signal 18 2024-04-19T00:53:55.303 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:53:55.303 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:53:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient: get_monmap_and_config 2024-04-19T00:53:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient: build_initial_monmap 2024-04-19T00:53:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient: monmap: 2024-04-19T00:53:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.469 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:55.472021+0000 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:55.472021+0000 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:55.471 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 Processor -- start 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 -- start start 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient: init 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient: _add_conns ranks=[1,9,7,6,8,5,2,3,0,4,10] 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe694138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe694139540 0x7fe69413dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): picked mon.noname-a con 0x7fe694139540 addr v2:172.21.15.43:3300/0 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6940a5bb0 0x7fe694140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): picked mon.noname-i con 0x7fe6940a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6940a0d40 con 0x7fe694138a50 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fe6940a1020 con 0x7fe694139540 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe6940a0780 con 0x7fe6940a5bb0 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe69b453640 10 monclient(hunting): authenticate will time out at 1505.598744s 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe694139540 0x7fe69413dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe694139540 0x7fe69413dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42774/0 (socket says 172.21.15.43:42774) 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 -- 172.21.15.43:0/1655943058 learned_addr learned my addr 172.21.15.43:0/1655943058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): get_auth_request con 0x7fe694139540 auth_method 0 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 1 --2- 172.21.15.43:0/1655943058 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6940a5bb0 0x7fe694140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): get_auth_request con 0x7fe6940a5bb0 auth_method 0 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6999f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient(hunting): handle_auth_done global_id 64413 payload 274 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 -- 172.21.15.43:0/1655943058 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6940a5bb0 msgr2=0x7fe694140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 --2- 172.21.15.43:0/1655943058 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6940a5bb0 0x7fe694140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 -- 172.21.15.43:0/1655943058 >> v1:172.21.15.161:6789/0 conn(0x7fe694138a50 legacy=0x7fe694138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient: _finish_hunting 0 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 monclient: found mon.noname-a 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.466+0000 7fe6989ee640 1 -- 172.21.15.43:0/1655943058 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6940a0a60 con 0x7fe694139540 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 1 --2- 172.21.15.43:0/1655943058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694139540 0x7fe69413dfe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6880098a0 tx=0x7fe68801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b72808e26e38981 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 1 -- 172.21.15.43:0/1655943058 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6880b1070 con 0x7fe694139540 2024-04-19T00:53:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: dump: 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: _finish_auth 0 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: _check_auth_tickets 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 5 monclient: authenticate success, global_id 64413 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 1 -- 172.21.15.43:0/1655943058 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6880ab810 con 0x7fe694139540 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 4 set_mon_vals no callback set 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 1 -- 172.21.15.43:0/1655943058 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6880aa300 con 0x7fe694139540 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe687fff640 10 monclient: dump: 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: shutdown 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- 172.21.15.43:0/1655943058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694139540 msgr2=0x7fe69413dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 --2- 172.21.15.43:0/1655943058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694139540 0x7fe69413dfe0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe6880098a0 tx=0x7fe68801c490 comp rx=0 tx=0).stop 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- 172.21.15.43:0/1655943058 shutdown_connections 2024-04-19T00:53:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 --2- 172.21.15.43:0/1655943058 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6940a5bb0 0x7fe694140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 --2- 172.21.15.43:0/1655943058 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe694139540 0x7fe69413dfe0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- 172.21.15.43:0/1655943058 >> 172.21.15.43:0/1655943058 conn(0x7fe6940a7cb0 msgr2=0x7fe6940a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- 172.21.15.43:0/1655943058 shutdown_connections 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- 172.21.15.43:0/1655943058 wait complete. 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: build_initial_monmap 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: monmap: 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 librados: starting msgr at 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 librados: starting objecter 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 Processor -- start 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- start start 2024-04-19T00:53:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 librados: setting wanted keys 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 librados: calling monclient init 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: init 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient: _add_conns ranks=[1,6,7,3,8,5,0,4,2] 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6940a5bb0 0x7fe6941d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): picked mon.noname-f con 0x7fe6940a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 0x7fe6941d1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): picked mon.noname-d con 0x7fe694138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe694139540 0x7fe69413c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): picked mon.noname-i con 0x7fe694139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe694143620 con 0x7fe6940a5bb0 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fe6941434a0 con 0x7fe694138a50 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe6941437a0 con 0x7fe694139540 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe69b453640 10 monclient(hunting): authenticate will time out at 1505.603750s 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 0x7fe6941d1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 0x7fe6941d1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33120/0 (socket says 172.21.15.43:33120) 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 1 -- 172.21.15.43:0/2884256653 learned_addr learned my addr 172.21.15.43:0/2884256653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 10 monclient(hunting): get_auth_request con 0x7fe694138a50 auth_method 0 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.470+0000 7fe6989ee640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe694139540 0x7fe69413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): get_auth_request con 0x7fe694139540 auth_method 0 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6940a5bb0 0x7fe6941d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 10 monclient(hunting): get_auth_request con 0x7fe6940a5bb0 auth_method 0 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6999f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 10 monclient(hunting): handle_auth_done global_id 74103 payload 824 2024-04-19T00:53:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 -- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe694139540 msgr2=0x7fe69413c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe694139540 0x7fe69413c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 -- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6940a5bb0 msgr2=0x7fe6941d0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6940a5bb0 0x7fe6941d0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 10 monclient: _finish_hunting 0 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 monclient: found mon.noname-d 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 -- 172.21.15.43:0/2884256653 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6941220c0 con 0x7fe694138a50 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6940a5bb0 0x7fe6941d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6989ee640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 0x7fe6941d1060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe68801a0c0 tx=0x7fe688019230 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 client.?.objecter ms_handle_connect 0x7fe694138a50 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6880bc070 con 0x7fe694138a50 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: dump: 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.493 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: _finish_auth 0 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6880044b0 con 0x7fe694138a50 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6880b1070 con 0x7fe694138a50 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 5 monclient: authenticate success, global_id 74103 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6877fe640 4 set_mon_vals no callback set 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 monclient: dump: 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.495 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 monclient: _renew_subs 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69413f200 con 0x7fe694138a50 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 client.74103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 monclient: _renew_subs 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69413f980 con 0x7fe694138a50 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 1 librados: init done 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe688004040 con 0x7fe694138a50 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:55.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69413e460 con 0x7fe694138a50 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe678038e10 0x7fe67803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(367..367 src has 1..367) ==== 6170+0+0 (secure 0 0 0) 0x7fe6880e2b30 con 0x7fe694138a50 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 client.74103.objecter ms_dispatch 0x7fe694000ba0 osd_map(367..367 src has 1..367) 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 3 client.74103.objecter handle_osd_map got epochs [367,367] > 0 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 3 client.74103.objecter handle_osd_map decoding full epoch 367 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 20 client.74103.objecter dump_active .. 0 homeless 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe678038e10 0x7fe67803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 10 monclient: get_auth_request con 0x7fe678038e10 auth_method 0 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe685ffb640 10 client.74103.objecter ms_handle_connect 0x7fe678038e10 2024-04-19T00:53:55.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.474+0000 7fe6991ef640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe678038e10 0x7fe67803b2d0 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7fe690001e50 tx=0x7fe690009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.478+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe688016110 con 0x7fe694138a50 2024-04-19T00:53:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.478+0000 7fe685ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:55.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.478+0000 7fe685ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:53:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:53:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe694121800 con 0x7fe694138a50 2024-04-19T00:53:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe685ffb640 1 -- 172.21.15.43:0/2884256653 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7fe688003d80 con 0x7fe694138a50 2024-04-19T00:53:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe685ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:53:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe685ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:55.705 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":66,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:53:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 librados: watch_flush enter 2024-04-19T00:53:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 librados: watch_flush exit 2024-04-19T00:53:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 20 client.74103.objecter shutdown clearing up homeless session... 2024-04-19T00:53:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 client.74103.objecter successfully canceled tick 2024-04-19T00:53:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe678038e10 msgr2=0x7fe67803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe678038e10 0x7fe67803b2d0 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7fe690001e50 tx=0x7fe690009040 comp rx=0 tx=0).stop 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 10 monclient: shutdown 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 msgr2=0x7fe6941d1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 0x7fe6941d1060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe68801a0c0 tx=0x7fe688019230 comp rx=0 tx=0).stop 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 shutdown_connections 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe678038e10 0x7fe67803b2d0 unknown :-1 s=CLOSED pgs=3487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe694139540 0x7fe69413c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe694138a50 0x7fe6941d1060 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 --2- 172.21.15.43:0/2884256653 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6940a5bb0 0x7fe6941d0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 >> 172.21.15.43:0/2884256653 conn(0x7fe6940a7cb0 msgr2=0x7fe6940a46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 shutdown_connections 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 -- 172.21.15.43:0/2884256653 wait complete. 2024-04-19T00:53:55.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.702+0000 7fe69b453640 1 librados: shutdown 2024-04-19T00:53:55.722 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:53:55.723 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:53:55.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.842+0000 7f235c1df640 10 monclient: get_monmap_and_config 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.842+0000 7f235c1df640 10 monclient: build_initial_monmap 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.842+0000 7f235c1df640 10 monclient: monmap: 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:55.848691+0000 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:55.848691+0000 2024-04-19T00:53:55.847 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.842+0000 7f235c1df640 1 Processor -- start 2024-04-19T00:53:55.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.842+0000 7f235c1df640 1 -- start start 2024-04-19T00:53:55.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.842+0000 7f235c1df640 10 monclient: init 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient: _add_conns ranks=[7,10,2,6,1,0,5,3,4,9,8] 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f23540a7930 0x7f23540a2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): picked mon.noname-i con 0x7f23540a7930 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2354134b40 0x7f2354134f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): picked mon.noname-f con 0x7f2354134b40 addr v2:172.21.15.161:3300/0 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 0x7f235409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): picked mon.noname-b con 0x7f2354135460 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f23540a3740 con 0x7f2354134b40 2024-04-19T00:53:55.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f23540a3250 con 0x7f2354135460 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f2354137e50 con 0x7f23540a7930 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235c1df640 10 monclient(hunting): authenticate will time out at 1505.976584s 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 0x7f235409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 0x7f235409ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40594/0 (socket says 172.21.15.43:40594) 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 -- 172.21.15.43:0/1904044245 learned_addr learned my addr 172.21.15.43:0/1904044245 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): get_auth_request con 0x7f2354135460 auth_method 0 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient(hunting): handle_auth_done global_id 74099 payload 274 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 -- 172.21.15.43:0/1904044245 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f23540a7930 msgr2=0x7f23540a2930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 --2- 172.21.15.43:0/1904044245 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f23540a7930 0x7f23540a2930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 -- 172.21.15.43:0/1904044245 >> v2:172.21.15.161:3300/0 conn(0x7f2354134b40 msgr2=0x7f2354134f20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 --2- 172.21.15.43:0/1904044245 >> v2:172.21.15.161:3300/0 conn(0x7f2354134b40 0x7f2354134f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient: _finish_hunting 0 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 monclient: found mon.noname-b 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:53:55.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 -- 172.21.15.43:0/1904044245 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23540a3490 con 0x7f2354135460 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.846+0000 7f235a77c640 1 --2- 172.21.15.43:0/1904044245 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 0x7f235409ba70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f235409e5a0 tx=0x7f234401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6d814a66d0afaf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 1 -- 172.21.15.43:0/1904044245 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23440b2070 con 0x7f2354135460 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: dump: 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.854 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: _finish_auth 0 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: _check_auth_tickets 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 1 -- 172.21.15.43:0/1904044245 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23440ab7f0 con 0x7f2354135460 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 5 monclient: authenticate success, global_id 74099 2024-04-19T00:53:55.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 1 -- 172.21.15.43:0/1904044245 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23440abe50 con 0x7f2354135460 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 4 set_mon_vals no callback set 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f2358f79640 10 monclient: dump: 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.856 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 10 monclient: shutdown 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 -- 172.21.15.43:0/1904044245 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 msgr2=0x7f235409ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 --2- 172.21.15.43:0/1904044245 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 0x7f235409ba70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f235409e5a0 tx=0x7f234401c3e0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 -- 172.21.15.43:0/1904044245 shutdown_connections 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 --2- 172.21.15.43:0/1904044245 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354135460 0x7f235409ba70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 --2- 172.21.15.43:0/1904044245 >> v2:172.21.15.161:3300/0 conn(0x7f2354134b40 0x7f2354134f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 --2- 172.21.15.43:0/1904044245 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f23540a7930 0x7f23540a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 -- 172.21.15.43:0/1904044245 >> 172.21.15.43:0/1904044245 conn(0x7f23540a7520 msgr2=0x7f23540a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 -- 172.21.15.43:0/1904044245 shutdown_connections 2024-04-19T00:53:55.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 -- 172.21.15.43:0/1904044245 wait complete. 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 10 monclient: build_initial_monmap 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 10 monclient: monmap: 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:55.860 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 librados: starting msgr at 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 librados: starting objecter 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 Processor -- start 2024-04-19T00:53:55.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 -- start start 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 librados: setting wanted keys 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 1 librados: calling monclient init 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.850+0000 7f235c1df640 10 monclient: init 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient: _add_conns ranks=[5,2,4,1,3,6,8,7,0] 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23540a7930 0x7f23541d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): picked mon.noname-h con 0x7f23540a7930 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354134b40 0x7f2354133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): picked mon.noname-b con 0x7f2354134b40 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2354135460 0x7f235406f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): picked mon.noname-c con 0x7f2354135460 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f235409cb50 con 0x7f2354134b40 2024-04-19T00:53:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f235409c9d0 con 0x7f2354135460 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f235413e080 con 0x7f23540a7930 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient(hunting): authenticate will time out at 1505.984578s 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2354135460 0x7f235406f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235977a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354134b40 0x7f2354133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2354135460 0x7f235406f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44894/0 (socket says 172.21.15.43:44894) 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 -- 172.21.15.43:0/3662776331 learned_addr learned my addr 172.21.15.43:0/3662776331 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): get_auth_request con 0x7f2354135460 auth_method 0 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235977a640 10 monclient(hunting): get_auth_request con 0x7f2354134b40 auth_method 0 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235977a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235977a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235977a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f2359f7b640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23540a7930 0x7f23541d4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient(hunting): handle_auth_done global_id 64309 payload 824 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23540a7930 msgr2=0x7f23541d4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23540a7930 0x7f23541d4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354134b40 msgr2=0x7f2354133a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2354134b40 0x7f2354133a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient: _finish_hunting 0 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 monclient: found mon.noname-c 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:53:55.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23540716b0 con 0x7f2354135460 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235a77c640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2354135460 0x7f235406f070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f23440b0040 tx=0x7f23440191e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 client.?.objecter ms_handle_connect 0x7f2354135460 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23440b2040 con 0x7f2354135460 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: dump: 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23440bb070 con 0x7f2354135460 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 5 monclient: authenticate success, global_id 64309 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient: _renew_subs 2024-04-19T00:53:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23541d5970 con 0x7f2354135460 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23440044b0 con 0x7f2354135460 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 client.64309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234b7fe640 4 set_mon_vals no callback set 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f234a7fc640 10 monclient: dump: 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient: _renew_subs 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f2354072bd0 con 0x7f2354135460 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.854+0000 7f235c1df640 1 librados: init done 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235c1df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235c1df640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 0x7f235406f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f23540a08b0 con 0x7f235406f5b0 2024-04-19T00:53:55.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2344004040 con 0x7f2354135460 2024-04-19T00:53:55.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 0x7f235406f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient: get_auth_request con 0x7f235406f5b0 auth_method 0 2024-04-19T00:53:55.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2338038e10 0x7f233803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(368..368 src has 1..368) ==== 6170+0+0 (secure 0 0 0) 0x7f23440e2ed0 con 0x7f2354135460 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 client.64309.objecter ms_dispatch 0x7f2354000ba0 osd_map(368..368 src has 1..368) 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 3 client.64309.objecter handle_osd_map got epochs [368,368] > 0 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 3 client.64309.objecter handle_osd_map decoding full epoch 368 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 10 monclient(hunting): handle_auth_done global_id 64422 payload 824 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2354072610 con 0x7f235406f5b0 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 20 client.64309.objecter dump_active .. 0 homeless 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 client.64309.objecter ms_handle_connect 0x7f235406f5b0 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 client.64309.objecter resend_mon_ops 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f2359f7b640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 0x7f235406f990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f235000c440 tx=0x7f235000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2350015070 con 0x7f235406f5b0 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 monclient: dump: 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.874 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235977a640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2338038e10 0x7f233803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235977a640 10 monclient: get_auth_request con 0x7f2338038e10 auth_method 0 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2350011600 con 0x7f235406f5b0 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 msgr2=0x7f235406f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 0x7f235406f990 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f235000c440 tx=0x7f235000c900 comp rx=0 tx=0).stop 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 msgr2=0x7f235406f990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406f5b0 0x7f235406f990 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f235000c440 tx=0x7f235000c900 comp rx=0 tx=0).stop 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f234a7fc640 10 client.64309.objecter ms_handle_connect 0x7f2338038e10 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.858+0000 7f235977a640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2338038e10 0x7f233803b2d0 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7f235413bfc0 tx=0x7f234c0094f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f235c1df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f235c1df640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 0x7f2354138030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f235c1df640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f2354071060 con 0x7f235406fed0 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 0x7f2354138030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient: get_auth_request con 0x7f235406fed0 auth_method 0 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:55.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 10 monclient(hunting): handle_auth_done global_id 64431 payload 824 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 1 -- 172.21.15.43:0/3662776331 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23540693b0 con 0x7f235406fed0 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f2359f7b640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 0x7f2354138030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f235000c440 tx=0x7f2350004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 client.64309.objecter ms_handle_connect 0x7f235406fed0 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 client.64309.objecter resend_mon_ops 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f235001aba0 con 0x7f235406fed0 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 monclient: dump: 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f23500243f0 con 0x7f235406fed0 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 162138, 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:55.879 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:55.880 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:55.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:55.882 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:55.883 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.887 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:55.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:55.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:55.898 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:55.901 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 msgr2=0x7f2354138030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 0x7f2354138030 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f235000c440 tx=0x7f2350004760 comp rx=0 tx=0).stop 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 msgr2=0x7f2354138030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.866+0000 7f234a7fc640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f235406fed0 0x7f2354138030 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 10 librados: watch_flush enter 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 10 librados: watch_flush exit 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 20 client.64309.objecter shutdown clearing up homeless session... 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 10 client.64309.objecter successfully canceled tick 2024-04-19T00:53:55.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2338038e10 msgr2=0x7f233803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2338038e10 0x7f233803b2d0 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7f235413bfc0 tx=0x7f234c0094f0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 10 monclient: shutdown 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2354135460 msgr2=0x7f235406f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 --2- 172.21.15.43:0/3662776331 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2354135460 0x7f235406f070 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f23440b0040 tx=0x7f23440191e0 comp rx=0 tx=0).stop 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235a77c640 1 -- 172.21.15.43:0/3662776331 reap_dead start 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235a77c640 1 -- 172.21.15.43:0/3662776331 reap_dead start 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 shutdown_connections 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 >> 172.21.15.43:0/3662776331 conn(0x7f23540a7520 msgr2=0x7f23540a25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 shutdown_connections 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 -- 172.21.15.43:0/3662776331 wait complete. 2024-04-19T00:53:55.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:55.870+0000 7f235c1df640 1 librados: shutdown 2024-04-19T00:53:55.905 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:55.905 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:53:55.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:53:56.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient: get_monmap_and_config 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient: monmap: 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:56.020729+0000 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:56.020729+0000 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.021 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 Processor -- start 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 -- start start 2024-04-19T00:53:56.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient: init 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient: _add_conns ranks=[8,3,10,1,7,0,6,2,4,5,9] 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 0x7f624c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): picked mon.noname-e con 0x7f624c138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): picked mon.noname-g con 0x7f624c0a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f624c1358b0 0x7f624c135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): picked mon.noname-f con 0x7f624c1358b0 addr v2:172.21.15.161:3300/0 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f624c0a4410 con 0x7f624c1358b0 2024-04-19T00:53:56.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f624c139b30 con 0x7f624c0a5b80 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f624c0a3dc0 con 0x7f624c138cd0 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.014+0000 7f6253fcf640 10 monclient(hunting): authenticate will time out at 1506.147927s 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 0x7f624c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 0x7f624c1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54382/0 (socket says 172.21.15.43:54382) 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 -- 172.21.15.43:0/1884894243 learned_addr learned my addr 172.21.15.43:0/1884894243 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): get_auth_request con 0x7f624c138cd0 auth_method 0 2024-04-19T00:53:56.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625156a640 1 --2- 172.21.15.43:0/1884894243 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625256c640 1 --2- 172.21.15.43:0/1884894243 >> v2:172.21.15.161:3300/0 conn(0x7f624c1358b0 0x7f624c135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625156a640 10 monclient(hunting): get_auth_request con 0x7f624c0a5b80 auth_method 0 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625156a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625156a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625156a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625256c640 10 monclient(hunting): get_auth_request con 0x7f624c1358b0 auth_method 0 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625256c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625256c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f625256c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient(hunting): handle_auth_done global_id 23237 payload 274 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 -- 172.21.15.43:0/1884894243 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 msgr2=0x7f624c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 --2- 172.21.15.43:0/1884894243 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 -- 172.21.15.43:0/1884894243 >> v2:172.21.15.161:3300/0 conn(0x7f624c1358b0 msgr2=0x7f624c135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 --2- 172.21.15.43:0/1884894243 >> v2:172.21.15.161:3300/0 conn(0x7f624c1358b0 0x7f624c135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 monclient: found mon.noname-e 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 -- 172.21.15.43:0/1884894243 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624c0a40a0 con 0x7f624c138cd0 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6251d6b640 1 --2- 172.21.15.43:0/1884894243 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 0x7f624c1390b0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f6244009e50 tx=0x7f624401c890 comp rx=0 tx=0).ready entity=mon.8 client_cookie=3d12a906b3e432f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 1 -- 172.21.15.43:0/1884894243 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62440090d0 con 0x7f624c138cd0 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: dump: 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.033 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.034 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 5 monclient: authenticate success, global_id 23237 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 1 -- 172.21.15.43:0/1884894243 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62440a5a50 con 0x7f624c138cd0 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 4 set_mon_vals no callback set 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 1 -- 172.21.15.43:0/1884894243 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62440a5c40 con 0x7f624c138cd0 2024-04-19T00:53:56.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6250d69640 10 monclient: dump: 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.036 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 10 monclient: shutdown 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 -- 172.21.15.43:0/1884894243 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 msgr2=0x7f624c1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 --2- 172.21.15.43:0/1884894243 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 0x7f624c1390b0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7f6244009e50 tx=0x7f624401c890 comp rx=0 tx=0).stop 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 -- 172.21.15.43:0/1884894243 shutdown_connections 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 --2- 172.21.15.43:0/1884894243 >> v2:172.21.15.161:3300/0 conn(0x7f624c1358b0 0x7f624c135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 --2- 172.21.15.43:0/1884894243 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 --2- 172.21.15.43:0/1884894243 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c138cd0 0x7f624c1390b0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 -- 172.21.15.43:0/1884894243 >> 172.21.15.43:0/1884894243 conn(0x7f624c0ab2f0 msgr2=0x7f624c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 -- 172.21.15.43:0/1884894243 shutdown_connections 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.018+0000 7f6253fcf640 1 -- 172.21.15.43:0/1884894243 wait complete. 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: monmap: 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:56.038 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 librados: starting msgr at 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 librados: starting objecter 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 Processor -- start 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- start start 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 librados: setting wanted keys 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 librados: calling monclient init 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: init 2024-04-19T00:53:56.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: _add_conns ranks=[3,8,6,0,1,2,5,7,4] 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c0a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): picked mon.noname-g con 0x7f624c0a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 0x7f624c134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): picked mon.noname-e con 0x7f624c1358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f624c138cd0 0x7f624c1351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): picked mon.noname-d con 0x7f624c138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f624c0a2010 con 0x7f624c0a5b80 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f624c0a1e90 con 0x7f624c138cd0 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f624c142aa0 con 0x7f624c1358b0 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): authenticate will time out at 1506.153084s 2024-04-19T00:53:56.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 0x7f624c134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f624c138cd0 0x7f624c1351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 0x7f624c134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54396/0 (socket says 172.21.15.43:54396) 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 -- 172.21.15.43:0/3213367834 learned_addr learned my addr 172.21.15.43:0/3213367834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): get_auth_request con 0x7f624c1358b0 auth_method 0 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): get_auth_request con 0x7f624c138cd0 auth_method 0 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c0a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): get_auth_request con 0x7f624c0a5b80 auth_method 0 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient(hunting): handle_auth_done global_id 23246 payload 824 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f624c138cd0 msgr2=0x7f624c1351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f624c138cd0 0x7f624c1351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 msgr2=0x7f624c0a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f624c0a5b80 0x7f624c0a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 monclient: found mon.noname-e 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624c0a06a0 con 0x7f624c1358b0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f624c138cd0 0x7f624c1351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625156a640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 0x7f624c134c80 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f623c00d7d0 tx=0x7f623c00dc90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 client.?.objecter ms_handle_connect 0x7f624c1358b0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f623c00bb30 con 0x7f624c1358b0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:56.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: dump: 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.044 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f623c00bcf0 con 0x7f624c1358b0 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 5 monclient: authenticate success, global_id 23246 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: _renew_subs 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f624c0a3040 con 0x7f624c1358b0 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f623c0148a0 con 0x7f624c1358b0 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6243fff640 4 set_mon_vals no callback set 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: dump: 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.046 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 client.23246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: _renew_subs 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f624c0a03c0 con 0x7f624c1358b0 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 librados: init done 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 0x7f624c09cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f624c0a0040 con 0x7f624c09c7b0 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f623c01b030 con 0x7f624c1358b0 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 0x7f624c09cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient: get_auth_request con 0x7f624c09c7b0 auth_method 0 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6230038e10 0x7f623003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(368..368 src has 1..368) ==== 6170+0+0 (secure 0 0 0) 0x7f623c019020 con 0x7f624c1358b0 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 client.23246.objecter ms_dispatch 0x7f624c000ba0 osd_map(368..368 src has 1..368) 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 3 client.23246.objecter handle_osd_map got epochs [368,368] > 0 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 3 client.23246.objecter handle_osd_map decoding full epoch 368 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 10 monclient(hunting): handle_auth_done global_id 74117 payload 824 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f624c135700 con 0x7f624c09c7b0 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 20 client.23246.objecter dump_active .. 0 homeless 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 client.23246.objecter ms_handle_connect 0x7f624c09c7b0 2024-04-19T00:53:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f6251d6b640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 0x7f624c09cb90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f624c0a4380 tx=0x7f6244019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 client.23246.objecter resend_mon_ops 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6230038e10 0x7f623003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6244003e10 con 0x7f624c09c7b0 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f62427fc640 10 monclient: dump: 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.022+0000 7f625256c640 10 monclient: get_auth_request con 0x7f6230038e10 auth_method 0 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f62440039a0 con 0x7f624c09c7b0 2024-04-19T00:53:56.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 msgr2=0x7f624c09cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 0x7f624c09cb90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f624c0a4380 tx=0x7f6244019690 comp rx=0 tx=0).stop 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 msgr2=0x7f624c09cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09c7b0 0x7f624c09cb90 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f624c0a4380 tx=0x7f6244019690 comp rx=0 tx=0).stop 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f625256c640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6230038e10 0x7f623003b2d0 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f624c139990 tx=0x7f6248008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.026+0000 7f62427fc640 10 client.23246.objecter ms_handle_connect 0x7f6230038e10 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 0x7f624c13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f624c09e340 con 0x7f624c09d0d0 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 0x7f624c13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient: get_auth_request con 0x7f624c09d0d0 auth_method 0 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 10 monclient(hunting): handle_auth_done global_id 74126 payload 824 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 1 -- 172.21.15.43:0/3213367834 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f624c0693b0 con 0x7f624c09d0d0 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6251d6b640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 0x7f624c13bcc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f624401c8d0 tx=0x7f62440a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 client.23246.objecter ms_handle_connect 0x7f624c09d0d0 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 client.23246.objecter resend_mon_ops 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f62440cc430 con 0x7f624c09d0d0 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 monclient: dump: 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f62440cc430 con 0x7f624c09d0d0 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 msgr2=0x7f624c13bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 0x7f624c13bcc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f624401c8d0 tx=0x7f62440a7000 comp rx=0 tx=0).stop 2024-04-19T00:53:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 msgr2=0x7f624c13bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f62427fc640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f624c09d0d0 0x7f624c13bcc0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f624401c8d0 tx=0x7f62440a7000 comp rx=0 tx=0).stop 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15651, 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:56.056 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.057 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.058 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.059 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.060 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:56.061 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:56.063 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.064 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.073 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.077 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:56.078 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 10 librados: watch_flush enter 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 10 librados: watch_flush exit 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 20 client.23246.objecter shutdown clearing up homeless session... 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 10 client.23246.objecter successfully canceled tick 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6230038e10 msgr2=0x7f623003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6230038e10 0x7f623003b2d0 secure :-1 s=READY pgs=3489 cs=0 l=1 rev1=1 crypto rx=0x7f624c139990 tx=0x7f6248008040 comp rx=0 tx=0).stop 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 10 monclient: shutdown 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 msgr2=0x7f624c134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 0x7f624c134c80 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f623c00d7d0 tx=0x7f623c00dc90 comp rx=0 tx=0).stop 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f625256c640 1 -- 172.21.15.43:0/3213367834 reap_dead start 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 shutdown_connections 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 --2- 172.21.15.43:0/3213367834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f624c1358b0 0x7f624c134c80 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 >> 172.21.15.43:0/3213367834 conn(0x7f624c0ab2f0 msgr2=0x7f624c1368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 shutdown_connections 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 -- 172.21.15.43:0/3213367834 wait complete. 2024-04-19T00:53:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.030+0000 7f6253fcf640 1 librados: shutdown 2024-04-19T00:53:56.081 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:56.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:53:56.082 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:53:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient: get_monmap_and_config 2024-04-19T00:53:56.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient: monmap: 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:56.199355+0000 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:56.199355+0000 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:56.197 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:56.198 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 Processor -- start 2024-04-19T00:53:56.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 -- start start 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient: init 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient: _add_conns ranks=[9,0,3,6,2,7,4,10,1,5,8] 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f53d0138cd0 0x7f53d01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): picked mon.noname-a con 0x7f53d0138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53d00a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d0135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): picked mon.noname-g con 0x7f53d01395f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d00a3dc0 con 0x7f53d00a5b80 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f53d00a4410 con 0x7f53d0138cd0 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f53d00a40a0 con 0x7f53d01395f0 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d6ac9640 10 monclient(hunting): authenticate will time out at 1506.327283s 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f53d0138cd0 0x7f53d01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cffff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f53d00a5b80 0x7f53d01358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37298/0 (socket says 172.21.15.43:37298) 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cffff640 1 -- 172.21.15.43:0/1595742578 learned_addr learned my addr 172.21.15.43:0/1595742578 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): get_auth_request con 0x7f53d0138cd0 auth_method 0 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d5066640 1 --2- 172.21.15.43:0/1595742578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d0135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d5066640 10 monclient(hunting): get_auth_request con 0x7f53d01395f0 auth_method 0 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d5066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d5066640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d5066640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cf7fe640 1 -- 172.21.15.43:0/1595742578 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1790520232 0 0) 0x7f53d00a3dc0 con 0x7f53d00a5b80 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cf7fe640 10 monclient(hunting): _negotiate 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cf7fe640 10 monclient(hunting): my global_id is 64449 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53cf7fe640 1 -- 172.21.15.43:0/1595742578 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53bc003530 con 0x7f53d00a5b80 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient(hunting): handle_auth_done global_id 64440 payload 274 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 1 -- 172.21.15.43:0/1595742578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 msgr2=0x7f53d0135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 1 --2- 172.21.15.43:0/1595742578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d0135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 1 -- 172.21.15.43:0/1595742578 >> v1:172.21.15.43:6789/0 conn(0x7f53d00a5b80 legacy=0x7f53d01358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 1 monclient: found mon.noname-a 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:56.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.194+0000 7f53d4865640 1 -- 172.21.15.43:0/1595742578 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d00a2ae0 con 0x7f53d0138cd0 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d4865640 1 --2- 172.21.15.43:0/1595742578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53d0138cd0 0x7f53d01390b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7f58f61b564c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 1 -- 172.21.15.43:0/1595742578 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c00b2070 con 0x7f53d0138cd0 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: dump: 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.212 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.213 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.214 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 1 -- 172.21.15.43:0/1595742578 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c00ab7f0 con 0x7f53d0138cd0 2024-04-19T00:53:56.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 1 -- 172.21.15.43:0/1595742578 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c00abe10 con 0x7f53d0138cd0 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53cf7fe640 10 monclient: dump: 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.215 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 5 monclient: authenticate success, global_id 64440 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 4 set_mon_vals no callback set 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 10 monclient: shutdown 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 -- 172.21.15.43:0/1595742578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53d0138cd0 msgr2=0x7f53d01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/1595742578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53d0138cd0 0x7f53d01390b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c001c360 comp rx=0 tx=0).stop 2024-04-19T00:53:56.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 -- 172.21.15.43:0/1595742578 shutdown_connections 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/1595742578 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d0135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/1595742578 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f53d0138cd0 0x7f53d01390b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 -- 172.21.15.43:0/1595742578 >> 172.21.15.43:0/1595742578 conn(0x7f53d00ab2f0 msgr2=0x7f53d00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 -- 172.21.15.43:0/1595742578 shutdown_connections 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 1 -- 172.21.15.43:0/1595742578 wait complete. 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.198+0000 7f53d6ac9640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: monmap: 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:56.217 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 librados: starting msgr at 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 librados: starting objecter 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 Processor -- start 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- start start 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 librados: setting wanted keys 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 librados: calling monclient init 2024-04-19T00:53:56.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: init 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: _add_conns ranks=[8,2,3,5,7,6,1,0,4] 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f53d00a5b80 0x7f53d012e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): picked mon.noname-e con 0x7f53d00a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53d0138cd0 0x7f53d012ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): picked mon.noname-b con 0x7f53d0138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d012f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): picked mon.noname-g con 0x7f53d01395f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f53d0136500 con 0x7f53d0138cd0 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f53d00a20a0 con 0x7f53d01395f0 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f53d0136680 con 0x7f53d00a5b80 2024-04-19T00:53:56.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): authenticate will time out at 1506.333285s 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f53d00a5b80 0x7f53d012e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53d0138cd0 0x7f53d012ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f53d00a5b80 0x7f53d012e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54406/0 (socket says 172.21.15.43:54406) 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 -- 172.21.15.43:0/3222912720 learned_addr learned my addr 172.21.15.43:0/3222912720 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): get_auth_request con 0x7f53d00a5b80 auth_method 0 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): get_auth_request con 0x7f53d0138cd0 auth_method 0 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient(hunting): handle_auth_done global_id 23255 payload 824 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d5066640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d012f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 msgr2=0x7f53d012f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f53d01395f0 0x7f53d012f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53d0138cd0 msgr2=0x7f53d012ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f53d0138cd0 0x7f53d012ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 monclient: found mon.noname-e 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d00a0070 con 0x7f53d00a5b80 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d4865640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f53d00a5b80 0x7f53d012e840 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c0019100 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 client.?.objecter ms_handle_connect 0x7f53d00a5b80 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c00ba020 con 0x7f53d00a5b80 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: dump: 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.222 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c00044b0 con 0x7f53d00a5b80 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 5 monclient: authenticate success, global_id 23255 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: _renew_subs 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d00a18d0 con 0x7f53d00a5b80 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53c00b2040 con 0x7f53d00a5b80 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: dump: 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.224 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 client.23255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: _renew_subs 2024-04-19T00:53:56.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d00a3040 con 0x7f53d00a5b80 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53ceffd640 4 set_mon_vals no callback set 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 librados: init done 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 0x7f53d009c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f53d006d140 con 0x7f53d00a0ad0 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 0x7f53d009c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient: get_auth_request con 0x7f53d00a0ad0 auth_method 0 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f53c00bd020 con 0x7f53d00a5b80 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 10 monclient(hunting): handle_auth_done global_id 64318 payload 824 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53d00a3590 con 0x7f53d00a0ad0 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f53b0038e00 0x7f53b003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(368..368 src has 1..368) ==== 6170+0+0 (secure 0 0 0) 0x7f53c00e2dd0 con 0x7f53d00a5b80 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 client.23255.objecter ms_dispatch 0x7f53d0000ba0 osd_map(368..368 src has 1..368) 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 3 client.23255.objecter handle_osd_map got epochs [368,368] > 0 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 3 client.23255.objecter handle_osd_map decoding full epoch 368 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 20 client.23255.objecter dump_active .. 0 homeless 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 client.23255.objecter ms_handle_connect 0x7f53d00a0ad0 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 client.23255.objecter resend_mon_ops 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cffff640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 0x7f53d009c490 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f53d009e660 tx=0x7f53b8001a00 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53b8005d30 con 0x7f53d00a0ad0 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53cd7fa640 10 monclient: dump: 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.228 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d5066640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f53b0038e00 0x7f53b003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.202+0000 7f53d5066640 10 monclient: get_auth_request con 0x7f53b0038e00 auth_method 0 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53b8018070 con 0x7f53d00a0ad0 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 msgr2=0x7f53d009c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 0x7f53d009c490 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f53d009e660 tx=0x7f53b8001a00 comp rx=0 tx=0).stop 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 msgr2=0x7f53d009c490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d00a0ad0 0x7f53d009c490 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f53d009e660 tx=0x7f53b8001a00 comp rx=0 tx=0).stop 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53cd7fa640 10 client.23255.objecter ms_handle_connect 0x7f53b0038e00 2024-04-19T00:53:56.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.206+0000 7f53d5066640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f53b0038e00 0x7f53b003b2c0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7f53d009e4d0 tx=0x7f53c800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 0x7f53d009cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f53d0068240 con 0x7f53d009c9d0 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 0x7f53d009cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient: get_auth_request con 0x7f53d009c9d0 auth_method 0 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 10 monclient(hunting): handle_auth_done global_id 64327 payload 824 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 1 -- 172.21.15.43:0/3222912720 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53d009d2f0 con 0x7f53d009c9d0 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cffff640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 0x7f53d009cdb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f53b8017000 tx=0x7f53b8005800 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 client.23255.objecter ms_handle_connect 0x7f53d009c9d0 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 client.23255.objecter resend_mon_ops 2024-04-19T00:53:56.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f53b80066d0 con 0x7f53d009c9d0 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 monclient: dump: 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.233 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f53b8006260 con 0x7f53d009c9d0 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 msgr2=0x7f53d009cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 0x7f53d009cdb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f53b8017000 tx=0x7f53b8005800 comp rx=0 tx=0).stop 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 msgr2=0x7f53d009cdb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53cd7fa640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f53d009c9d0 0x7f53d009cdb0 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f53b8017000 tx=0x7f53b8005800 comp rx=0 tx=0).stop 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 162482, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:56.235 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.237 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:56.238 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.239 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.241 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.246 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:56.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:56.255 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 10 librados: watch_flush enter 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 10 librados: watch_flush exit 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 20 client.23255.objecter shutdown clearing up homeless session... 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 10 client.23255.objecter successfully canceled tick 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f53b0038e00 msgr2=0x7f53b003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f53b0038e00 0x7f53b003b2c0 secure :-1 s=READY pgs=3490 cs=0 l=1 rev1=1 crypto rx=0x7f53d009e4d0 tx=0x7f53c800b040 comp rx=0 tx=0).stop 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 10 monclient: shutdown 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f53d00a5b80 msgr2=0x7f53d012e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 --2- 172.21.15.43:0/3222912720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f53d00a5b80 0x7f53d012e840 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c0019100 comp rx=0 tx=0).stop 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d5066640 1 -- 172.21.15.43:0/3222912720 reap_dead start 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d5066640 1 -- 172.21.15.43:0/3222912720 reap_dead start 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 shutdown_connections 2024-04-19T00:53:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 >> 172.21.15.43:0/3222912720 conn(0x7f53d00ab2f0 msgr2=0x7f53d0131b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 shutdown_connections 2024-04-19T00:53:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 -- 172.21.15.43:0/3222912720 wait complete. 2024-04-19T00:53:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.210+0000 7f53d6ac9640 1 librados: shutdown 2024-04-19T00:53:56.258 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:56.258 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:53:56.258 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:53:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient: get_monmap_and_config 2024-04-19T00:53:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient: monmap: 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:56.375515+0000 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:56.375515+0000 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.373 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:56.374 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:56.374 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 Processor -- start 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 -- start start 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient: init 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient: _add_conns ranks=[4,0,3,7,1,8,10,5,6,9,2] 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fce58138cd0 0x7fce581390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): picked mon.noname-c con 0x7fce58138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce580a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce58135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): picked mon.noname-g con 0x7fce581395f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce580a3dc0 con 0x7fce580a5b80 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fce580a4410 con 0x7fce581395f0 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fce580a40a0 con 0x7fce58138cd0 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5ec2c640 10 monclient(hunting): authenticate will time out at 1506.503183s 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fce58138cd0 0x7fce581390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce57fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fce580a5b80 0x7fce581358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37300/0 (socket says 172.21.15.43:37300) 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce57fff640 1 -- 172.21.15.43:0/1432306703 learned_addr learned my addr 172.21.15.43:0/1432306703 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): get_auth_request con 0x7fce58138cd0 auth_method 0 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5d1c9640 1 --2- 172.21.15.43:0/1432306703 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce58135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5d1c9640 10 monclient(hunting): get_auth_request con 0x7fce581395f0 auth_method 0 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5d1c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5d1c9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5d1c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce577fe640 1 -- 172.21.15.43:0/1432306703 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3524589201 0 0) 0x7fce580a3dc0 con 0x7fce580a5b80 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce577fe640 10 monclient(hunting): _negotiate 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce577fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce577fe640 10 monclient(hunting): my global_id is 64458 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce577fe640 1 -- 172.21.15.43:0/1432306703 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce48003530 con 0x7fce580a5b80 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_done global_id 64336 payload 274 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1432306703 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 msgr2=0x7fce58135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 1 --2- 172.21.15.43:0/1432306703 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce58135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1432306703 >> v1:172.21.15.43:6789/0 conn(0x7fce580a5b80 legacy=0x7fce581358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 1 monclient: found mon.noname-c 2024-04-19T00:53:56.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:53:56.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.370+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1432306703 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce58139a80 con 0x7fce58138cd0 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5c9c8640 1 --2- 172.21.15.43:0/1432306703 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fce58138cd0 0x7fce581390b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0097f0 tx=0x7fce4c01c340 comp rx=0 tx=0).ready entity=mon.4 client_cookie=873211e433a32e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 1 -- 172.21.15.43:0/1432306703 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fce4c0b0070 con 0x7fce58138cd0 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: dump: 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 5 monclient: authenticate success, global_id 64336 2024-04-19T00:53:56.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 1 -- 172.21.15.43:0/1432306703 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4c0aa810 con 0x7fce58138cd0 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 4 set_mon_vals no callback set 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 1 -- 172.21.15.43:0/1432306703 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fce4c0a9300 con 0x7fce58138cd0 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce577fe640 10 monclient: dump: 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.387 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.389 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 10 monclient: shutdown 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1432306703 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fce58138cd0 msgr2=0x7fce581390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1432306703 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fce58138cd0 0x7fce581390b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0097f0 tx=0x7fce4c01c340 comp rx=0 tx=0).stop 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1432306703 shutdown_connections 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1432306703 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce58135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1432306703 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fce58138cd0 0x7fce581390b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1432306703 >> 172.21.15.43:0/1432306703 conn(0x7fce580ab2f0 msgr2=0x7fce580ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1432306703 shutdown_connections 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1432306703 wait complete. 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 10 monclient: monmap: 2024-04-19T00:53:56.390 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 librados: starting msgr at 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.374+0000 7fce5ec2c640 1 librados: starting objecter 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 Processor -- start 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- start start 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 librados: setting wanted keys 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 librados: calling monclient init 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: init 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: _add_conns ranks=[6,1,3,7,0,4,8,2,5] 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce580a5b80 0x7fce580a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): picked mon.noname-d con 0x7fce580a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce58138cd0 0x7fce5812e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): picked mon.noname-f con 0x7fce58138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce5812ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): picked mon.noname-g con 0x7fce581395f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fce58136500 con 0x7fce58138cd0 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fce580a20a0 con 0x7fce581395f0 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fce58136680 con 0x7fce580a5b80 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): authenticate will time out at 1506.508679s 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce580a5b80 0x7fce580a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce580a5b80 0x7fce580a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33132/0 (socket says 172.21.15.43:33132) 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1473046313 learned_addr learned my addr 172.21.15.43:0/1473046313 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): get_auth_request con 0x7fce580a5b80 auth_method 0 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce58138cd0 0x7fce5812e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5d1c9640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce5812ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient(hunting): handle_auth_done global_id 74121 payload 824 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 msgr2=0x7fce5812ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fce581395f0 0x7fce5812ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce58138cd0 msgr2=0x7fce5812e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce58138cd0 0x7fce5812e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 monclient: found mon.noname-d 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce580a0090 con 0x7fce580a5b80 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient: get_auth_request con 0x7fce58138cd0 auth_method 0 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce58138cd0 0x7fce5812e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5c9c8640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce580a5b80 0x7fce580a0af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0ae040 tx=0x7fce4c019140 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 client.?.objecter ms_handle_connect 0x7fce580a5b80 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fce4c0b0040 con 0x7fce580a5b80 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: dump: 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.395 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4c0b9070 con 0x7fce580a5b80 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fce4c0044b0 con 0x7fce580a5b80 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 5 monclient: authenticate success, global_id 74121 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: _renew_subs 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5809e520 con 0x7fce580a5b80 2024-04-19T00:53:56.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce56ffd640 4 set_mon_vals no callback set 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: dump: 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.397 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 client.74121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: _renew_subs 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fce580a3040 con 0x7fce580a5b80 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 librados: init done 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 0x7fce5809c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fce5806d140 con 0x7fce5812f2c0 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 0x7fce5809c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient: get_auth_request con 0x7fce5812f2c0 auth_method 0 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fce4c0aaca0 con 0x7fce580a5b80 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fce3c038e10 0x7fce3c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 10 monclient(hunting): handle_auth_done global_id 74130 payload 824 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce5812f6a0 con 0x7fce5812f2c0 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(368..368 src has 1..368) ==== 6170+0+0 (secure 0 0 0) 0x7fce4c0e3010 con 0x7fce580a5b80 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 client.74121.objecter ms_dispatch 0x7fce58000ba0 osd_map(368..368 src has 1..368) 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 3 client.74121.objecter handle_osd_map got epochs [368,368] > 0 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 3 client.74121.objecter handle_osd_map decoding full epoch 368 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 20 client.74121.objecter dump_active .. 0 homeless 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 client.74121.objecter ms_handle_connect 0x7fce5812f2c0 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 client.74121.objecter resend_mon_ops 2024-04-19T00:53:56.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce57fff640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 0x7fce5809c490 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce44005890 tx=0x7fce4400db80 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5d1c9640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fce3c038e10 0x7fce3c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5d1c9640 10 monclient: get_auth_request con 0x7fce3c038e10 auth_method 0 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fce44005d30 con 0x7fce5812f2c0 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: dump: 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.400 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fce44017070 con 0x7fce5812f2c0 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 msgr2=0x7fce5809c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 0x7fce5809c490 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce44005890 tx=0x7fce4400db80 comp rx=0 tx=0).stop 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 msgr2=0x7fce5809c490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5812f2c0 0x7fce5809c490 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce44005890 tx=0x7fce4400db80 comp rx=0 tx=0).stop 2024-04-19T00:53:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce557fa640 10 client.74121.objecter ms_handle_connect 0x7fce3c038e10 2024-04-19T00:53:56.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.378+0000 7fce5d1c9640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fce3c038e10 0x7fce3c03b2d0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7fce5809dbb0 tx=0x7fce5000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 0x7fce5809cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fce58068240 con 0x7fce5809c9d0 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 0x7fce5809cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient: get_auth_request con 0x7fce5809c9d0 auth_method 0 2024-04-19T00:53:56.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 10 monclient(hunting): handle_auth_done global_id 74139 payload 824 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 1 -- 172.21.15.43:0/1473046313 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fce5809d320 con 0x7fce5809c9d0 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 client.74121.objecter ms_handle_connect 0x7fce5809c9d0 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 client.74121.objecter resend_mon_ops 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce57fff640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 0x7fce5809cde0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce44016000 tx=0x7fce44005800 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fce4401ce10 con 0x7fce5809c9d0 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 monclient: dump: 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fce44007280 con 0x7fce5809c9d0 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:56.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 msgr2=0x7fce5809cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 0x7fce5809cde0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce44016000 tx=0x7fce44005800 comp rx=0 tx=0).stop 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 msgr2=0x7fce5809cde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce557fa640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce5809c9d0 0x7fce5809cde0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce44016000 tx=0x7fce44005800 comp rx=0 tx=0).stop 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15999, 2024-04-19T00:53:56.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.407 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.408 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:56.409 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.410 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:56.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.415 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:56.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.417 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:56.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.423 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:56.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:56.426 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 10 librados: watch_flush enter 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 10 librados: watch_flush exit 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 20 client.74121.objecter shutdown clearing up homeless session... 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 10 client.74121.objecter successfully canceled tick 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fce3c038e10 msgr2=0x7fce3c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fce3c038e10 0x7fce3c03b2d0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7fce5809dbb0 tx=0x7fce5000b040 comp rx=0 tx=0).stop 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 10 monclient: shutdown 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce580a5b80 msgr2=0x7fce580a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 --2- 172.21.15.43:0/1473046313 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fce580a5b80 0x7fce580a0af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0ae040 tx=0x7fce4c019140 comp rx=0 tx=0).stop 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5d1c9640 1 -- 172.21.15.43:0/1473046313 reap_dead start 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5d1c9640 1 -- 172.21.15.43:0/1473046313 reap_dead start 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 shutdown_connections 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 >> 172.21.15.43:0/1473046313 conn(0x7fce580ab2f0 msgr2=0x7fce581368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 shutdown_connections 2024-04-19T00:53:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 -- 172.21.15.43:0/1473046313 wait complete. 2024-04-19T00:53:56.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.386+0000 7fce5ec2c640 1 librados: shutdown 2024-04-19T00:53:56.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:56.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:53:56.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:53:56.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.550+0000 7f4ae89c4640 10 monclient: get_monmap_and_config 2024-04-19T00:53:56.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.550+0000 7f4ae89c4640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.550+0000 7f4ae89c4640 10 monclient: monmap: 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:56.558314+0000 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:56.558314+0000 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.556 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.557 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.557 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.557 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:56.557 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:56.557 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 Processor -- start 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 -- start start 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient: init 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient: _add_conns ranks=[6,8,4,9,0,1,7,3,2,10,5] 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4ae4138cd0 0x7f4ae41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): picked mon.noname-d con 0x7f4ae4138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae41395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): picked mon.noname-e con 0x7f4ae40a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae41358b0 0x7f4ae4135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ae41358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:56.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4ae40a4410 con 0x7f4ae41358b0 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f4ae4139b30 con 0x7f4ae4138cd0 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4ae40a3dc0 con 0x7f4ae40a5b80 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae89c4640 10 monclient(hunting): authenticate will time out at 1506.685407s 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4ae4138cd0 0x7f4ae41390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae41395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae41395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54418/0 (socket says 172.21.15.43:54418) 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4ae4138cd0 0x7f4ae41390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33166/0 (socket says 172.21.15.43:33166) 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae41358b0 0x7f4ae4135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 -- 172.21.15.43:0/452102058 learned_addr learned my addr 172.21.15.43:0/452102058 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): get_auth_request con 0x7f4ae40a5b80 auth_method 0 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): get_auth_request con 0x7f4ae41358b0 auth_method 0 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): get_auth_request con 0x7f4ae4138cd0 auth_method 0 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_done global_id 23264 payload 274 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 -- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4ae4138cd0 msgr2=0x7f4ae41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4ae4138cd0 0x7f4ae41390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 -- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae41358b0 msgr2=0x7f4ae4135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae41358b0 0x7f4ae4135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 monclient: found mon.noname-e 2024-04-19T00:53:56.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.554+0000 7f4ae1d9b640 1 -- 172.21.15.43:0/452102058 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae40a40a0 con 0x7f4ae40a5b80 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae41395f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80098a0 tx=0x7f4ad801c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a1a9d2042191742c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 1 -- 172.21.15.43:0/452102058 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ad80b0070 con 0x7f4ae40a5b80 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: dump: 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.564 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 1 -- 172.21.15.43:0/452102058 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad80aa810 con 0x7f4ae40a5b80 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 1 -- 172.21.15.43:0/452102058 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ad80a9300 con 0x7f4ae40a5b80 2024-04-19T00:53:56.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 5 monclient: authenticate success, global_id 23264 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 4 set_mon_vals no callback set 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae159a640 10 monclient: dump: 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.566 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.567 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.567 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: shutdown 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 msgr2=0x7f4ae41395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae41395f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80098a0 tx=0x7f4ad801c3e0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- 172.21.15.43:0/452102058 shutdown_connections 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae41358b0 0x7f4ae4135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae41395f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/452102058 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4ae4138cd0 0x7f4ae41390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- 172.21.15.43:0/452102058 >> 172.21.15.43:0/452102058 conn(0x7f4ae40ab2f0 msgr2=0x7f4ae40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- 172.21.15.43:0/452102058 shutdown_connections 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- 172.21.15.43:0/452102058 wait complete. 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: monmap: 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.576 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 librados: starting msgr at 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 librados: starting objecter 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 Processor -- start 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- start start 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 librados: setting wanted keys 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 librados: calling monclient init 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: init 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient: _add_conns ranks=[8,2,4,3,1,0,5,6,7] 2024-04-19T00:53:56.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae4134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): picked mon.noname-e con 0x7f4ae40a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4ae41358b0 0x7f4ae4135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): picked mon.noname-b con 0x7f4ae41358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae4138cd0 0x7f4ae41d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): picked mon.noname-c con 0x7f4ae4138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4ae40a2010 con 0x7f4ae41358b0 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4ae40a1e90 con 0x7f4ae4138cd0 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4ae4142aa0 con 0x7f4ae40a5b80 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae89c4640 10 monclient(hunting): authenticate will time out at 1506.691017s 2024-04-19T00:53:56.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae4134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae4138cd0 0x7f4ae41d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae4134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54426/0 (socket says 172.21.15.43:54426) 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 1 -- 172.21.15.43:0/3482473503 learned_addr learned my addr 172.21.15.43:0/3482473503 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): get_auth_request con 0x7f4ae40a5b80 auth_method 0 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): get_auth_request con 0x7f4ae4138cd0 auth_method 0 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4ae41358b0 0x7f4ae4135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): get_auth_request con 0x7f4ae41358b0 auth_method 0 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient(hunting): handle_auth_done global_id 64354 payload 824 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 msgr2=0x7f4ae4134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae4134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4ae41358b0 msgr2=0x7f4ae4135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4ae41358b0 0x7f4ae4135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 monclient: found mon.noname-c 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ae409e2c0 con 0x7f4ae4138cd0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae259c640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae40a5b80 0x7f4ae4134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ae2d9d640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae4138cd0 0x7f4ae41d0aa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00efc0 tx=0x7f4adc00c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ad2ffd640 10 client.?.objecter ms_handle_connect 0x7f4ae4138cd0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ad2ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4adc018020 con 0x7f4ae4138cd0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.558+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: dump: 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.581 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4adc00cd30 con 0x7f4ae4138cd0 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4adc014d60 con 0x7f4ae4138cd0 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 5 monclient: authenticate success, global_id 64354 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae0d99640 4 set_mon_vals no callback set 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: dump: 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.583 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 monclient: _renew_subs 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ae40a3040 con 0x7f4ae4138cd0 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 client.64354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 monclient: _renew_subs 2024-04-19T00:53:56.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ae409fff0 con 0x7f4ae4138cd0 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 1 librados: init done 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 0x7f4ae41d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4ae406d140 con 0x7f4ae41d0fe0 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 0x7f4ae41d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient: get_auth_request con 0x7f4ae41d0fe0 auth_method 0 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4adc01b020 con 0x7f4ae4138cd0 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 10 monclient(hunting): handle_auth_done global_id 23282 payload 824 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ae41d1900 con 0x7f4ae41d0fe0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ac4038e10 0x7f4ac403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(368..368 src has 1..368) ==== 6170+0+0 (secure 0 0 0) 0x7f4adc04a140 con 0x7f4ae4138cd0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 client.64354.objecter ms_dispatch 0x7f4ae4000ba0 osd_map(368..368 src has 1..368) 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 3 client.64354.objecter handle_osd_map got epochs [368,368] > 0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 3 client.64354.objecter handle_osd_map decoding full epoch 368 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 20 client.64354.objecter dump_active .. 0 homeless 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 client.64354.objecter ms_handle_connect 0x7f4ae41d0fe0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 client.64354.objecter resend_mon_ops 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae259c640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 0x7f4ae41d13c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011d20 tx=0x7f4ad400f610 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ad40148b0 con 0x7f4ae41d0fe0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: dump: 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.587 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae1d9b640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ac4038e10 0x7f4ac403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae1d9b640 10 monclient: get_auth_request con 0x7f4ac4038e10 auth_method 0 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ad4014440 con 0x7f4ae41d0fe0 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 msgr2=0x7f4ae41d13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 0x7f4ae41d13c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011d20 tx=0x7f4ad400f610 comp rx=0 tx=0).stop 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 msgr2=0x7f4ae41d13c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae41d0fe0 0x7f4ae41d13c0 secure :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011d20 tx=0x7f4ad400f610 comp rx=0 tx=0).stop 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ad2ffd640 10 client.64354.objecter ms_handle_connect 0x7f4ac4038e10 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.562+0000 7f4ae1d9b640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ac4038e10 0x7f4ac403b2d0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80ad000 tx=0x7f4ad80a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 0x7f4ae413c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4ae4068240 con 0x7f4ae413bcf0 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 0x7f4ae413c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient: get_auth_request con 0x7f4ae413bcf0 auth_method 0 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 10 monclient(hunting): handle_auth_done global_id 23291 payload 824 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 1 -- 172.21.15.43:0/3482473503 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ae40693b0 con 0x7f4ae413bcf0 2024-04-19T00:53:56.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 client.64354.objecter ms_handle_connect 0x7f4ae413bcf0 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 client.64354.objecter resend_mon_ops 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae259c640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 0x7f4ae413c100 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011d20 tx=0x7f4ad40046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ad40273f0 con 0x7f4ae413bcf0 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 monclient: dump: 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.590 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7f4ad40273f0 con 0x7f4ae413bcf0 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 msgr2=0x7f4ae413c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 0x7f4ae413c100 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011d20 tx=0x7f4ad40046d0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 msgr2=0x7f4ae413c100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ad2ffd640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4ae413bcf0 0x7f4ae413c100 secure :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f4ad4011d20 tx=0x7f4ad40046d0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.593 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.593 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 580082, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:56.594 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.595 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.596 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:56.597 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.598 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.601 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.603 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.604 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.607 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.608 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.614 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:56.614 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 10 librados: watch_flush enter 2024-04-19T00:53:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 10 librados: watch_flush exit 2024-04-19T00:53:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 20 client.64354.objecter shutdown clearing up homeless session... 2024-04-19T00:53:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 10 client.64354.objecter successfully canceled tick 2024-04-19T00:53:56.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ac4038e10 msgr2=0x7f4ac403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4ac4038e10 0x7f4ac403b2d0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f4ad80ad000 tx=0x7f4ad80a4f90 comp rx=0 tx=0).stop 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 10 monclient: shutdown 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae4138cd0 msgr2=0x7f4ae41d0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 --2- 172.21.15.43:0/3482473503 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4ae4138cd0 0x7f4ae41d0aa0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00efc0 tx=0x7f4adc00c490 comp rx=0 tx=0).stop 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae2d9d640 1 -- 172.21.15.43:0/3482473503 reap_dead start 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae2d9d640 1 -- 172.21.15.43:0/3482473503 reap_dead start 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 shutdown_connections 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 >> 172.21.15.43:0/3482473503 conn(0x7f4ae40ab2f0 msgr2=0x7f4ae41368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 shutdown_connections 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 -- 172.21.15.43:0/3482473503 wait complete. 2024-04-19T00:53:56.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.570+0000 7f4ae89c4640 1 librados: shutdown 2024-04-19T00:53:56.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:56.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:53:56.617 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:53:56.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient: get_monmap_and_config 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient: monmap: 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:56.742808+0000 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:56.742808+0000 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.740 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:56.741 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:56.741 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 Processor -- start 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 -- start start 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient: init 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient: _add_conns ranks=[5,8,1,6,3,7,10,9,2,0,4] 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f770c138cd0 0x7f770c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): picked mon.noname-h con 0x7f770c138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 0x7f770c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): picked mon.noname-e con 0x7f770c0a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f770c1358b0 addr v1:172.21.15.161:6789/0 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f770c0a3dc0 con 0x7f770c1358b0 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f770c0a4410 con 0x7f770c138cd0 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f770c139b30 con 0x7f770c0a5b80 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7713ff4640 10 monclient(hunting): authenticate will time out at 1506.870032s 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 0x7f770c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 0x7f770c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54456/0 (socket says 172.21.15.43:54456) 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 -- 172.21.15.43:0/3683308490 learned_addr learned my addr 172.21.15.43:0/3683308490 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): get_auth_request con 0x7f770c0a5b80 auth_method 0 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7711d90640 1 --2- 172.21.15.43:0/3683308490 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f770c138cd0 0x7f770c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7711d90640 10 monclient(hunting): get_auth_request con 0x7f770c138cd0 auth_method 0 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7711d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7711d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f7711d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient(hunting): handle_auth_done global_id 23300 payload 274 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 -- 172.21.15.43:0/3683308490 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f770c138cd0 msgr2=0x7f770c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 --2- 172.21.15.43:0/3683308490 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f770c138cd0 0x7f770c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 -- 172.21.15.43:0/3683308490 >> v1:172.21.15.161:6789/0 conn(0x7f770c1358b0 legacy=0x7f770c135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 monclient: found mon.noname-e 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:56.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.738+0000 7f771158f640 1 -- 172.21.15.43:0/3683308490 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c0a40a0 con 0x7f770c0a5b80 2024-04-19T00:53:56.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 --2- 172.21.15.43:0/3683308490 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 0x7f770c1395f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0098a0 tx=0x7f76fc01c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=ac555c04fec5e668 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 1 -- 172.21.15.43:0/3683308490 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76fc0b1070 con 0x7f770c0a5b80 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: dump: 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.752 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 5 monclient: authenticate success, global_id 23300 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 1 -- 172.21.15.43:0/3683308490 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76fc0ab830 con 0x7f770c0a5b80 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 4 set_mon_vals no callback set 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 1 -- 172.21.15.43:0/3683308490 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76fc0aa320 con 0x7f770c0a5b80 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:56.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7710d8e640 10 monclient: dump: 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.754 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: shutdown 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- 172.21.15.43:0/3683308490 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 msgr2=0x7f770c1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 --2- 172.21.15.43:0/3683308490 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 0x7f770c1395f0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0098a0 tx=0x7f76fc01c3e0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- 172.21.15.43:0/3683308490 shutdown_connections 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 --2- 172.21.15.43:0/3683308490 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f770c0a5b80 0x7f770c1395f0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 --2- 172.21.15.43:0/3683308490 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f770c138cd0 0x7f770c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- 172.21.15.43:0/3683308490 >> 172.21.15.43:0/3683308490 conn(0x7f770c0ab2f0 msgr2=0x7f770c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- 172.21.15.43:0/3683308490 shutdown_connections 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- 172.21.15.43:0/3683308490 wait complete. 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: monmap: 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:56.755 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 librados: starting msgr at 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 librados: starting objecter 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 Processor -- start 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- start start 2024-04-19T00:53:56.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 librados: setting wanted keys 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 librados: calling monclient init 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: init 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient: _add_conns ranks=[7,6,1,0,8,2,3,4,5] 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f770c0a5b80 0x7f770c1d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): picked mon.noname-i con 0x7f770c0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f770c1358b0 0x7f770c0a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): picked mon.noname-d con 0x7f770c1358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c138cd0 0x7f770c0a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): picked mon.noname-f con 0x7f770c138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f770c143ab0 con 0x7f770c138cd0 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f770c0a20a0 con 0x7f770c1358b0 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f770c143c30 con 0x7f770c0a5b80 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7713ff4640 10 monclient(hunting): authenticate will time out at 1506.875475s 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f770c1358b0 0x7f770c0a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f770c1358b0 0x7f770c0a0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33180/0 (socket says 172.21.15.43:33180) 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 -- 172.21.15.43:0/2235345941 learned_addr learned my addr 172.21.15.43:0/2235345941 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): get_auth_request con 0x7f770c1358b0 auth_method 0 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7711d90640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f770c0a5b80 0x7f770c1d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7711d90640 10 monclient(hunting): get_auth_request con 0x7f770c0a5b80 auth_method 0 2024-04-19T00:53:56.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7711d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7711d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7711d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7712591640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c138cd0 0x7f770c0a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7712591640 10 monclient(hunting): get_auth_request con 0x7f770c138cd0 auth_method 0 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7712591640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7712591640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f7712591640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient(hunting): handle_auth_done global_id 74157 payload 824 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f770c0a5b80 msgr2=0x7f770c1d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f770c0a5b80 0x7f770c1d0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c138cd0 msgr2=0x7f770c0a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c138cd0 0x7f770c0a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 monclient: found mon.noname-d 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.742+0000 7f771158f640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c09c1f0 con 0x7f770c1358b0 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f771158f640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f770c1358b0 0x7f770c0a0a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0097c0 tx=0x7f76fc0191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 client.?.objecter ms_handle_connect 0x7f770c1358b0 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76fc0ba070 con 0x7f770c1358b0 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: dump: 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.760 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76fc0044b0 con 0x7f770c1358b0 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f76fc0b1040 con 0x7f770c1358b0 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:56.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7703fff640 4 set_mon_vals no callback set 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: dump: 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.762 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 5 monclient: authenticate success, global_id 74157 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 monclient: _renew_subs 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c134d30 con 0x7f770c1358b0 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 client.74157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 monclient: _renew_subs 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c133fb0 con 0x7f770c1358b0 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 1 librados: init done 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 0x7f770c1328f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f770c06d140 con 0x7f770c132510 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f76fc004040 con 0x7f770c1358b0 2024-04-19T00:53:56.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f76f0038e10 0x7f76f003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 0x7f770c1328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(368..368 src has 1..368) ==== 6170+0+0 (secure 0 0 0) 0x7f76fc0e25e0 con 0x7f770c1358b0 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 client.74157.objecter ms_dispatch 0x7f770c000ba0 osd_map(368..368 src has 1..368) 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 3 client.74157.objecter handle_osd_map got epochs [368,368] > 0 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 3 client.74157.objecter handle_osd_map decoding full epoch 368 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient: get_auth_request con 0x7f770c132510 auth_method 0 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 20 client.74157.objecter dump_active .. 0 homeless 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7712591640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f76f0038e10 0x7f76f003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7712591640 10 monclient: get_auth_request con 0x7f76f0038e10 auth_method 0 2024-04-19T00:53:56.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 10 monclient(hunting): handle_auth_done global_id 64342 payload 824 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f770c0a3040 con 0x7f770c132510 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 client.74157.objecter ms_handle_connect 0x7f770c132510 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 client.74157.objecter resend_mon_ops 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7711d90640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 0x7f770c1328f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77080045e0 tx=0x7f7708007330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 client.74157.objecter ms_handle_connect 0x7f76f0038e10 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f7712591640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f76f0038e10 0x7f76f003b2d0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f770c133b20 tx=0x7f7704009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7708007db0 con 0x7f770c132510 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: dump: 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.765 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7708019070 con 0x7f770c132510 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 msgr2=0x7f770c1328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 0x7f770c1328f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77080045e0 tx=0x7f7708007330 comp rx=0 tx=0).stop 2024-04-19T00:53:56.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 msgr2=0x7f770c1328f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.746+0000 7f77027fc640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132510 0x7f770c1328f0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f77080045e0 tx=0x7f7708007330 comp rx=0 tx=0).stop 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7713ff4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7713ff4640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 0x7f770c12bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7713ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f770c068240 con 0x7f770c132e30 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 0x7f770c12bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient: get_auth_request con 0x7f770c132e30 auth_method 0 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 10 monclient(hunting): handle_auth_done global_id 64351 payload 824 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 1 -- 172.21.15.43:0/2235345941 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f770c09bc30 con 0x7f770c132e30 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f77027fc640 10 client.74157.objecter ms_handle_connect 0x7f770c132e30 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f7711d90640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 0x7f770c12bbb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f77080045e0 tx=0x7f7708004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f77027fc640 10 client.74157.objecter resend_mon_ops 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7708014d90 con 0x7f770c132e30 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f77027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f77027fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.754+0000 7f77027fc640 10 monclient: dump: 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.769 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f7708014920 con 0x7f770c132e30 2024-04-19T00:53:56.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 msgr2=0x7f770c12bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 0x7f770c12bbb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f77080045e0 tx=0x7f7708004a50 comp rx=0 tx=0).stop 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 msgr2=0x7f770c12bbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f77027fc640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f770c132e30 0x7f770c12bbb0 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f77080045e0 tx=0x7f7708004a50 comp rx=0 tx=0).stop 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:56.771 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 163025, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.772 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:56.773 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:56.774 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.775 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:56.776 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.779 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.780 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:56.781 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.782 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.783 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.784 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.785 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.786 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.787 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.788 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.789 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.790 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:56.791 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 10 librados: watch_flush enter 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 10 librados: watch_flush exit 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 20 client.74157.objecter shutdown clearing up homeless session... 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 10 client.74157.objecter successfully canceled tick 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f76f0038e10 msgr2=0x7f76f003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f76f0038e10 0x7f76f003b2d0 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f770c133b20 tx=0x7f7704009040 comp rx=0 tx=0).stop 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 10 monclient: shutdown 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f770c1358b0 msgr2=0x7f770c0a0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 --2- 172.21.15.43:0/2235345941 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f770c1358b0 0x7f770c0a0a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76fc0097c0 tx=0x7f76fc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7712591640 1 -- 172.21.15.43:0/2235345941 reap_dead start 2024-04-19T00:53:56.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7712591640 1 -- 172.21.15.43:0/2235345941 reap_dead start 2024-04-19T00:53:56.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 shutdown_connections 2024-04-19T00:53:56.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 >> 172.21.15.43:0/2235345941 conn(0x7f770c0ab2f0 msgr2=0x7f770c0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 shutdown_connections 2024-04-19T00:53:56.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 -- 172.21.15.43:0/2235345941 wait complete. 2024-04-19T00:53:56.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.758+0000 7f7713ff4640 1 librados: shutdown 2024-04-19T00:53:56.794 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:56.794 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:53:56.794 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient: get_monmap_and_config 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient: monmap: 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:56.914716+0000 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:56.914716+0000 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:56.912 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:56.913 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 Processor -- start 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 -- start start 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient: init 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient: _add_conns ranks=[0,2,9,6,10,7,3,8,4,5,1] 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8850138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8850139540 0x7f885013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): picked mon.noname-b con 0x7f8850139540 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f88500a5bb0 0x7f8850140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): picked mon.noname-a con 0x7f88500a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88500a0d40 con 0x7f8850138a50 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f88500a1020 con 0x7f88500a5bb0 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f88500a0780 con 0x7f8850139540 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f885532e640 10 monclient(hunting): authenticate will time out at 1507.041751s 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f88500a5bb0 0x7f8850140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8850139540 0x7f885013dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884effd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f8850138a50 0x7f8850138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37306/0 (socket says 172.21.15.43:37306) 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884effd640 1 -- 172.21.15.43:0/2990247466 learned_addr learned my addr 172.21.15.43:0/2990247466 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): get_auth_request con 0x7f88500a5bb0 auth_method 0 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): get_auth_request con 0x7f8850139540 auth_method 0 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient(hunting): handle_auth_done global_id 64467 payload 274 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 1 -- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8850139540 msgr2=0x7f885013dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 1 --2- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8850139540 0x7f885013dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 1 -- 172.21.15.43:0/2990247466 >> v1:172.21.15.43:6789/0 conn(0x7f8850138a50 legacy=0x7f8850138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 1 monclient: found mon.noname-a 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884dffb640 1 -- 172.21.15.43:0/2990247466 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836205260 0 0) 0x7f88500a0d40 con 0x7f8850138a50 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884f7fe640 1 -- 172.21.15.43:0/2990247466 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88500a0a60 con 0x7f88500a5bb0 2024-04-19T00:53:56.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.910+0000 7f884e7fc640 1 --2- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8850139540 0x7f885013dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 --2- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88500a5bb0 0x7f8850140530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8848009dc0 tx=0x7f884801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ea6a70a83a7ffcf server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 1 -- 172.21.15.43:0/2990247466 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88480b2070 con 0x7f88500a5bb0 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: dump: 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.920 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 1 -- 172.21.15.43:0/2990247466 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88480a5a50 con 0x7f88500a5bb0 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 5 monclient: authenticate success, global_id 64467 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 1 -- 172.21.15.43:0/2990247466 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88480a5c40 con 0x7f88500a5bb0 2024-04-19T00:53:56.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 4 set_mon_vals no callback set 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884dffb640 10 monclient: dump: 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.922 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: shutdown 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88500a5bb0 msgr2=0x7f8850140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 --2- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88500a5bb0 0x7f8850140530 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8848009dc0 tx=0x7f884801c810 comp rx=0 tx=0).stop 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2990247466 shutdown_connections 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 --2- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f88500a5bb0 0x7f8850140530 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 --2- 172.21.15.43:0/2990247466 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8850139540 0x7f885013dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2990247466 >> 172.21.15.43:0/2990247466 conn(0x7f88500a7cb0 msgr2=0x7f88500a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2990247466 shutdown_connections 2024-04-19T00:53:56.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2990247466 wait complete. 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: build_initial_monmap 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: monmap: 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:56.930 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 librados: starting msgr at 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 librados: starting objecter 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 Processor -- start 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- start start 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 librados: setting wanted keys 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 librados: calling monclient init 2024-04-19T00:53:56.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: init 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: _add_conns ranks=[4,3,0,1,6,5,8,7,2] 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f88500a5bb0 0x7f8850137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): picked mon.noname-c con 0x7f88500a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8850138a50 0x7f885013c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): picked mon.noname-g con 0x7f8850138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8850139540 0x7f885013cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): picked mon.noname-a con 0x7f8850139540 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8850143620 con 0x7f8850139540 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f88501434a0 con 0x7f8850138a50 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f88501437a0 con 0x7f88500a5bb0 2024-04-19T00:53:56.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient(hunting): authenticate will time out at 1507.046792s 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f88500a5bb0 0x7f8850137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f88500a5bb0 0x7f8850137800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44940/0 (socket says 172.21.15.43:44940) 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8850139540 0x7f885013cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 1 -- 172.21.15.43:0/2535787531 learned_addr learned my addr 172.21.15.43:0/2535787531 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): get_auth_request con 0x7f88500a5bb0 auth_method 0 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): get_auth_request con 0x7f8850139540 auth_method 0 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient(hunting): handle_auth_done global_id 64485 payload 824 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f88500a5bb0 msgr2=0x7f8850137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f88500a5bb0 0x7f8850137800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8850138a50 msgr2=0x7f885013c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8850138a50 0x7f885013c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 monclient: found mon.noname-a 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:56.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8850122190 con 0x7f8850139540 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884effd640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f88500a5bb0 0x7f8850137800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884f7fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8850139540 0x7f885013cad0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88500a11a0 tx=0x7f88480195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 client.?.objecter ms_handle_connect 0x7f8850139540 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88480b2050 con 0x7f8850139540 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: dump: 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.935 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: _finish_auth 0 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: _check_auth_tickets 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88480b3040 con 0x7f8850139540 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 5 monclient: authenticate success, global_id 64485 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: _renew_subs 2024-04-19T00:53:56.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f885013edc0 con 0x7f8850139540 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8848003bb0 con 0x7f8850139540 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 client.64485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f88377fe640 10 monclient: dump: 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.937 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f884d7fa640 4 set_mon_vals no callback set 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: _renew_subs 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f885009ee60 con 0x7f8850139540 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.914+0000 7f885532e640 1 librados: init done 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f885532e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f885532e640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 0x7f885002c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f885006d140 con 0x7f885002c450 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f88480c0030 con 0x7f8850139540 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f883c038e10 0x7f883c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 0x7f885002c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(369..369 src has 1..369) ==== 6170+0+0 (secure 0 0 0) 0x7f88480be020 con 0x7f8850139540 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 client.64485.objecter ms_dispatch 0x7f8850000ba0 osd_map(369..369 src has 1..369) 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 3 client.64485.objecter handle_osd_map got epochs [369,369] > 0 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 3 client.64485.objecter handle_osd_map decoding full epoch 369 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient: get_auth_request con 0x7f885002c450 auth_method 0 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 20 client.64485.objecter dump_active .. 0 homeless 2024-04-19T00:53:56.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884e7fc640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f883c038e10 0x7f883c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884e7fc640 10 monclient: get_auth_request con 0x7f883c038e10 auth_method 0 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 10 monclient(hunting): handle_auth_done global_id 19227 payload 824 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f885009e190 con 0x7f885002c450 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884effd640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 0x7f885002c830 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f8844004a20 tx=0x7f8844004ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 client.64485.objecter ms_handle_connect 0x7f885002c450 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 client.64485.objecter resend_mon_ops 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 client.64485.objecter ms_handle_connect 0x7f883c038e10 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f884e7fc640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f883c038e10 0x7f883c03b2d0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f8840006e90 tx=0x7f8840006de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f88440133f0 con 0x7f885002c450 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:53:56.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 monclient: dump: 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.941 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8844018070 con 0x7f885002c450 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 msgr2=0x7f885002c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 0x7f885002c830 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f8844004a20 tx=0x7f8844004ee0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 msgr2=0x7f885002c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.918+0000 7f88377fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002c450 0x7f885002c830 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 0x7f885002d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f88501d14f0 con 0x7f885002cd70 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 0x7f885002d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient: get_auth_request con 0x7f885002cd70 auth_method 0 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:56.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 10 monclient(hunting): handle_auth_done global_id 19236 payload 824 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 1 -- 172.21.15.43:0/2535787531 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f885013eb10 con 0x7f885002cd70 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884effd640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 0x7f885002d180 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f8844004a20 tx=0x7f8844000ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 client.64485.objecter ms_handle_connect 0x7f885002cd70 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 client.64485.objecter resend_mon_ops 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8844013dc0 con 0x7f885002cd70 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 monclient: dump: 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:56.943 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:56.944 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f884401d030 con 0x7f885002cd70 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 msgr2=0x7f885002d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 0x7f885002d180 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f8844004a20 tx=0x7f8844000ee0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 msgr2=0x7f885002d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:56.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f88377fe640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f885002cd70 0x7f885002d180 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 580414, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:56.946 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.947 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.948 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:56.949 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:56.950 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.952 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.954 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:56.957 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:56.961 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.962 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.963 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:56.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:56.965 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:56.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:56.966 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:56.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:56.966 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:56.966 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 10 librados: watch_flush enter 2024-04-19T00:53:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 10 librados: watch_flush exit 2024-04-19T00:53:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 20 client.64485.objecter shutdown clearing up homeless session... 2024-04-19T00:53:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 10 client.64485.objecter successfully canceled tick 2024-04-19T00:53:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f883c038e10 msgr2=0x7f883c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f883c038e10 0x7f883c03b2d0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f8840006e90 tx=0x7f8840006de0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 10 monclient: shutdown 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8850139540 msgr2=0x7f885013cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 --2- 172.21.15.43:0/2535787531 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8850139540 0x7f885013cad0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f88500a11a0 tx=0x7f88480195b0 comp rx=0 tx=0).stop 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884f7fe640 1 -- 172.21.15.43:0/2535787531 reap_dead start 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f884f7fe640 1 -- 172.21.15.43:0/2535787531 reap_dead start 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 shutdown_connections 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 >> 172.21.15.43:0/2535787531 conn(0x7f88500a7cb0 msgr2=0x7f88500a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 shutdown_connections 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 -- 172.21.15.43:0/2535787531 wait complete. 2024-04-19T00:53:56.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:56.926+0000 7f885532e640 1 librados: shutdown 2024-04-19T00:53:56.968 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:56.968 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:53:56.969 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:53:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient: get_monmap_and_config 2024-04-19T00:53:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient: build_initial_monmap 2024-04-19T00:53:57.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient: monmap: 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:57.083519+0000 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:57.083519+0000 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:57.082 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 Processor -- start 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 -- start start 2024-04-19T00:53:57.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient: init 2024-04-19T00:53:57.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:57.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient: _add_conns ranks=[2,9,8,7,10,1,0,5,3,6,4] 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 0x7f7f88138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): picked mon.noname-b con 0x7f7f88138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7f880a5bb0 0x7f7f88139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): picked mon.noname-a con 0x7f7f880a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7f88137960 0x7f7f88137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): picked mon.noname-e con 0x7f7f88137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f7f880a1020 con 0x7f7f880a5bb0 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7f881398b0 con 0x7f7f88138a50 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7f880a0780 con 0x7f7f88137960 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:57.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8f70f640 10 monclient(hunting): authenticate will time out at 1507.210620s 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 0x7f7f88138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7f88137960 0x7f7f88137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7f880a5bb0 0x7f7f88139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7f88137960 0x7f7f88137de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54470/0 (socket says 172.21.15.43:54470) 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 0x7f7f88138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40662/0 (socket says 172.21.15.43:40662) 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 1 -- 172.21.15.43:0/3138068559 learned_addr learned my addr 172.21.15.43:0/3138068559 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): get_auth_request con 0x7f7f88137960 auth_method 0 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): get_auth_request con 0x7f7f880a5bb0 auth_method 0 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): get_auth_request con 0x7f7f88138a50 auth_method 0 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_done global_id 74162 payload 274 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 -- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7f88137960 msgr2=0x7f7f88137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:57.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7f88137960 0x7f7f88137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 -- 172.21.15.43:0/3138068559 >> v2:172.21.15.43:3300/0 conn(0x7f7f880a5bb0 msgr2=0x7f7f88139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/3138068559 >> v2:172.21.15.43:3300/0 conn(0x7f7f880a5bb0 0x7f7f88139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient: _finish_hunting 0 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 monclient: found mon.noname-b 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8d4ab640 1 -- 172.21.15.43:0/3138068559 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f880a0d40 con 0x7f7f88138a50 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.078+0000 7f7f8ccaa640 1 --2- 172.21.15.43:0/3138068559 >> v2:172.21.15.43:3300/0 conn(0x7f7f880a5bb0 0x7f7f88139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 0x7f7f88138e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f840097c0 tx=0x7f7f8401c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7ac658b6fcf1a21 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 1 -- 172.21.15.43:0/3138068559 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f840b0070 con 0x7f7f88138a50 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: dump: 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.095 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: _finish_auth 0 2024-04-19T00:53:57.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 1 -- 172.21.15.43:0/3138068559 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f840aa810 con 0x7f7f88138a50 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 5 monclient: authenticate success, global_id 74162 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 1 -- 172.21.15.43:0/3138068559 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f840a9300 con 0x7f7f88138a50 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 4 set_mon_vals no callback set 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f7ffff640 10 monclient: dump: 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.097 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: shutdown 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 msgr2=0x7f7f88138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 0x7f7f88138e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7f840097c0 tx=0x7f7f8401c330 comp rx=0 tx=0).stop 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- 172.21.15.43:0/3138068559 shutdown_connections 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7f88137960 0x7f7f88137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/3138068559 >> v2:172.21.15.43:3300/0 conn(0x7f7f880a5bb0 0x7f7f88139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/3138068559 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f88138a50 0x7f7f88138e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- 172.21.15.43:0/3138068559 >> 172.21.15.43:0/3138068559 conn(0x7f7f880a7cb0 msgr2=0x7f7f880a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- 172.21.15.43:0/3138068559 shutdown_connections 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- 172.21.15.43:0/3138068559 wait complete. 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: build_initial_monmap 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: monmap: 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:57.099 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 librados: starting msgr at 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 librados: starting objecter 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 Processor -- start 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- start start 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 librados: setting wanted keys 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 librados: calling monclient init 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: init 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:57.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient: _add_conns ranks=[2,5,0,1,4,3,7,6,8] 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f880a5bb0 0x7f7f88136980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): picked mon.noname-b con 0x7f7f880a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88137960 0x7f7f8813c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): picked mon.noname-h con 0x7f7f88137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 0x7f7f8813cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): picked mon.noname-a con 0x7f7f88138a50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7f880a2100 con 0x7f7f88138a50 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7f880a1f80 con 0x7f7f880a5bb0 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7f88142810 con 0x7f7f88137960 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:57.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8f70f640 10 monclient(hunting): authenticate will time out at 1507.215475s 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f880a5bb0 0x7f7f88136980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 0x7f7f8813cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f880a5bb0 0x7f7f88136980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40666/0 (socket says 172.21.15.43:40666) 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 0x7f7f8813cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42844/0 (socket says 172.21.15.43:42844) 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 1 -- 172.21.15.43:0/652966852 learned_addr learned my addr 172.21.15.43:0/652966852 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): get_auth_request con 0x7f7f880a5bb0 auth_method 0 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): get_auth_request con 0x7f7f88138a50 auth_method 0 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient(hunting): handle_auth_done global_id 64503 payload 824 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88137960 msgr2=0x7f7f8813c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88137960 0x7f7f8813c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f880a5bb0 msgr2=0x7f7f88136980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f880a5bb0 0x7f7f88136980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient: _finish_hunting 0 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 monclient: found mon.noname-a 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8dcac640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f88122040 con 0x7f7f88138a50 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.082+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7f880a5bb0 0x7f7f88136980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:57.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8dcac640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 0x7f7f8813cce0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000efc0 tx=0x7f7f8000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 client.?.objecter ms_handle_connect 0x7f7f88138a50 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f800039a0 con 0x7f7f88138a50 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: dump: 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.104 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: _finish_auth 0 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f80003b60 con 0x7f7f88138a50 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f80003d10 con 0x7f7f88138a50 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 5 monclient: authenticate success, global_id 64503 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 monclient: _renew_subs 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:57.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f88122320 con 0x7f7f88138a50 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7f7fe640 4 set_mon_vals no callback set 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: dump: 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.106 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 client.64503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 monclient: _renew_subs 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f88137720 con 0x7f7f88138a50 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 1 librados: init done 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 0x7f7f8813d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7f8806d140 con 0x7f7f881ce920 2024-04-19T00:53:57.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7f80012050 con 0x7f7f88138a50 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f6c038e10 0x7f7f6c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(369..369 src has 1..369) ==== 6170+0+0 (secure 0 0 0) 0x7f7f80049800 con 0x7f7f88138a50 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 client.64503.objecter ms_dispatch 0x7f7f88000ba0 osd_map(369..369 src has 1..369) 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 3 client.64503.objecter handle_osd_map got epochs [369,369] > 0 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 3 client.64503.objecter handle_osd_map decoding full epoch 369 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 20 client.64503.objecter dump_active .. 0 homeless 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 0x7f7f8813d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient: get_auth_request con 0x7f7f881ce920 auth_method 0 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8ccaa640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f6c038e10 0x7f7f6c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8ccaa640 10 monclient: get_auth_request con 0x7f7f6c038e10 auth_method 0 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_done global_id 74111 payload 824 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f881d17e0 con 0x7f7f881ce920 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 client.64503.objecter ms_handle_connect 0x7f7f881ce920 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 0x7f7f8813d220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f840097c0 tx=0x7f7f840190d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 client.64503.objecter resend_mon_ops 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 client.64503.objecter ms_handle_connect 0x7f7f6c038e10 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f8ccaa640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f6c038e10 0x7f7f6c03b2d0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f7f78006e90 tx=0x7f7f78006de0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f840b0040 con 0x7f7f881ce920 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: dump: 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.109 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f840b1040 con 0x7f7f881ce920 2024-04-19T00:53:57.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:57.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 msgr2=0x7f7f8813d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 0x7f7f8813d220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f840097c0 tx=0x7f7f840190d0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 msgr2=0x7f7f8813d220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:57.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.086+0000 7f7f7dffb640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f881ce920 0x7f7f8813d220 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7f840097c0 tx=0x7f7f840190d0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8f70f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 0x7f7f8813d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8f70f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7f88076190 con 0x7f7f88136ec0 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 0x7f7f8813d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient: get_auth_request con 0x7f7f88136ec0 auth_method 0 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 10 monclient(hunting): handle_auth_done global_id 74120 payload 824 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 1 -- 172.21.15.43:0/652966852 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f88121c20 con 0x7f7f88136ec0 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f7dffb640 10 client.64503.objecter ms_handle_connect 0x7f7f88136ec0 2024-04-19T00:53:57.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f7dffb640 10 client.64503.objecter resend_mon_ops 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f8d4ab640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 0x7f7f8813d760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f8401c160 tx=0x7f7f840a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f840b0040 con 0x7f7f88136ec0 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f7dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.094+0000 7f7f7dffb640 10 monclient: dump: 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.114 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f7f84003d60 con 0x7f7f88136ec0 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 msgr2=0x7f7f8813d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 0x7f7f8813d760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f8401c160 tx=0x7f7f840a8000 comp rx=0 tx=0).stop 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 msgr2=0x7f7f8813d760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:57.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f7dffb640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7f88136ec0 0x7f7f8813d760 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7f8401c160 tx=0x7f7f840a8000 comp rx=0 tx=0).stop 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16715, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:57.116 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:57.117 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:57.118 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:57.119 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.120 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:57.122 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:57.123 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:57.126 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.127 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.129 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.132 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:57.133 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.134 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:57.135 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 10 librados: watch_flush enter 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 10 librados: watch_flush exit 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 20 client.64503.objecter shutdown clearing up homeless session... 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 10 client.64503.objecter successfully canceled tick 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f6c038e10 msgr2=0x7f7f6c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f6c038e10 0x7f7f6c03b2d0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f7f78006e90 tx=0x7f7f78006de0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 10 monclient: shutdown 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 msgr2=0x7f7f8813cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8dcac640 1 -- 172.21.15.43:0/652966852 reap_dead start 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 0x7f7f8813cce0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7f8000efc0 tx=0x7f7f8000c490 comp rx=0 tx=0).stop 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 shutdown_connections 2024-04-19T00:53:57.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 --2- 172.21.15.43:0/652966852 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7f88138a50 0x7f7f8813cce0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 >> 172.21.15.43:0/652966852 conn(0x7f7f880a7cb0 msgr2=0x7f7f8809ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 shutdown_connections 2024-04-19T00:53:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 -- 172.21.15.43:0/652966852 wait complete. 2024-04-19T00:53:57.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.098+0000 7f7f8f70f640 1 librados: shutdown 2024-04-19T00:53:57.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:57.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:53:57.138 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient: get_monmap_and_config 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient: build_initial_monmap 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient: monmap: 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:57.250927+0000 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:57.250927+0000 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:57.248 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:57.249 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:57.249 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 Processor -- start 2024-04-19T00:53:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 -- start start 2024-04-19T00:53:57.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient: init 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient: _add_conns ranks=[4,10,0,9,6,1,3,2,7,5,8] 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 0x7f95741390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): picked mon.noname-c con 0x7f9574138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95740a5b80 0x7f95741395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): picked mon.noname-f con 0x7f95740a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95741358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95740a3dc0 con 0x7f95741358b0 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f95740a4410 con 0x7f95740a5b80 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9574139b30 con 0x7f9574138cd0 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f957a065640 10 monclient(hunting): authenticate will time out at 1507.377992s 2024-04-19T00:53:57.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 0x7f95741390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 0x7f95741390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44954/0 (socket says 172.21.15.43:44954) 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f9573fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f95741358b0 0x7f9574135d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37322/0 (socket says 172.21.15.43:37322) 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 -- 172.21.15.43:0/334675072 learned_addr learned my addr 172.21.15.43:0/334675072 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): get_auth_request con 0x7f9574138cd0 auth_method 0 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f956bfff640 1 --2- 172.21.15.43:0/334675072 >> v2:172.21.15.161:3300/0 conn(0x7f95740a5b80 0x7f95741395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient(hunting): handle_auth_done global_id 64372 payload 274 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 -- 172.21.15.43:0/334675072 >> v2:172.21.15.161:3300/0 conn(0x7f95740a5b80 msgr2=0x7f95741395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 --2- 172.21.15.43:0/334675072 >> v2:172.21.15.161:3300/0 conn(0x7f95740a5b80 0x7f95741395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 -- 172.21.15.43:0/334675072 >> v1:172.21.15.43:6789/0 conn(0x7f95741358b0 legacy=0x7f9574135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient: _finish_hunting 0 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 monclient: found mon.noname-c 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f9572ffd640 1 -- 172.21.15.43:0/334675072 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732379974 0 0) 0x7f95740a3dc0 con 0x7f95741358b0 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f95737fe640 1 -- 172.21.15.43:0/334675072 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95740a40a0 con 0x7f9574138cd0 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f956bfff640 10 monclient: get_auth_request con 0x7f95740a5b80 auth_method 0 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f956bfff640 1 --2- 172.21.15.43:0/334675072 >> v2:172.21.15.161:3300/0 conn(0x7f95740a5b80 0x7f95741395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:53:57.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.246+0000 7f9572ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:53:57.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f95737fe640 1 --2- 172.21.15.43:0/334675072 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 0x7f95741390b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f955c009870 tx=0x7f955c01c3c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=74ec1ba99e697bac server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 1 -- 172.21.15.43:0/334675072 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f955c0b0070 con 0x7f9574138cd0 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: dump: 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.261 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: _finish_auth 0 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 1 -- 172.21.15.43:0/334675072 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f955c0aa810 con 0x7f9574138cd0 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 1 -- 172.21.15.43:0/334675072 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f955c0a9300 con 0x7f9574138cd0 2024-04-19T00:53:57.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 5 monclient: authenticate success, global_id 64372 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 4 set_mon_vals no callback set 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9572ffd640 10 monclient: dump: 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.263 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: shutdown 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- 172.21.15.43:0/334675072 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 msgr2=0x7f95741390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 --2- 172.21.15.43:0/334675072 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 0x7f95741390b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f955c009870 tx=0x7f955c01c3c0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- 172.21.15.43:0/334675072 shutdown_connections 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 --2- 172.21.15.43:0/334675072 >> v2:172.21.15.161:3300/0 conn(0x7f95740a5b80 0x7f95741395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 --2- 172.21.15.43:0/334675072 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9574138cd0 0x7f95741390b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- 172.21.15.43:0/334675072 >> 172.21.15.43:0/334675072 conn(0x7f95740ab2f0 msgr2=0x7f95740ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- 172.21.15.43:0/334675072 shutdown_connections 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- 172.21.15.43:0/334675072 wait complete. 2024-04-19T00:53:57.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: build_initial_monmap 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: monmap: 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:57.265 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 librados: starting msgr at 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 librados: starting objecter 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 Processor -- start 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- start start 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 librados: setting wanted keys 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 librados: calling monclient init 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: init 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: _add_conns ranks=[7,4,5,1,6,0,2,8,3] 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f95740a5b80 0x7f9574132280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): picked mon.noname-i con 0x7f95740a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 0x7f95741327c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): picked mon.noname-c con 0x7f95741358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:57.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9574138cd0 0x7f9574132d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): picked mon.noname-h con 0x7f9574138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9574143ab0 con 0x7f95741358b0 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f95740a20a0 con 0x7f9574138cd0 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9574143c30 con 0x7f95740a5b80 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient(hunting): authenticate will time out at 1507.383000s 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 0x7f95741327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 0x7f95741327c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44970/0 (socket says 172.21.15.43:44970) 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 -- 172.21.15.43:0/174842768 learned_addr learned my addr 172.21.15.43:0/174842768 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): get_auth_request con 0x7f95741358b0 auth_method 0 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9573fff640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9574138cd0 0x7f9574132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f95737fe640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f95740a5b80 0x7f9574132280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9573fff640 10 monclient(hunting): get_auth_request con 0x7f9574138cd0 auth_method 0 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9573fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9573fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9573fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f95737fe640 10 monclient(hunting): get_auth_request con 0x7f95740a5b80 auth_method 0 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f95737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f95737fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f95737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient(hunting): handle_auth_done global_id 64381 payload 824 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f95740a5b80 msgr2=0x7f9574132280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f95740a5b80 0x7f9574132280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9574138cd0 msgr2=0x7f9574132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9574138cd0 0x7f9574132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 monclient: found mon.noname-c 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f957409c030 con 0x7f95741358b0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f956bfff640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 0x7f95741327c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f956400c7f0 tx=0x7f956400ccb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 client.?.objecter ms_handle_connect 0x7f95741358b0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9564015070 con 0x7f95741358b0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: dump: 2024-04-19T00:53:57.269 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.270 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: _finish_auth 0 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9564007ca0 con 0x7f95741358b0 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 5 monclient: authenticate success, global_id 64381 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9564007e50 con 0x7f95741358b0 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9571ffb640 4 set_mon_vals no callback set 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f9570ff9640 10 monclient: dump: 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.271 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: _renew_subs 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.250+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95741342f0 con 0x7f95741358b0 2024-04-19T00:53:57.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 10 client.64381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 10 monclient: _renew_subs 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f957409c310 con 0x7f95741358b0 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 1 librados: init done 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 0x7f957412abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f957409cef0 con 0x7f957412a800 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9564018070 con 0x7f95741358b0 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9554038fb0 0x7f955403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 0x7f957412abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(369..369 src has 1..369) ==== 6170+0+0 (secure 0 0 0) 0x7f9564048fd0 con 0x7f95741358b0 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 client.64381.objecter ms_dispatch 0x7f9574000ba0 osd_map(369..369 src has 1..369) 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 3 client.64381.objecter handle_osd_map got epochs [369,369] > 0 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 3 client.64381.objecter handle_osd_map decoding full epoch 369 2024-04-19T00:53:57.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient: get_auth_request con 0x7f957412a800 auth_method 0 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 20 client.64381.objecter dump_active .. 0 homeless 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9573fff640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9554038fb0 0x7f955403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9573fff640 10 monclient: get_auth_request con 0x7f9554038fb0 auth_method 0 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 10 monclient(hunting): handle_auth_done global_id 74140 payload 824 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95740a3590 con 0x7f957412a800 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 client.64381.objecter ms_handle_connect 0x7f957412a800 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 client.64381.objecter resend_mon_ops 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f95737fe640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 0x7f957412abe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95740a3520 tx=0x7f955c0191c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 client.64381.objecter ms_handle_connect 0x7f9554038fb0 2024-04-19T00:53:57.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9573fff640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9554038fb0 0x7f955403b470 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f956c0012f0 tx=0x7f956c007e50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f955c0b9070 con 0x7f957412a800 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 monclient: dump: 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.275 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f955c0b1040 con 0x7f957412a800 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 msgr2=0x7f957412abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 0x7f957412abe0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95740a3520 tx=0x7f955c0191c0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 msgr2=0x7f957412abe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:57.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.254+0000 7f9570ff9640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412a800 0x7f957412abe0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95740a3520 tx=0x7f955c0191c0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 0x7f9574137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f957406d140 con 0x7f957412b120 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 0x7f9574137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient: get_auth_request con 0x7f957412b120 auth_method 0 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 10 monclient(hunting): handle_auth_done global_id 74149 payload 824 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 1 -- 172.21.15.43:0/174842768 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f95740a3040 con 0x7f957412b120 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f95737fe640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 0x7f9574137850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f955c009870 tx=0x7f955c0a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 client.64381.objecter ms_handle_connect 0x7f957412b120 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 client.64381.objecter resend_mon_ops 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f955c0b0050 con 0x7f957412b120 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 monclient: dump: 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.279 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f955c0c7410 con 0x7f957412b120 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 msgr2=0x7f9574137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 0x7f9574137850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f955c009870 tx=0x7f955c0a8000 comp rx=0 tx=0).stop 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 msgr2=0x7f9574137850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:57.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f9570ff9640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f957412b120 0x7f9574137850 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f955c009870 tx=0x7f955c0a8000 comp rx=0 tx=0).stop 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16883, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:57.281 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:57.282 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:57.283 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:57.284 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:57.285 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.288 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:57.289 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:57.291 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:57.294 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.297 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.298 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:57.299 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:57.300 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:57.301 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:57.301 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 10 librados: watch_flush enter 2024-04-19T00:53:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 10 librados: watch_flush exit 2024-04-19T00:53:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 20 client.64381.objecter shutdown clearing up homeless session... 2024-04-19T00:53:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.262+0000 7f957a065640 10 client.64381.objecter successfully canceled tick 2024-04-19T00:53:57.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9554038fb0 msgr2=0x7f955403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9554038fb0 0x7f955403b470 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7f956c0012f0 tx=0x7f956c007e50 comp rx=0 tx=0).stop 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 10 monclient: shutdown 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 msgr2=0x7f95741327c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 0x7f95741327c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f956400c7f0 tx=0x7f956400ccb0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f9573fff640 1 -- 172.21.15.43:0/174842768 reap_dead start 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 shutdown_connections 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 --2- 172.21.15.43:0/174842768 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95741358b0 0x7f95741327c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 >> 172.21.15.43:0/174842768 conn(0x7f95740ab2f0 msgr2=0x7f95740a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 shutdown_connections 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 -- 172.21.15.43:0/174842768 wait complete. 2024-04-19T00:53:57.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.266+0000 7f957a065640 1 librados: shutdown 2024-04-19T00:53:57.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:57.304 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T00:53:57.304 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient: get_monmap_and_config 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient: build_initial_monmap 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient: monmap: 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:57.416989+0000 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:57.416989+0000 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:57.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 Processor -- start 2024-04-19T00:53:57.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 -- start start 2024-04-19T00:53:57.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient: init 2024-04-19T00:53:57.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:57.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient: _add_conns ranks=[3,1,6,0,9,7,2,5,10,8,4] 2024-04-19T00:53:57.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3d24135c80 0x7f3d24073000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): picked mon.noname-g con 0x7f3d24135c80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f3d24136060 addr v1:172.21.15.161:6789/0 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 0x7f3d240a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): picked mon.noname-d con 0x7f3d241364c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d24071df0 con 0x7f3d24136060 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3d24072060 con 0x7f3d24135c80 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3d24072340 con 0x7f3d241364c0 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.410+0000 7f3d2bc8d640 10 monclient(hunting): authenticate will time out at 1507.543864s 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 0x7f3d240a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 0x7f3d240a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33190/0 (socket says 172.21.15.43:33190) 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 -- 172.21.15.43:0/3637105981 learned_addr learned my addr 172.21.15.43:0/3637105981 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:57.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): get_auth_request con 0x7f3d241364c0 auth_method 0 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d29a29640 1 --2- 172.21.15.43:0/3637105981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3d24135c80 0x7f3d24073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d29a29640 10 monclient(hunting): get_auth_request con 0x7f3d24135c80 auth_method 0 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d29a29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d29a29640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d29a29640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient(hunting): handle_auth_done global_id 74166 payload 274 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 -- 172.21.15.43:0/3637105981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3d24135c80 msgr2=0x7f3d24073000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 --2- 172.21.15.43:0/3637105981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3d24135c80 0x7f3d24073000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 -- 172.21.15.43:0/3637105981 >> v1:172.21.15.161:6789/0 conn(0x7f3d24136060 legacy=0x7f3d240a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient: _finish_hunting 0 2024-04-19T00:53:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 monclient: found mon.noname-d 2024-04-19T00:53:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 -- 172.21.15.43:0/3637105981 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d24072620 con 0x7f3d241364c0 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2a22a640 1 --2- 172.21.15.43:0/3637105981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 0x7f3d240a6f10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0098a0 tx=0x7f3d1c01ce60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=1a43dd2cd1cc529a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 1 -- 172.21.15.43:0/3637105981 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d1c0ab890 con 0x7f3d241364c0 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: dump: 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.425 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: _finish_auth 0 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: _check_auth_tickets 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 5 monclient: authenticate success, global_id 74166 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 1 -- 172.21.15.43:0/3637105981 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0b0070 con 0x7f3d241364c0 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:57.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 4 set_mon_vals no callback set 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 1 -- 172.21.15.43:0/3637105981 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d1c0aa380 con 0x7f3d241364c0 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d28a27640 10 monclient: dump: 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.427 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 10 monclient: shutdown 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3637105981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 msgr2=0x7f3d240a6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3637105981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 0x7f3d240a6f10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0098a0 tx=0x7f3d1c01ce60 comp rx=0 tx=0).stop 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3637105981 shutdown_connections 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3637105981 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d241364c0 0x7f3d240a6f10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3637105981 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3d24135c80 0x7f3d24073000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3637105981 >> 172.21.15.43:0/3637105981 conn(0x7f3d240a7ef0 msgr2=0x7f3d240a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3637105981 shutdown_connections 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.414+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3637105981 wait complete. 2024-04-19T00:53:57.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: build_initial_monmap 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: monmap: 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:57.429 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 librados: starting msgr at 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 librados: starting objecter 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 Processor -- start 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- start start 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 librados: setting wanted keys 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 librados: calling monclient init 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: init 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _add_conns ranks=[6,8,7,5,4,1,2,3,0] 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 0x7f3d241d0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): picked mon.noname-d con 0x7f3d24135c80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3d24136060 0x7f3d241d1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): picked mon.noname-e con 0x7f3d24136060 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3d241364c0 0x7f3d2413eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): picked mon.noname-i con 0x7f3d241364c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3d2413e350 con 0x7f3d24135c80 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3d240a7660 con 0x7f3d241364c0 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f3d2413e4d0 con 0x7f3d24136060 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient(hunting): authenticate will time out at 1507.548900s 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 0x7f3d241d0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3d24136060 0x7f3d241d1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 0x7f3d241d0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33206/0 (socket says 172.21.15.43:33206) 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 -- 172.21.15.43:0/3454369428 learned_addr learned my addr 172.21.15.43:0/3454369428 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): get_auth_request con 0x7f3d24135c80 auth_method 0 2024-04-19T00:53:57.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): get_auth_request con 0x7f3d24136060 auth_method 0 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2a22a640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3d241364c0 0x7f3d2413eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2a22a640 10 monclient(hunting): get_auth_request con 0x7f3d241364c0 auth_method 0 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2a22a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:57.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2a22a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2a22a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient(hunting): handle_auth_done global_id 74175 payload 824 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 -- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3d24136060 msgr2=0x7f3d241d1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3d24136060 0x7f3d241d1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 -- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3d241364c0 msgr2=0x7f3d2413eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3d241364c0 0x7f3d2413eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient: _finish_hunting 0 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 monclient: found mon.noname-d 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 -- 172.21.15.43:0/3454369428 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d240a17b0 con 0x7f3d24135c80 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2a22a640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3d241364c0 0x7f3d2413eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29a29640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 0x7f3d241d0d10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d20007ec0 tx=0x7f3d2000e4b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 client.?.objecter ms_handle_connect 0x7f3d24135c80 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:57.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d20014890 con 0x7f3d24135c80 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: dump: 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.434 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d20019070 con 0x7f3d24135c80 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3d20013380 con 0x7f3d24135c80 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 monclient: dump: 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:57.435 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:57.436 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1bfff640 4 set_mon_vals no callback set 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 5 monclient: authenticate success, global_id 74175 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _renew_subs 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2409dbe0 con 0x7f3d24135c80 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 client.74175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _renew_subs 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2409d200 con 0x7f3d24135c80 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 librados: init done 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3d20014420 con 0x7f3d24135c80 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d24130060 con 0x7f3d24135c80 2024-04-19T00:53:57.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3d08038e10 0x7f3d0803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(369..369 src has 1..369) ==== 6170+0+0 (secure 0 0 0) 0x7f3d20049cd0 con 0x7f3d24135c80 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 10 client.74175.objecter ms_dispatch 0x7f3d24000ba0 osd_map(369..369 src has 1..369) 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 3 client.74175.objecter handle_osd_map got epochs [369,369] > 0 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 3 client.74175.objecter handle_osd_map decoding full epoch 369 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d1a7fc640 20 client.74175.objecter dump_active .. 0 homeless 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3d08038e10 0x7f3d0803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.418+0000 7f3d29228640 10 monclient: get_auth_request con 0x7f3d08038e10 auth_method 0 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.422+0000 7f3d29228640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3d08038e10 0x7f3d0803b2d0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f3d240a2790 tx=0x7f3d14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.422+0000 7f3d1a7fc640 10 client.74175.objecter ms_handle_connect 0x7f3d08038e10 2024-04-19T00:53:57.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.426+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3d2409dbe0 con 0x7f3d24135c80 2024-04-19T00:53:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.426+0000 7f3d1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.426+0000 7f3d1a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:57.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.650+0000 7f3d2bc8d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T00:53:57.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.650+0000 7f3d2bc8d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T00:53:57.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.650+0000 7f3d2bc8d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:57.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.650+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f3d240713e0 con 0x7f3d24135c80 2024-04-19T00:53:57.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f3d1a7fc640 1 -- 172.21.15.43:0/3454369428 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3d2001c360 con 0x7f3d24135c80 2024-04-19T00:53:57.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f3d1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T00:53:57.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f3d1a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=1335873913,config=2586702516,health=3513953949,kv=4171600807,logm=520099848}) 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=17,config=2,health=12,kv=10,logm=59} crc {auth=988758593,config=2586702516,health=3524627081,kv=1840489377,logm=1821110508}) 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=2033056945,config=2586702516,health=3524627081,kv=4196696685,logm=1243391347}) 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.660 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=17,config=2,health=11,kv=12,logm=58} crc {auth=1898122683,config=2586702516,health=2253595315,kv=4196696685,logm=844808373}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=1214147196,config=2586702516,health=3524627081,kv=4196696685,logm=2704742153}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=4099626174,config=2586702516,health=3524627081,kv=4196696685,logm=3600375460}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=2033056945,config=2586702516,health=3524627081,kv=4196696685,logm=3816563599}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=988758593,config=2586702516,health=3524627081,kv=4196696685,logm=2060380512}) 2024-04-19T00:53:57.661 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.654+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=2033056945,config=2586702516,health=2822996232,kv=271807280,logm=2993550512}) 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 10 librados: watch_flush enter 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 10 librados: watch_flush exit 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 20 client.74175.objecter shutdown clearing up homeless session... 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 10 client.74175.objecter successfully canceled tick 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3d08038e10 msgr2=0x7f3d0803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3d08038e10 0x7f3d0803b2d0 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7f3d240a2790 tx=0x7f3d14009040 comp rx=0 tx=0).stop 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 10 monclient: shutdown 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 msgr2=0x7f3d241d0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 0x7f3d241d0d10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d20007ec0 tx=0x7f3d2000e4b0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 shutdown_connections 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3d241364c0 0x7f3d2413eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3d08038e10 0x7f3d0803b2d0 unknown :-1 s=CLOSED pgs=3497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3d24136060 0x7f3d241d1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 --2- 172.21.15.43:0/3454369428 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3d24135c80 0x7f3d241d0d10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:57.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 >> 172.21.15.43:0/3454369428 conn(0x7f3d240a7ef0 msgr2=0x7f3d2412a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:57.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 shutdown_connections 2024-04-19T00:53:57.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 -- 172.21.15.43:0/3454369428 wait complete. 2024-04-19T00:53:57.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:57.658+0000 7f3d2bc8d640 1 librados: shutdown 2024-04-19T00:53:57.666 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.666 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.666 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2434481088}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2804581321}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136596613}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1761684388}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=408334347}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.667 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3129567816}) 2024-04-19T00:53:57.668 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.668 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.668 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3293883032}) 2024-04-19T00:53:57.668 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.668 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1794754545}) 2024-04-19T00:53:57.668 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.662+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4180960741}) 2024-04-19T00:53:57.671 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427047041}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1888716524}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738118546}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1475861514}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1528950042}) 2024-04-19T00:53:57.672 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2032353265}) 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1051616101}) 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3668991057}) 2024-04-19T00:53:57.673 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.666+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3139889886}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.670+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.670+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348311874}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2675096447}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707297432}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=206126839}) 2024-04-19T00:53:57.677 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4287870507}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2092469142}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=855258050}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=704411654}) 2024-04-19T00:53:57.678 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.674+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1017936723}) 2024-04-19T00:53:57.681 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.681 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4274543141}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747396787}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209478484}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=506040148}) 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.682 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3412586980}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1473232115}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3919659154}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3730261620}) 2024-04-19T00:53:57.683 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.678+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=385237805}) 2024-04-19T00:53:57.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2542024820}) 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690524735}) 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.686 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386117849}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3902213486}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1324593553}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1339779735}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1973614087}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3360475813}) 2024-04-19T00:53:57.687 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.682+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=792318023}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607798393}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345179435}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760131731}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.691 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=669637661}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=760118959}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4224978027}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1436848760}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378772807}) 2024-04-19T00:53:57.692 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.686+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3739246873}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036397070}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169391998}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556669912}) 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.695 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1412307527}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1543845955}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2707560846}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1623320837}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4281716084}) 2024-04-19T00:53:57.696 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.690+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1201777299}) 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1573559117}) 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4285023792}) 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.699 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590721506}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2934510454}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2461600855}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1325465200}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1819285199}) 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.700 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1645418411}) 2024-04-19T00:53:57.701 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.694+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2622684972}) 2024-04-19T00:53:57.703 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.703 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.703 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103747949}) 2024-04-19T00:53:57.703 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.703 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.703 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=310116872}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577354741}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3411794135}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1011133330}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2119777003}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.704 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4030862982}) 2024-04-19T00:53:57.705 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.705 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3397729131}) 2024-04-19T00:53:57.705 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.698+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=597467548}) 2024-04-19T00:53:57.707 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.707 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.707 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851176838}) 2024-04-19T00:53:57.707 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.707 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865778296}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641711374}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003574091}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=871573530}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3025968490}) 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.708 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.709 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2284812818}) 2024-04-19T00:53:57.709 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.709 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1049180157}) 2024-04-19T00:53:57.709 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.702+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3719079918}) 2024-04-19T00:53:57.711 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.711 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.711 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1829513294}) 2024-04-19T00:53:57.711 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.711 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.711 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=249270147}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505575933}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1339310167}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086990428}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2027215290}) 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.712 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.713 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2336791450}) 2024-04-19T00:53:57.713 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.713 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4201408834}) 2024-04-19T00:53:57.713 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.706+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=101895582}) 2024-04-19T00:53:57.715 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.715 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.715 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797513845}) 2024-04-19T00:53:57.715 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.715 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807793816}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501395900}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1936761644}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=582349812}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3395549803}) 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.716 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.717 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1363798499}) 2024-04-19T00:53:57.717 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.717 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373390193}) 2024-04-19T00:53:57.717 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.710+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3732025477}) 2024-04-19T00:53:57.719 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.719 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.719 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082319545}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123433515}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094922132}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1633992685}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1556314173}) 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.720 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=739084662}) 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2602102726}) 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=351383774}) 2024-04-19T00:53:57.721 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.714+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1438332864}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526405241}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066850595}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1234510971}) 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.723 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2296649214}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2442322628}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=445012649}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1961278476}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375051354}) 2024-04-19T00:53:57.724 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.718+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=650304587}) 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820108000}) 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=559819405}) 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.727 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486870496}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2304051326}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1574767715}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3424049883}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=735659659}) 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.728 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1895420377}) 2024-04-19T00:53:57.729 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.722+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4154450411}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573506863}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1186332075}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2102645505}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.731 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=17979011}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3112667327}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3071436288}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2649457872}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2508517912}) 2024-04-19T00:53:57.732 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.726+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3524719228}) 2024-04-19T00:53:57.733 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713805965}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=971890391}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230442931}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3099559090}) 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.734 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=444443158}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3772261627}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3745260933}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623044031}) 2024-04-19T00:53:57.735 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.730+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3035856816}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818444724}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246546963}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187696641}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.737 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3838545488}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2274978860}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3571837331}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=230961209}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2374261733}) 2024-04-19T00:53:57.738 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1404320038}) 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.734+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801516946}) 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266885856}) 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.741 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3579921725}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1808324529}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1405618021}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3621315143}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2596895162}) 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.742 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3939250989}) 2024-04-19T00:53:57.743 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.738+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3290035802}) 2024-04-19T00:53:57.744 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.744 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.744 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3414621452}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2166097491}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700732354}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3479844305}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1468641644}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=970092083}) 2024-04-19T00:53:57.745 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.746 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.746 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1803930005}) 2024-04-19T00:53:57.746 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.746 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1378352429}) 2024-04-19T00:53:57.746 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2404537171}) 2024-04-19T00:53:57.747 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261738830}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197869885}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1153677632}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3225817195}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=102370121}) 2024-04-19T00:53:57.748 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2233157636}) 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2642628217}) 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1900412344}) 2024-04-19T00:53:57.749 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.742+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=409972813}) 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201213849}) 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309456184}) 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.751 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1258191362}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2258101021}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4162024502}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2176846379}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=906371055}) 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.752 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2153735670}) 2024-04-19T00:53:57.753 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.746+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1790789945}) 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132194953}) 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115591361}) 2024-04-19T00:53:57.755 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3512929339}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2838590177}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2675357505}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3315307108}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3082609590}) 2024-04-19T00:53:57.756 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.757 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3941847182}) 2024-04-19T00:53:57.757 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.750+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3849191034}) 2024-04-19T00:53:57.758 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.758 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1873200716}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661938297}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534717422}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1697235560}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2268921472}) 2024-04-19T00:53:57.759 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1682470608}) 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=633316392}) 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2974767003}) 2024-04-19T00:53:57.760 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.754+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4209089284}) 2024-04-19T00:53:57.762 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.762 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.762 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2791627505}) 2024-04-19T00:53:57.762 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.762 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.762 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1160664299}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622995323}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=464042303}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3271799832}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3903887552}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.763 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4216120797}) 2024-04-19T00:53:57.764 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.764 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3417963936}) 2024-04-19T00:53:57.764 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.758+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=58268317}) 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2593416009}) 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3388643516}) 2024-04-19T00:53:57.766 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256098372}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1068680448}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4191715250}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=542853808}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1081223282}) 2024-04-19T00:53:57.767 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.768 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=900244898}) 2024-04-19T00:53:57.768 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.762+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=771443865}) 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2368173518}) 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427163270}) 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.770 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1207477841}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2281759297}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2170362481}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2795067763}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2637098339}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=850507831}) 2024-04-19T00:53:57.771 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=596137064}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161587085}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021700542}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837207508}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4094003049}) 2024-04-19T00:53:57.776 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=297388910}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1261026486}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4075945561}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.766+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2490981665}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999438782}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787119037}) 2024-04-19T00:53:57.777 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=904888047}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787072318}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1305447158}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=344521039}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2086528449}) 2024-04-19T00:53:57.778 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.779 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.779 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4089280902}) 2024-04-19T00:53:57.779 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.779 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4289880879}) 2024-04-19T00:53:57.779 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.770+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1921521207}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=449992695}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972149339}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3074066368}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1368653571}) 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.781 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2624606214}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=749234189}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1301598053}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3461147024}) 2024-04-19T00:53:57.782 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4243382908}) 2024-04-19T00:53:57.784 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.784 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.784 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341882269}) 2024-04-19T00:53:57.784 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4275085870}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733795424}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2013934950}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.778+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=431448075}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1469590585}) 2024-04-19T00:53:57.785 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.786 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.786 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2457921045}) 2024-04-19T00:53:57.786 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.786 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3881364038}) 2024-04-19T00:53:57.786 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4111248363}) 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999128478}) 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674255985}) 2024-04-19T00:53:57.788 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3934178297}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3948528061}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=107584365}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1442488340}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=696332975}) 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.789 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.782+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=327259587}) 2024-04-19T00:53:57.790 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2557973358}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3228998147}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11364970}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968919021}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1495744529}) 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.792 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1580510258}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=844154620}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1652471253}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708193741}) 2024-04-19T00:53:57.793 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.786+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3242003687}) 2024-04-19T00:53:57.795 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.795 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403490030}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1108802684}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353472927}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1971548778}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1046381544}) 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.796 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=36312664}) 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3741018040}) 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2631371195}) 2024-04-19T00:53:57.797 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.790+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2016643731}) 2024-04-19T00:53:57.798 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.798 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.798 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574494242}) 2024-04-19T00:53:57.798 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.798 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.798 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548727952}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447285495}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3773424448}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4274332335}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3718604361}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2678558442}) 2024-04-19T00:53:57.799 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.800 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868845361}) 2024-04-19T00:53:57.800 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.794+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3484160698}) 2024-04-19T00:53:57.801 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.801 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66221480}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4061501776}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117192004}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1501110738}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.802 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1887413290}) 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=503954077}) 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2291605835}) 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2068319375}) 2024-04-19T00:53:57.803 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.798+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3100028795}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249712863}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=405171193}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036938998}) 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.806 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=578581565}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3171578801}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1060323327}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2608729258}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1268239249}) 2024-04-19T00:53:57.807 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.802+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2832455491}) 2024-04-19T00:53:57.809 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.809 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864759764}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052281265}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878207408}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=862853551}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=693539619}) 2024-04-19T00:53:57.810 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1683734195}) 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=973715652}) 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2026400405}) 2024-04-19T00:53:57.811 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2611471546}) 2024-04-19T00:53:57.813 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.813 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.813 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262453577}) 2024-04-19T00:53:57.813 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849433595}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=64739307}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.806+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=634993173}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4115923447}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.814 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1767892529}) 2024-04-19T00:53:57.815 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.815 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.815 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2195456807}) 2024-04-19T00:53:57.815 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.815 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1943371997}) 2024-04-19T00:53:57.815 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1583325674}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581270659}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3806151330}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787096799}) 2024-04-19T00:53:57.817 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3376310387}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3232076894}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2784145527}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=407454772}) 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.818 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2660618616}) 2024-04-19T00:53:57.819 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.810+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2772516177}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1897452372}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257661045}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=603718180}) 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.821 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3496693760}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3579988193}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3957329359}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=847382638}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2332943857}) 2024-04-19T00:53:57.822 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.814+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1964214693}) 2024-04-19T00:53:57.826 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.826 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.826 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789947673}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1306372443}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2526866475}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2613381506}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2652163576}) 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.827 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3561379320}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2323410113}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2427050205}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.818+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4196348594}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967665790}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890583644}) 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.828 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241035464}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=631171509}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2296455530}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1998463681}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=465741215}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4033471059}) 2024-04-19T00:53:57.829 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.822+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3061433773}) 2024-04-19T00:53:57.831 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.831 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.831 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241717679}) 2024-04-19T00:53:57.831 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.831 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505726772}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94361717}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2856285110}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3502585107}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2063220368}) 2024-04-19T00:53:57.832 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.833 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.833 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2862352084}) 2024-04-19T00:53:57.833 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.833 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2023374688}) 2024-04-19T00:53:57.833 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.826+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3124035147}) 2024-04-19T00:53:57.835 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.835 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.835 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904684442}) 2024-04-19T00:53:57.835 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.835 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.835 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=430445603}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=752052552}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=825450886}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2935205696}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1972350404}) 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.836 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.837 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=655093947}) 2024-04-19T00:53:57.837 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.837 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291909149}) 2024-04-19T00:53:57.837 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.830+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3714874692}) 2024-04-19T00:53:57.839 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.839 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.839 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=724246042}) 2024-04-19T00:53:57.839 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.839 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.839 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451835269}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3431255131}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=570196684}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1376806627}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2623475115}) 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.840 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.841 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4250959216}) 2024-04-19T00:53:57.841 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.841 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2580235016}) 2024-04-19T00:53:57.841 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.834+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2625346137}) 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748608156}) 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4276642268}) 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.845 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231695836}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3910314500}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=647297982}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=737044345}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3947296285}) 2024-04-19T00:53:57.846 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1708585205}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1313501415}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820878955}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3944503651}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3381342204}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1229919668}) 2024-04-19T00:53:57.847 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3788467744}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.838+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1194064056}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2322748010}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3796337408}) 2024-04-19T00:53:57.848 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=708634811}) 2024-04-19T00:53:57.850 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.850 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562763164}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52796537}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003299557}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1515013650}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=852455191}) 2024-04-19T00:53:57.851 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3925057019}) 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4019835984}) 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162911422}) 2024-04-19T00:53:57.852 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.842+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=382833125}) 2024-04-19T00:53:57.853 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2097836244}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3680798777}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3796962053}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1150964962}) 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.854 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1090099047}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=969385595}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2707082560}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3076970752}) 2024-04-19T00:53:57.855 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.850+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1790393814}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3813173162}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775520196}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2650557587}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.862 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3301432000}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2650923415}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2167297250}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2420591693}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4126135185}) 2024-04-19T00:53:57.863 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.858+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=550459108}) 2024-04-19T00:53:57.865 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.865 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338701377}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782091234}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2632619693}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1907354444}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1459794148}) 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.866 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1463912243}) 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=45982718}) 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=623401178}) 2024-04-19T00:53:57.867 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.862+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3306313344}) 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=8607465}) 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=705558040}) 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.871 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711324037}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325064984}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3016211170}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1907768301}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2332784584}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2329617272}) 2024-04-19T00:53:57.872 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.866+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1326593608}) 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502799957}) 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=970945588}) 2024-04-19T00:53:57.882 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=502222623}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2930931980}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4158796024}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4197644790}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1410697592}) 2024-04-19T00:53:57.883 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.884 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=799156062}) 2024-04-19T00:53:57.884 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.878+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1482057139}) 2024-04-19T00:53:57.891 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1139367057}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37534209}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1694246262}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.886+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823010406}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1000453751}) 2024-04-19T00:53:57.892 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=584405797}) 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1573676965}) 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1062179579}) 2024-04-19T00:53:57.893 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.890+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3497879046}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033430283}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3798225379}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2289481342}) 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.941 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1459868134}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2760263668}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1211080892}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=672289981}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3407251367}) 2024-04-19T00:53:57.942 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1800187967}) 2024-04-19T00:53:57.951 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.951 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553146628}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216451122}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1126927001}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=406244850}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2958012956}) 2024-04-19T00:53:57.952 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3053230955}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=146267865}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3166587549}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1792282629}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521806522}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2035448649}) 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.953 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.938+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714601967}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1451519210}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2932701590}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1800855672}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1817598233}) 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.954 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1852991267}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1458311306}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528743774}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956884678}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730665448}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2747395487}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.955 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3410196575}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3963421212}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3873852913}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2604121237}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4083693929}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386622376}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3674607281}) 2024-04-19T00:53:57.956 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753597941}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=222511827}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2215649521}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=485199256}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2250017474}) 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.957 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=834238383}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.942+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3550184878}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2855537764}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769377008}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999073924}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3761731570}) 2024-04-19T00:53:57.958 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2784863639}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2242870632}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1277782798}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2777922407}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3007429204}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863670776}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.959 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3517008763}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1243343703}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2505229619}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3725580617}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3604061121}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1519561530}) 2024-04-19T00:53:57.960 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.961 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=184712563}) 2024-04-19T00:53:57.961 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.946+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3326324993}) 2024-04-19T00:53:57.988 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.988 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.988 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171289912}) 2024-04-19T00:53:57.988 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.988 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3725952131}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679905626}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2961190752}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2359921059}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1632451797}) 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.989 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3201685156}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160629758}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3060005192}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751029425}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52466228}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900725067}) 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.990 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=637253646}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=972179485}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1595587235}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=105717725}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1654153870}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.950+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3692381019}) 2024-04-19T00:53:57.991 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205777091}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778289113}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061689940}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1163690598}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=507860667}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.992 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3087112885}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=394594028}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4056201827}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2264149059}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997990412}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1221779774}) 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.993 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1418744657}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3161993760}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=685003288}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3019969597}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1116194094}) 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.994 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333202163}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.954+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=323440558}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586977852}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997184251}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869791534}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4122635255}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632503564}) 2024-04-19T00:53:57.995 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4103144439}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1587971950}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43839978}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2288479982}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871762966}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811746071}) 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.996 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70951500}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2872004053}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1718072877}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2564640763}) 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.997 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=627021357}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3112936290}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=258404231}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134598918}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865286855}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2254790014}) 2024-04-19T00:53:57.998 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1831021792}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3308652926}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3723533154}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1345302220}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=498831937}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.958+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=69744238}) 2024-04-19T00:53:57.999 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708665231}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=397103490}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069883336}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=67501539}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1839703679}) 2024-04-19T00:53:58.000 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3183690167}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4242741169}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288518437}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2700271638}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1831378937}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2886607056}) 2024-04-19T00:53:58.001 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629113393}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3159102237}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=789215253}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2878622556}) 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.002 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4218282272}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1167775149}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1361900912}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607304702}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809626508}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3840921296}) 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.003 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2177472104}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3123843499}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2873310747}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2632964242}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83174459}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.962+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4177782585}) 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.004 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267561037}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=665504678}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3333512228}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=321443947}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1080921544}) 2024-04-19T00:53:58.005 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=306490755}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1211807773}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3925981238}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3992717335}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719672675}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183018451}) 2024-04-19T00:53:58.006 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=475859046}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=402562056}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2541812476}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1397784790}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=795736499}) 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.007 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=946876559}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3437181624}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3756603833}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1925544249}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105907508}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664719916}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.008 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2679343079}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4182298780}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3307477343}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=940693034}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.966+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1561696505}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334433216}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940470896}) 2024-04-19T00:53:58.009 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1807792118}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=625361913}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=362848380}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1598240157}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2350933940}) 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.010 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1966748518}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4093812013}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=883866004}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336168777}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655583703}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=496595006}) 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.011 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=972214402}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2856037963}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3093851334}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1155187118}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3698868168}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009644869}) 2024-04-19T00:53:58.012 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1830498899}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=500620876}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1306331124}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=506729555}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3809548315}) 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.013 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3399904675}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338474806}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.970+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2405925838}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5} crc {logm=3441680721,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2888004437,mon_config_key=2040647713}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9} crc {logm=347580248,mdsmap=2840952586,mgr=1170334752,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=4246663844,mon_config_key=258286766}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=65,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11} crc {logm=2265774908,mdsmap=2840952586,mgr=2484425204,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2184189624,mon_config_key=433660206}) 2024-04-19T00:53:58.014 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=70,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=5} crc {logm=813501298,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=692040121,mon_config_key=2040647713}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=2657708075,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=2040647713}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=2839535284,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=2040647713}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=1615441091,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=2040647713}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=69,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=6} crc {logm=3750888342,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=11793952}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=5} crc {logm=2247140557,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=226717729,mon_config_key=2040647713}) 2024-04-19T00:53:58.015 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=52} crc {mon_config_key=3882216626,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2901575075,osd_snap=1154821678,osdmap=3036328369}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=51} crc {mon_config_key=3882216626,monmap=4080741109,osd_metadata=3305883572,osd_pg_creating=2901575075,osd_snap=33608336,osdmap=750503422}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=51} crc {mon_config_key=3882216626,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=2901575075,osd_snap=33608336,osdmap=1396872559}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=53} crc {mon_config_key=3866522675,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2901575075,osd_snap=807858280,osdmap=4119065716}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=53} crc {mon_config_key=3882216626,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2901575075,osd_snap=3181008846,osdmap=1051457329}) 2024-04-19T00:53:58.016 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=52} crc {mon_config_key=3882216626,monmap=4080741109,osd_metadata=1206563275,osd_pg_creating=2901575075,osd_snap=1810601762,osdmap=1463253169}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=51} crc {mon_config_key=3882216626,monmap=4080741109,osd_metadata=2106968705,osd_pg_creating=2901575075,osd_snap=33608336,osdmap=3217561974}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=11,monmap=2,osd_metadata=4,osd_pg_creating=1,osd_snap=29,osdmap=53} crc {mon_config_key=3882216626,monmap=2862618235,osd_metadata=84505474,osd_pg_creating=2901575075,osd_snap=660638928,osdmap=49765422}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.974+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=52} crc {mon_config_key=2120643142,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2901575075,osd_snap=723953564,osdmap=1418097916}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1508525064}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2257583454}) 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.017 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=406583224}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1908992781}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1975499572}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2474615376}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=434661149}) 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.018 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3097067502}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.978+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1778610202}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2689725824}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=171076748}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=906593963}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3756614151}) 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.019 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1233787049}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1198811625}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2766908360}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056060216}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4015794466}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2346596487}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=208870818}) 2024-04-19T00:53:58.020 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=214140880}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1242232741}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3738613594}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2945020098}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1252526625}) 2024-04-19T00:53:58.021 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.022 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1790392153}) 2024-04-19T00:53:58.022 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.982+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2955636006}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=394160987}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2359405408}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4029838008}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.025 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=4237190358}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1308226829}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2652102852}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3918475202}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=133729533}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.986+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2178558827}) 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.994+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.026 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.994+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.994+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=97} crc {osdmap=2707245184}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=93} crc {osdmap=2148494396}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=93} crc {osdmap=3922073131}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=92} crc {osdmap=3101911336}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=95} crc {osdmap=1110410393}) 2024-04-19T00:53:58.027 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=95} crc {osdmap=2692942469}) 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=96} crc {osdmap=4165236911}) 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=96} crc {osdmap=3929568962}) 2024-04-19T00:53:58.028 INFO:tasks.ceph.mon.e.smithi043.stderr:2024-04-19T00:53:57.998+0000 7f6ec7a41640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=91} crc {osdmap=4232929503}) 2024-04-19T00:53:58.684 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:53:58.684 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient: get_monmap_and_config 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient: build_initial_monmap 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient: monmap: 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:58.828221+0000 2024-04-19T00:53:58.825 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:58.828221+0000 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:58.826 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:58.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 Processor -- start 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 -- start start 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient: init 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient: _add_conns ranks=[5,6,4,10,0,3,2,8,7,1,9] 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8138a50 0x7f0de8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): picked mon.noname-h con 0x7f0de8138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 0x7f0de8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): picked mon.noname-d con 0x7f0de80a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 0x7f0de8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): picked mon.noname-c con 0x7f0de8137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0de80a1020 con 0x7f0de8137960 2024-04-19T00:53:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0de81398b0 con 0x7f0de8138a50 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0de80a0780 con 0x7f0de80a5bb0 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0ded89c640 10 monclient(hunting): authenticate will time out at 1508.955407s 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 0x7f0de8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 0x7f0de8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 0x7f0de8137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:44976/0 (socket says 172.21.15.43:44976) 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 0x7f0de8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33216/0 (socket says 172.21.15.43:33216) 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 1 -- 172.21.15.43:0/4057282550 learned_addr learned my addr 172.21.15.43:0/4057282550 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): get_auth_request con 0x7f0de8137960 auth_method 0 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:58.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): get_auth_request con 0x7f0de80a5bb0 auth_method 0 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de6ffd640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8138a50 0x7f0de8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de6ffd640 10 monclient(hunting): get_auth_request con 0x7f0de8138a50 auth_method 0 2024-04-19T00:53:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient(hunting): handle_auth_done global_id 74184 payload 274 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 -- 172.21.15.43:0/4057282550 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8138a50 msgr2=0x7f0de8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8138a50 0x7f0de8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 -- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 msgr2=0x7f0de8137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 0x7f0de8137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient: _finish_hunting 0 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 monclient: found mon.noname-d 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de67fc640 1 -- 172.21.15.43:0/4057282550 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de80a0d40 con 0x7f0de80a5bb0 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de77fe640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 0x7f0de8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.822+0000 7f0de6ffd640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8138a50 0x7f0de8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de67fc640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 0x7f0de8139370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0097f0 tx=0x7f0ddc01c370 comp rx=0 tx=0).ready entity=mon.6 client_cookie=2c4f4c4b62f76113 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 1 -- 172.21.15.43:0/4057282550 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ddc0b0070 con 0x7f0de80a5bb0 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: dump: 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:58.838 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: _finish_auth 0 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 1 -- 172.21.15.43:0/4057282550 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ddc0aa810 con 0x7f0de80a5bb0 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 1 -- 172.21.15.43:0/4057282550 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ddc0a9300 con 0x7f0de80a5bb0 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 5 monclient: authenticate success, global_id 74184 2024-04-19T00:53:58.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 4 set_mon_vals no callback set 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0de5ffb640 10 monclient: dump: 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:58.840 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 10 monclient: shutdown 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 -- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 msgr2=0x7f0de8139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 0x7f0de8139370 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc0097f0 tx=0x7f0ddc01c370 comp rx=0 tx=0).stop 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 -- 172.21.15.43:0/4057282550 shutdown_connections 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0de8137960 0x7f0de8137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de80a5bb0 0x7f0de8139370 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 --2- 172.21.15.43:0/4057282550 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8138a50 0x7f0de8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 -- 172.21.15.43:0/4057282550 >> 172.21.15.43:0/4057282550 conn(0x7f0de80a7cb0 msgr2=0x7f0de80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 -- 172.21.15.43:0/4057282550 shutdown_connections 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 -- 172.21.15.43:0/4057282550 wait complete. 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 10 monclient: build_initial_monmap 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 10 monclient: monmap: 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:58.842 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:58.845 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 librados: starting msgr at 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 librados: starting objecter 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.826+0000 7f0ded89c640 1 Processor -- start 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- start start 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 librados: setting wanted keys 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 librados: calling monclient init 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: init 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:58.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _add_conns ranks=[3,5,6,0,7,1,4,8,2] 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0de80a5bb0 0x7f0de81368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): picked mon.noname-g con 0x7f0de80a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8137960 0x7f0de8136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): picked mon.noname-h con 0x7f0de8137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 0x7f0de813c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): picked mon.noname-d con 0x7f0de8138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0de80a2100 con 0x7f0de80a5bb0 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0de80a1f80 con 0x7f0de8137960 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0de8142810 con 0x7f0de8138a50 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient(hunting): authenticate will time out at 1508.960570s 2024-04-19T00:53:58.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 0x7f0de813c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 0x7f0de813c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33230/0 (socket says 172.21.15.43:33230) 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 -- 172.21.15.43:0/85524701 learned_addr learned my addr 172.21.15.43:0/85524701 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): get_auth_request con 0x7f0de8138a50 auth_method 0 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de67fc640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8137960 0x7f0de8136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient(hunting): handle_auth_done global_id 74193 payload 824 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de6ffd640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0de80a5bb0 0x7f0de81368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 -- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8137960 msgr2=0x7f0de8136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8137960 0x7f0de8136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 -- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0de80a5bb0 msgr2=0x7f0de81368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0de80a5bb0 0x7f0de81368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 monclient: found mon.noname-d 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 -- 172.21.15.43:0/85524701 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de809ee60 con 0x7f0de8138a50 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de6ffd640 10 monclient: get_auth_request con 0x7f0de80a5bb0 auth_method 0 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de67fc640 10 monclient: get_auth_request con 0x7f0de8137960 auth_method 0 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de6ffd640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0de80a5bb0 0x7f0de81368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de67fc640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8137960 0x7f0de8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de77fe640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 0x7f0de813c810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0de0009ef0 tx=0x7f0de000c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 client.?.objecter ms_handle_connect 0x7f0de8138a50 2024-04-19T00:53:58.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de000cc40 con 0x7f0de8138a50 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: dump: 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:58.850 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: _finish_auth 0 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: _check_auth_tickets 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de0019070 con 0x7f0de8138a50 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 5 monclient: authenticate success, global_id 74193 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _renew_subs 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de81219e0 con 0x7f0de8138a50 2024-04-19T00:53:58.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0de000ce20 con 0x7f0de8138a50 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 monclient: dump: 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:58.852 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 client.74193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _renew_subs 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de81376a0 con 0x7f0de8138a50 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 librados: init done 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de57fa640 4 set_mon_vals no callback set 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:58.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0de0012050 con 0x7f0de8138a50 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de80008e0 con 0x7f0de8138a50 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0dd4038e10 0x7f0dd403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f0de0049930 con 0x7f0de8138a50 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 client.74193.objecter ms_dispatch 0x7f0de8000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 3 client.74193.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 3 client.74193.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 20 client.74193.objecter dump_active .. 0 homeless 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de6ffd640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0dd4038e10 0x7f0dd403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de6ffd640 10 monclient: get_auth_request con 0x7f0dd4038e10 auth_method 0 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0de6ffd640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0dd4038e10 0x7f0dd403b2d0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8009720 tx=0x7f0dd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:58.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.830+0000 7f0dcffff640 10 client.74193.objecter ms_handle_connect 0x7f0dd4038e10 2024-04-19T00:53:58.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.834+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0de0010040 con 0x7f0de8138a50 2024-04-19T00:53:58.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.834+0000 7f0dcffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:58.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:58.834+0000 7f0dcffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:59.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:53:59.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:53:59.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0de81d0b00 con 0x7f0de8138a50 2024-04-19T00:53:59.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0dcffff640 1 -- 172.21.15.43:0/85524701 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f0de001b310 con 0x7f0de8138a50 2024-04-19T00:53:59.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0dcffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:53:59.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0dcffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:59.093 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":66,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"e","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 librados: watch_flush enter 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 librados: watch_flush exit 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 20 client.74193.objecter shutdown clearing up homeless session... 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 client.74193.objecter successfully canceled tick 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0dd4038e10 msgr2=0x7f0dd403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0dd4038e10 0x7f0dd403b2d0 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8009720 tx=0x7f0dd8008040 comp rx=0 tx=0).stop 2024-04-19T00:53:59.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 10 monclient: shutdown 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.090+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 msgr2=0x7f0de813c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 0x7f0de813c810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0de0009ef0 tx=0x7f0de000c490 comp rx=0 tx=0).stop 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 shutdown_connections 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0dd4038e10 0x7f0dd403b2d0 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0de8138a50 0x7f0de813c810 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0de8137960 0x7f0de8136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 --2- 172.21.15.43:0/85524701 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0de80a5bb0 0x7f0de81368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 >> 172.21.15.43:0/85524701 conn(0x7f0de80a7cb0 msgr2=0x7f0de80a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 shutdown_connections 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 -- 172.21.15.43:0/85524701 wait complete. 2024-04-19T00:53:59.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.094+0000 7f0ded89c640 1 librados: shutdown 2024-04-19T00:53:59.112 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:53:59.112 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:53:59.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:53:59.113 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:53:59.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient: get_monmap_and_config 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient: monmap: 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:59.242725+0000 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:59.242725+0000 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.240 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:59.241 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 Processor -- start 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 -- start start 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient: init 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient: _add_conns ranks=[9,3,0,5,7,6,4,8,2,1,10] 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8000a2990 0x7fe8000725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): picked mon.noname-a con 0x7fe8000a2990 addr v2:172.21.15.43:3300/0 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8000a2d70 0x7fe8000a3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): picked mon.noname-g con 0x7fe8000a2d70 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe800135370 addr v1:172.21.15.43:6789/0 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe80006fe00 con 0x7fe800135370 2024-04-19T00:53:59.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fe800071e00 con 0x7fe8000a2990 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe800071560 con 0x7fe8000a2d70 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe8066ea640 10 monclient(hunting): authenticate will time out at 1509.369841s 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe8000a2990 0x7fe8000725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe804c87640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fe800135370 0x7fe800135800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37330/0 (socket says 172.21.15.43:37330) 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe804c87640 1 -- 172.21.15.43:0/2046111595 learned_addr learned my addr 172.21.15.43:0/2046111595 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): get_auth_request con 0x7fe8000a2990 auth_method 0 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7ff7fe640 1 --2- 172.21.15.43:0/2046111595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8000a2d70 0x7fe8000a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe8000a2d70 auth_method 0 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7ff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient(hunting): handle_auth_done global_id 64521 payload 274 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 1 -- 172.21.15.43:0/2046111595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8000a2d70 msgr2=0x7fe8000a3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 1 --2- 172.21.15.43:0/2046111595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8000a2d70 0x7fe8000a3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 1 -- 172.21.15.43:0/2046111595 >> v1:172.21.15.43:6789/0 conn(0x7fe800135370 legacy=0x7fe800135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 1 monclient: found mon.noname-a 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7fffff640 1 -- 172.21.15.43:0/2046111595 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8000706d0 con 0x7fe8000a2990 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7feffd640 1 -- 172.21.15.43:0/2046111595 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 17141764 0 0) 0x7fe80006fe00 con 0x7fe800135370 2024-04-19T00:53:59.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.238+0000 7fe7feffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 1 --2- 172.21.15.43:0/2046111595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2990 0x7fe8000725e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009d90 tx=0x7fe7f001c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2549b1bf7d46e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 1 -- 172.21.15.43:0/2046111595 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7f00a6980 con 0x7fe8000a2990 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: dump: 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.253 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 5 monclient: authenticate success, global_id 64521 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 1 -- 172.21.15.43:0/2046111595 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f00a6b40 con 0x7fe8000a2990 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 4 set_mon_vals no callback set 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 1 -- 172.21.15.43:0/2046111595 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7f00a6cf0 con 0x7fe8000a2990 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7feffd640 10 monclient: dump: 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.255 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: shutdown 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- 172.21.15.43:0/2046111595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2990 msgr2=0x7fe8000725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 --2- 172.21.15.43:0/2046111595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2990 0x7fe8000725e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0009d90 tx=0x7fe7f001c8c0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- 172.21.15.43:0/2046111595 shutdown_connections 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 --2- 172.21.15.43:0/2046111595 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8000a2d70 0x7fe8000a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 --2- 172.21.15.43:0/2046111595 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2990 0x7fe8000725e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- 172.21.15.43:0/2046111595 >> 172.21.15.43:0/2046111595 conn(0x7fe8000a7fb0 msgr2=0x7fe8000a83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- 172.21.15.43:0/2046111595 shutdown_connections 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- 172.21.15.43:0/2046111595 wait complete. 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: monmap: 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:59.257 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 librados: starting msgr at 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 librados: starting objecter 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 Processor -- start 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- start start 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 librados: setting wanted keys 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 librados: calling monclient init 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: init 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: _add_conns ranks=[8,0,7,1,2,4,5,6,3] 2024-04-19T00:53:59.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8000a2990 0x7fe8001d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): picked mon.noname-e con 0x7fe8000a2990 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 0x7fe800134580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): picked mon.noname-a con 0x7fe8000a2d70 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe800135370 0x7fe80013a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): picked mon.noname-i con 0x7fe800135370 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe80013d960 con 0x7fe8000a2d70 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe80013d7e0 con 0x7fe800135370 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe80013dae0 con 0x7fe8000a2990 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient(hunting): authenticate will time out at 1509.374891s 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 0x7fe800134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 0x7fe800134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42856/0 (socket says 172.21.15.43:42856) 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 -- 172.21.15.43:0/3549551040 learned_addr learned my addr 172.21.15.43:0/3549551040 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): get_auth_request con 0x7fe8000a2d70 auth_method 0 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8000a2990 0x7fe8001d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): get_auth_request con 0x7fe8000a2990 auth_method 0 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe804c87640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe800135370 0x7fe80013a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe804c87640 10 monclient(hunting): get_auth_request con 0x7fe800135370 auth_method 0 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe804c87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe804c87640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe804c87640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient(hunting): handle_auth_done global_id 64539 payload 824 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8000a2990 msgr2=0x7fe8001d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8000a2990 0x7fe8001d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe800135370 msgr2=0x7fe80013a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe800135370 0x7fe80013a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 monclient: found mon.noname-a 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe800070fc0 con 0x7fe8000a2d70 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7ff7fe640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 0x7fe800134580 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d800 tx=0x7fe7e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 client.?.objecter ms_handle_connect 0x7fe8000a2d70 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7e800bb30 con 0x7fe8000a2d70 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: dump: 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.262 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7e8019070 con 0x7fe8000a2d70 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7e800bcf0 con 0x7fe8000a2d70 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fdffb640 4 set_mon_vals no callback set 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe7fcff9640 10 monclient: dump: 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.264 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 5 monclient: authenticate success, global_id 64539 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: _renew_subs 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80013bf30 con 0x7fe8000a2d70 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.242+0000 7fe8066ea640 10 client.64539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 10 monclient: _renew_subs 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe800134c90 con 0x7fe8000a2d70 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 1 librados: init done 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 0x7fe80013ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe80013c8c0 con 0x7fe80013a7a0 2024-04-19T00:53:59.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe7e800b6c0 con 0x7fe8000a2d70 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 0x7fe80013ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient: get_auth_request con 0x7fe80013a7a0 auth_method 0 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe7e0038e10 0x7fe7e003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7fe7e800f040 con 0x7fe8000a2d70 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 client.64539.objecter ms_dispatch 0x7fe800000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 3 client.64539.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 3 client.64539.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 10 monclient(hunting): handle_auth_done global_id 64548 payload 824 2024-04-19T00:53:59.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 20 client.64539.objecter dump_active .. 0 homeless 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe80011ef70 con 0x7fe80013a7a0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 client.64539.objecter ms_handle_connect 0x7fe80013a7a0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 client.64539.objecter resend_mon_ops 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fffff640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 0x7fe80013ab80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a42d0 tx=0x7fe7f0019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe804c87640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe7e0038e10 0x7fe7e003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe804c87640 10 monclient: get_auth_request con 0x7fe7e0038e10 auth_method 0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7f00be070 con 0x7fe80013a7a0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 monclient: dump: 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.267 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe7f0004390 con 0x7fe80013a7a0 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 msgr2=0x7fe80013ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 0x7fe80013ab80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a42d0 tx=0x7fe7f0019660 comp rx=0 tx=0).stop 2024-04-19T00:53:59.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 msgr2=0x7fe80013ab80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013a7a0 0x7fe80013ab80 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe8000a42d0 tx=0x7fe7f0019660 comp rx=0 tx=0).stop 2024-04-19T00:53:59.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe804c87640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe7e0038e10 0x7fe7e003b2d0 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8007fd0 tx=0x7fe7f8007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.246+0000 7fe7fcff9640 10 client.64539.objecter ms_handle_connect 0x7fe7e0038e10 2024-04-19T00:53:59.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:59.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 0x7fe800136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe80006d140 con 0x7fe80013b0c0 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 0x7fe800136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient: get_auth_request con 0x7fe80013b0c0 auth_method 0 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 10 monclient(hunting): handle_auth_done global_id 64557 payload 824 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 1 -- 172.21.15.43:0/3549551040 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe800070a50 con 0x7fe80013b0c0 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fffff640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 0x7fe800136380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe7f001a4f0 tx=0x7fe7f00a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 client.64539.objecter ms_handle_connect 0x7fe80013b0c0 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 client.64539.objecter resend_mon_ops 2024-04-19T00:53:59.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe7f00cd8c0 con 0x7fe80013b0c0 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 monclient: dump: 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.272 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe7f00ba420 con 0x7fe80013b0c0 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 msgr2=0x7fe800136380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 0x7fe800136380 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe7f001a4f0 tx=0x7fe7f00a9000 comp rx=0 tx=0).stop 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 msgr2=0x7fe800136380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe7fcff9640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe80013b0c0 0x7fe800136380 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe7f001a4f0 tx=0x7fe7f00a9000 comp rx=0 tx=0).stop 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:59.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 165526, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:59.274 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.275 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.276 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.277 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.278 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:59.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.280 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.281 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:59.282 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:59.286 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.287 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:59.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.291 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:53:59.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:59.293 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 10 librados: watch_flush enter 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 10 librados: watch_flush exit 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 20 client.64539.objecter shutdown clearing up homeless session... 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 10 client.64539.objecter successfully canceled tick 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe7e0038e10 msgr2=0x7fe7e003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe7e0038e10 0x7fe7e003b2d0 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8007fd0 tx=0x7fe7f8007ca0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 10 monclient: shutdown 2024-04-19T00:53:59.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 msgr2=0x7fe800134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 0x7fe800134580 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe7e800d800 tx=0x7fe7e800dcc0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe804c87640 1 -- 172.21.15.43:0/3549551040 reap_dead start 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 shutdown_connections 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 --2- 172.21.15.43:0/3549551040 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8000a2d70 0x7fe800134580 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 >> 172.21.15.43:0/3549551040 conn(0x7fe8000a7fb0 msgr2=0x7fe8000a73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 shutdown_connections 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 -- 172.21.15.43:0/3549551040 wait complete. 2024-04-19T00:53:59.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.254+0000 7fe8066ea640 1 librados: shutdown 2024-04-19T00:53:59.296 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:59.296 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:53:59.297 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:53:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.410+0000 7f94b8cbb640 10 monclient: get_monmap_and_config 2024-04-19T00:53:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.410+0000 7f94b8cbb640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.410+0000 7f94b8cbb640 10 monclient: monmap: 2024-04-19T00:53:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.415 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:59.418185+0000 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:59.418185+0000 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:59.416 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 Processor -- start 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 -- start start 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient: init 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient: _add_conns ranks=[5,1,10,8,3,4,2,9,0,7,6] 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 0x7f94b41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): picked mon.noname-h con 0x7f94b4138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f94b40a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f94b41395f0 0x7f94b4135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): picked mon.noname-f con 0x7f94b41395f0 addr v2:172.21.15.161:3300/0 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b40a3dc0 con 0x7f94b40a5b80 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f94b40a4410 con 0x7f94b41395f0 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f94b40a40a0 con 0x7f94b4138cd0 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b8cbb640 10 monclient(hunting): authenticate will time out at 1509.545510s 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 0x7f94b41390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 0x7f94b41390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:36454/0 (socket says 172.21.15.43:36454) 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 -- 172.21.15.43:0/3679031820 learned_addr learned my addr 172.21.15.43:0/3679031820 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): get_auth_request con 0x7f94b4138cd0 auth_method 0 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 1 --2- 172.21.15.43:0/3679031820 >> v2:172.21.15.161:3300/0 conn(0x7f94b41395f0 0x7f94b4135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): get_auth_request con 0x7f94b41395f0 auth_method 0 2024-04-19T00:53:59.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient(hunting): handle_auth_done global_id 74147 payload 274 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 -- 172.21.15.43:0/3679031820 >> v2:172.21.15.161:3300/0 conn(0x7f94b41395f0 msgr2=0x7f94b4135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 --2- 172.21.15.43:0/3679031820 >> v2:172.21.15.161:3300/0 conn(0x7f94b41395f0 0x7f94b4135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 -- 172.21.15.43:0/3679031820 >> v1:172.21.15.161:6789/0 conn(0x7f94b40a5b80 legacy=0x7f94b41358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 monclient: found mon.noname-h 2024-04-19T00:53:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:53:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b259c640 1 -- 172.21.15.43:0/3679031820 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b40a2ae0 con 0x7f94b4138cd0 2024-04-19T00:53:59.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.414+0000 7f94b2d9d640 1 --2- 172.21.15.43:0/3679031820 >> v2:172.21.15.161:3300/0 conn(0x7f94b41395f0 0x7f94b4135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b259c640 1 --2- 172.21.15.43:0/3679031820 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 0x7f94b41390b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009870 tx=0x7f94a401c100 comp rx=0 tx=0).ready entity=mon.5 client_cookie=489518b2c6bdcc25 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 1 -- 172.21.15.43:0/3679031820 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f94a40b1070 con 0x7f94b4138cd0 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: dump: 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.424 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 1 -- 172.21.15.43:0/3679031820 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b4136710 con 0x7f94b4138cd0 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 5 monclient: authenticate success, global_id 74147 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 1 -- 172.21.15.43:0/3679031820 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f94b40a20a0 con 0x7f94b4138cd0 2024-04-19T00:53:59.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 4 set_mon_vals no callback set 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b159a640 10 monclient: dump: 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.426 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 10 monclient: shutdown 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3679031820 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 msgr2=0x7f94b41390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3679031820 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 0x7f94b41390b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f94a4009870 tx=0x7f94a401c100 comp rx=0 tx=0).stop 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3679031820 shutdown_connections 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3679031820 >> v2:172.21.15.161:3300/0 conn(0x7f94b41395f0 0x7f94b4135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3679031820 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f94b4138cd0 0x7f94b41390b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3679031820 >> 172.21.15.43:0/3679031820 conn(0x7f94b40ab2f0 msgr2=0x7f94b40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3679031820 shutdown_connections 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3679031820 wait complete. 2024-04-19T00:53:59.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.418+0000 7f94b8cbb640 10 monclient: monmap: 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.428 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 librados: starting msgr at 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 librados: starting objecter 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 Processor -- start 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- start start 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 librados: setting wanted keys 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 librados: calling monclient init 2024-04-19T00:53:59.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: init 2024-04-19T00:53:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: _add_conns ranks=[0,6,3,4,5,8,2,7,1] 2024-04-19T00:53:59.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b40a5b80 0x7f94b412e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): picked mon.noname-a con 0x7f94b40a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f94b4138cd0 0x7f94b412ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): picked mon.noname-d con 0x7f94b4138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94b41395f0 0x7f94b412f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): picked mon.noname-g con 0x7f94b41395f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f94b4144660 con 0x7f94b40a5b80 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f94b41444e0 con 0x7f94b41395f0 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f94b41447e0 con 0x7f94b4138cd0 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): authenticate will time out at 1509.553144s 2024-04-19T00:53:59.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b40a5b80 0x7f94b412e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f94b4138cd0 0x7f94b412ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b40a5b80 0x7f94b412e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42876/0 (socket says 172.21.15.43:42876) 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 -- 172.21.15.43:0/3017689001 learned_addr learned my addr 172.21.15.43:0/3017689001 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f94b4138cd0 0x7f94b412ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33232/0 (socket says 172.21.15.43:33232) 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): get_auth_request con 0x7f94b40a5b80 auth_method 0 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): get_auth_request con 0x7f94b4138cd0 auth_method 0 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b2d9d640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94b41395f0 0x7f94b412f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient(hunting): handle_auth_done global_id 64566 payload 824 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f94b4138cd0 msgr2=0x7f94b412ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f94b4138cd0 0x7f94b412ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94b41395f0 msgr2=0x7f94b412f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94b41395f0 0x7f94b412f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 monclient: found mon.noname-a 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b40a04d0 con 0x7f94b40a5b80 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b2d9d640 10 monclient: get_auth_request con 0x7f94b41395f0 auth_method 0 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b2d9d640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f94b41395f0 0x7f94b412f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b259c640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b40a5b80 0x7f94b412e840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94a401a7b0 tx=0x7f94a40191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 client.?.objecter ms_handle_connect 0x7f94b40a5b80 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f94a40b1040 con 0x7f94b40a5b80 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: dump: 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.439 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a4003a90 con 0x7f94b40a5b80 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f94a4003c40 con 0x7f94b40a5b80 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 monclient: dump: 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.441 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b0d99640 4 set_mon_vals no callback set 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 5 monclient: authenticate success, global_id 64566 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: _renew_subs 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b409e680 con 0x7f94b40a5b80 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 client.64566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:59.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: _renew_subs 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b40a3590 con 0x7f94b40a5b80 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 librados: init done 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 0x7f94b4133280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f94b406d140 con 0x7f94b4132ea0 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f94a4003620 con 0x7f94b40a5b80 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 0x7f94b4133280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient: get_auth_request con 0x7f94b4132ea0 auth_method 0 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9494038e10 0x7f949403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f94a40e1f70 con 0x7f94b40a5b80 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 client.64566.objecter ms_dispatch 0x7f94b4000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 3 client.64566.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 3 client.64566.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 20 client.64566.objecter dump_active .. 0 homeless 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_done global_id 74180 payload 824 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94b4136710 con 0x7f94b4132ea0 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 client.64566.objecter ms_handle_connect 0x7f94b4132ea0 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94a2ffd640 10 client.64566.objecter resend_mon_ops 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.422+0000 7f94b1d9b640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 0x7f94b4133280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94a8007fd0 tx=0x7f94a800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f94a800d6d0 con 0x7f94b4132ea0 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 10 monclient: dump: 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.445 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f94a8018070 con 0x7f94b4132ea0 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 msgr2=0x7f94b4133280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 0x7f94b4133280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94a8007fd0 tx=0x7f94a800fc00 comp rx=0 tx=0).stop 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 msgr2=0x7f94b4133280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b4132ea0 0x7f94b4133280 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f94a8007fd0 tx=0x7f94a800fc00 comp rx=0 tx=0).stop 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94b2d9d640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9494038e10 0x7f949403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94b2d9d640 10 monclient: get_auth_request con 0x7f9494038e10 auth_method 0 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94b2d9d640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9494038e10 0x7f949403b2d0 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f94ac007440 tx=0x7f94ac00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.426+0000 7f94a2ffd640 10 client.64566.objecter ms_handle_connect 0x7f9494038e10 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b8cbb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:59.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 0x7f94b409c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b8cbb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f94b4068240 con 0x7f94b41337c0 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 0x7f94b409c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient: get_auth_request con 0x7f94b41337c0 auth_method 0 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 10 monclient(hunting): handle_auth_done global_id 74189 payload 824 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 1 -- 172.21.15.43:0/3017689001 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94b409fdc0 con 0x7f94b41337c0 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94b1d9b640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 0x7f94b409c510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94a800fc40 tx=0x7f94a8000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94a2ffd640 10 client.64566.objecter ms_handle_connect 0x7f94b41337c0 2024-04-19T00:53:59.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94a2ffd640 10 client.64566.objecter resend_mon_ops 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f94a801ee60 con 0x7f94b41337c0 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.430+0000 7f94a2ffd640 10 monclient: dump: 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.448 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f94a800d8e0 con 0x7f94b41337c0 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 msgr2=0x7f94b409c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 0x7f94b409c510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94a800fc40 tx=0x7f94a8000ee0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 msgr2=0x7f94b409c510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94a2ffd640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f94b41337c0 0x7f94b409c510 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f94a800fc40 tx=0x7f94a8000ee0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19051, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:59.450 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.451 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.452 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:59.453 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.454 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.455 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.455 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.455 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.455 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.455 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.455 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.457 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.457 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.457 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.457 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.459 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.460 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.462 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:59.464 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.465 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:59.469 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:59.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.474 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:59.474 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 10 librados: watch_flush enter 2024-04-19T00:53:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 10 librados: watch_flush exit 2024-04-19T00:53:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 20 client.64566.objecter shutdown clearing up homeless session... 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 10 client.64566.objecter successfully canceled tick 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9494038e10 msgr2=0x7f949403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9494038e10 0x7f949403b2d0 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f94ac007440 tx=0x7f94ac00b040 comp rx=0 tx=0).stop 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 10 monclient: shutdown 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b40a5b80 msgr2=0x7f94b412e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 --2- 172.21.15.43:0/3017689001 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f94b40a5b80 0x7f94b412e840 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94a401a7b0 tx=0x7f94a40191c0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b2d9d640 1 -- 172.21.15.43:0/3017689001 reap_dead start 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b2d9d640 1 -- 172.21.15.43:0/3017689001 reap_dead start 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 shutdown_connections 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 >> 172.21.15.43:0/3017689001 conn(0x7f94b40ab2f0 msgr2=0x7f94b4131b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 shutdown_connections 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 -- 172.21.15.43:0/3017689001 wait complete. 2024-04-19T00:53:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.434+0000 7f94b8cbb640 1 librados: shutdown 2024-04-19T00:53:59.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:59.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:53:59.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:53:59.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient: get_monmap_and_config 2024-04-19T00:53:59.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient: monmap: 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:59.603563+0000 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:59.603563+0000 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.601 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.602 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.602 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.602 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:59.602 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:59.602 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 1 Processor -- start 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 1 -- start start 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient: init 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient: _add_conns ranks=[0,1,8,5,6,3,10,7,2,4,9] 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac440a1ec0 addr v1:172.21.15.43:6789/0 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fac440a22a0 addr v1:172.21.15.161:6789/0 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac440a12e0 0x7fac44141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient(hunting): picked mon.noname-e con 0x7fac440a12e0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac440a47d0 con 0x7fac440a1ec0 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4406bfc0 con 0x7fac440a22a0 2024-04-19T00:53:59.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fac4406c4b0 con 0x7fac440a12e0 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac49ff2640 10 monclient(hunting): authenticate will time out at 1509.730675s 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac440a12e0 0x7fac44141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac437fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fac440a1ec0 0x7fac440a39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:37340/0 (socket says 172.21.15.43:37340) 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac437fe640 1 -- 172.21.15.43:0/4065521720 learned_addr learned my addr 172.21.15.43:0/4065521720 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): get_auth_request con 0x7fac440a12e0 auth_method 0 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient(hunting): handle_auth_done global_id 23336 payload 274 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 1 -- 172.21.15.43:0/4065521720 >> v1:172.21.15.161:6789/0 conn(0x7fac440a22a0 legacy=0x7fac440a40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 1 -- 172.21.15.43:0/4065521720 >> v1:172.21.15.43:6789/0 conn(0x7fac440a1ec0 legacy=0x7fac440a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 1 monclient: found mon.noname-e 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:59.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.598+0000 7fac43fff640 1 -- 172.21.15.43:0/4065521720 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac440a0060 con 0x7fac440a12e0 2024-04-19T00:53:59.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac43fff640 1 --2- 172.21.15.43:0/4065521720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac440a12e0 0x7fac44141bc0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009dc0 tx=0x7fac3c01c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=36e39bf1569863e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 1 -- 172.21.15.43:0/4065521720 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac3c0b2070 con 0x7fac440a12e0 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: dump: 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.615 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 1 -- 172.21.15.43:0/4065521720 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c0a5a50 con 0x7fac440a12e0 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 1 -- 172.21.15.43:0/4065521720 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac3c0a5c80 con 0x7fac440a12e0 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac427fc640 10 monclient: dump: 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.617 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 5 monclient: authenticate success, global_id 23336 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:59.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 4 set_mon_vals no callback set 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 10 monclient: shutdown 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 -- 172.21.15.43:0/4065521720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac440a12e0 msgr2=0x7fac44141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 --2- 172.21.15.43:0/4065521720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac440a12e0 0x7fac44141bc0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009dc0 tx=0x7fac3c01c850 comp rx=0 tx=0).stop 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 -- 172.21.15.43:0/4065521720 shutdown_connections 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 --2- 172.21.15.43:0/4065521720 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac440a12e0 0x7fac44141bc0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 -- 172.21.15.43:0/4065521720 >> 172.21.15.43:0/4065521720 conn(0x7fac440a8600 msgr2=0x7fac440a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 -- 172.21.15.43:0/4065521720 shutdown_connections 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 -- 172.21.15.43:0/4065521720 wait complete. 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 10 monclient: monmap: 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.619 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 librados: starting msgr at 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 librados: starting objecter 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.602+0000 7fac49ff2640 1 Processor -- start 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- start start 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 librados: setting wanted keys 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 librados: calling monclient init 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: init 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: _add_conns ranks=[3,6,1,7,0,5,8,2,4] 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac440a12e0 0x7fac441d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): picked mon.noname-g con 0x7fac440a12e0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fac440a1ec0 0x7fac441d1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): picked mon.noname-d con 0x7fac440a1ec0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac441d18a0 0x7fac4413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): picked mon.noname-f con 0x7fac441d18a0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac44144aa0 con 0x7fac441d18a0 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fac44144920 con 0x7fac440a12e0 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fac44144c20 con 0x7fac440a1ec0 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): authenticate will time out at 1509.736724s 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fac440a1ec0 0x7fac441d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fac440a1ec0 0x7fac441d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33240/0 (socket says 172.21.15.43:33240) 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 -- 172.21.15.43:0/2285441858 learned_addr learned my addr 172.21.15.43:0/2285441858 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): get_auth_request con 0x7fac440a1ec0 auth_method 0 2024-04-19T00:53:59.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac440a12e0 0x7fac441d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac441d18a0 0x7fac4413f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): get_auth_request con 0x7fac440a12e0 auth_method 0 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 10 monclient(hunting): get_auth_request con 0x7fac441d18a0 auth_method 0 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient(hunting): handle_auth_done global_id 74211 payload 824 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac440a12e0 msgr2=0x7fac441d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac440a12e0 0x7fac441d0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac441d18a0 msgr2=0x7fac4413f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac441d18a0 0x7fac4413f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 monclient: found mon.noname-d 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac44141dc0 con 0x7fac440a1ec0 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac42ffd640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fac440a1ec0 0x7fac441d1360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac3400b410 tx=0x7fac3400b8d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 client.?.objecter ms_handle_connect 0x7fac440a1ec0 2024-04-19T00:53:59.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac34015030 con 0x7fac440a1ec0 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: dump: 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.625 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac34011aa0 con 0x7fac440a1ec0 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac34011c90 con 0x7fac440a1ec0 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: dump: 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.627 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac41ffb640 4 set_mon_vals no callback set 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 5 monclient: authenticate success, global_id 74211 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: _renew_subs 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac44141670 con 0x7fac440a1ec0 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 client.74211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: _renew_subs 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fac441240b0 con 0x7fac440a1ec0 2024-04-19T00:53:59.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 librados: init done 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 0x7fac4413fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fac4406d140 con 0x7fac440a22a0 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fac34018070 con 0x7fac440a1ec0 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 0x7fac4413fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient: get_auth_request con 0x7fac440a22a0 auth_method 0 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac24038e10 0x7fac2403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 10 monclient(hunting): handle_auth_done global_id 64399 payload 824 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac4409f4a0 con 0x7fac440a22a0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7fac34016070 con 0x7fac440a1ec0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 client.74211.objecter ms_dispatch 0x7fac44000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 3 client.74211.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 3 client.74211.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 20 client.74211.objecter dump_active .. 0 homeless 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 client.74211.objecter ms_handle_connect 0x7fac440a22a0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 client.74211.objecter resend_mon_ops 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac437fe640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 0x7fac4413fee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f70 tx=0x7fac2c00e940 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac24038e10 0x7fac2403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac43fff640 10 monclient: get_auth_request con 0x7fac24038e10 auth_method 0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac2c0070d0 con 0x7fac440a22a0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: dump: 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.631 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac2c018070 con 0x7fac440a22a0 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 msgr2=0x7fac4413fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 0x7fac4413fee0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f70 tx=0x7fac2c00e940 comp rx=0 tx=0).stop 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 msgr2=0x7fac4413fee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.606+0000 7fac3bfff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac440a22a0 0x7fac4413fee0 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f70 tx=0x7fac2c00e940 comp rx=0 tx=0).stop 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.610+0000 7fac43fff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac24038e10 0x7fac2403b2d0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0af000 tx=0x7fac3c0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.610+0000 7fac3bfff640 10 client.74211.objecter ms_handle_connect 0x7fac24038e10 2024-04-19T00:53:59.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:59.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 0x7fac4413bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fac44140830 con 0x7fac44140420 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 0x7fac4413bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient: get_auth_request con 0x7fac44140420 auth_method 0 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 10 monclient(hunting): handle_auth_done global_id 64408 payload 824 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 1 -- 172.21.15.43:0/2285441858 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac4409ef70 con 0x7fac44140420 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 client.74211.objecter ms_handle_connect 0x7fac44140420 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 client.74211.objecter resend_mon_ops 2024-04-19T00:53:59.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac437fe640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 0x7fac4413bda0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f70 tx=0x7fac2c000ee0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac2c007540 con 0x7fac44140420 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 monclient: dump: 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.636 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fac2c0070d0 con 0x7fac44140420 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 msgr2=0x7fac4413bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 0x7fac4413bda0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f70 tx=0x7fac2c000ee0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 msgr2=0x7fac4413bda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac3bfff640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac44140420 0x7fac4413bda0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fac2c000f70 tx=0x7fac2c000ee0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 165886, 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:59.638 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.639 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.640 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:59.641 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:59.642 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.643 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:59.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:59.649 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.650 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.652 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.654 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.656 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:59.657 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.658 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.663 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.664 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.665 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:59.665 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 10 librados: watch_flush enter 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 10 librados: watch_flush exit 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 20 client.74211.objecter shutdown clearing up homeless session... 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 10 client.74211.objecter successfully canceled tick 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac24038e10 msgr2=0x7fac2403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fac24038e10 0x7fac2403b2d0 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0af000 tx=0x7fac3c0a7070 comp rx=0 tx=0).stop 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 10 monclient: shutdown 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fac440a1ec0 msgr2=0x7fac441d1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 --2- 172.21.15.43:0/2285441858 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fac440a1ec0 0x7fac441d1360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac3400b410 tx=0x7fac3400b8d0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac43fff640 1 -- 172.21.15.43:0/2285441858 reap_dead start 2024-04-19T00:53:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac43fff640 1 -- 172.21.15.43:0/2285441858 reap_dead start 2024-04-19T00:53:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 shutdown_connections 2024-04-19T00:53:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.614+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 >> 172.21.15.43:0/2285441858 conn(0x7fac440a8600 msgr2=0x7fac440a39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.618+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 shutdown_connections 2024-04-19T00:53:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.618+0000 7fac49ff2640 1 -- 172.21.15.43:0/2285441858 wait complete. 2024-04-19T00:53:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.618+0000 7fac49ff2640 1 librados: shutdown 2024-04-19T00:53:59.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:59.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:53:59.669 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:53:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient: get_monmap_and_config 2024-04-19T00:53:59.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient: monmap: 2024-04-19T00:53:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:59.787260+0000 2024-04-19T00:53:59.785 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:59.787260+0000 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:59.786 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 Processor -- start 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 -- start start 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient: init 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient: _add_conns ranks=[5,10,2,7,9,6,8,0,4,3,1] 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16f4138a50 0x7f16f4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): picked mon.noname-h con 0x7f16f4138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f16f40a5bb0 0x7f16f4139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): picked mon.noname-f con 0x7f16f40a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 0x7f16f4137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): picked mon.noname-b con 0x7f16f4137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f16f40a1020 con 0x7f16f40a5bb0 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f16f41398b0 con 0x7f16f4137960 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f16f40a0780 con 0x7f16f4138a50 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f8f5b640 10 monclient(hunting): authenticate will time out at 1509.914325s 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 0x7f16f4137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 0x7f16f4137de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40714/0 (socket says 172.21.15.43:40714) 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 -- 172.21.15.43:0/4227703757 learned_addr learned my addr 172.21.15.43:0/4227703757 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): get_auth_request con 0x7f16f4137960 auth_method 0 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f259c640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16f4138a50 0x7f16f4138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f259c640 10 monclient(hunting): get_auth_request con 0x7f16f4138a50 auth_method 0 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f1d9b640 1 --2- 172.21.15.43:0/4227703757 >> v2:172.21.15.161:3300/0 conn(0x7f16f40a5bb0 0x7f16f4139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f1d9b640 10 monclient(hunting): get_auth_request con 0x7f16f40a5bb0 auth_method 0 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient(hunting): handle_auth_done global_id 74198 payload 274 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 -- 172.21.15.43:0/4227703757 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16f4138a50 msgr2=0x7f16f4138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16f4138a50 0x7f16f4138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 -- 172.21.15.43:0/4227703757 >> v2:172.21.15.161:3300/0 conn(0x7f16f40a5bb0 msgr2=0x7f16f4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 --2- 172.21.15.43:0/4227703757 >> v2:172.21.15.161:3300/0 conn(0x7f16f40a5bb0 0x7f16f4139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 monclient: found mon.noname-b 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f2d9d640 1 -- 172.21.15.43:0/4227703757 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f40a0d40 con 0x7f16f4137960 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.782+0000 7f16f259c640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16f4138a50 0x7f16f4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:53:59.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f2d9d640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 0x7f16f4137de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009e50 tx=0x7f16ec01c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c27f2235c7449bf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 1 -- 172.21.15.43:0/4227703757 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16ec0090d0 con 0x7f16f4137960 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: dump: 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.801 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 1 -- 172.21.15.43:0/4227703757 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec0a5b70 con 0x7f16f4137960 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 5 monclient: authenticate success, global_id 74198 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 1 -- 172.21.15.43:0/4227703757 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16ec0a5d60 con 0x7f16f4137960 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 4 set_mon_vals no callback set 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f159a640 10 monclient: dump: 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.803 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: shutdown 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- 172.21.15.43:0/4227703757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 msgr2=0x7f16f4137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 0x7f16f4137de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009e50 tx=0x7f16ec01c8c0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- 172.21.15.43:0/4227703757 shutdown_connections 2024-04-19T00:53:59.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f16f4137960 0x7f16f4137de0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/4227703757 >> v2:172.21.15.161:3300/0 conn(0x7f16f40a5bb0 0x7f16f4139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/4227703757 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16f4138a50 0x7f16f4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- 172.21.15.43:0/4227703757 >> 172.21.15.43:0/4227703757 conn(0x7f16f40a7cb0 msgr2=0x7f16f40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- 172.21.15.43:0/4227703757 shutdown_connections 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- 172.21.15.43:0/4227703757 wait complete. 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: monmap: 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:59.805 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 librados: starting msgr at 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 librados: starting objecter 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 Processor -- start 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- start start 2024-04-19T00:53:59.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 librados: setting wanted keys 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 librados: calling monclient init 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: init 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient: _add_conns ranks=[8,3,7,6,1,0,5,4,2] 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16f40a5bb0 0x7f16f41d0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): picked mon.noname-e con 0x7f16f40a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16f4137960 0x7f16f4136950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): picked mon.noname-g con 0x7f16f4137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16f4138a50 0x7f16f413c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): picked mon.noname-i con 0x7f16f4138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f16f40a2100 con 0x7f16f4137960 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f16f40a1f80 con 0x7f16f4138a50 2024-04-19T00:53:59.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f16f4142810 con 0x7f16f40a5bb0 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f8f5b640 10 monclient(hunting): authenticate will time out at 1509.919774s 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16f40a5bb0 0x7f16f41d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16f40a5bb0 0x7f16f41d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:54510/0 (socket says 172.21.15.43:54510) 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 1 -- 172.21.15.43:0/2499456661 learned_addr learned my addr 172.21.15.43:0/2499456661 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 10 monclient(hunting): get_auth_request con 0x7f16f40a5bb0 auth_method 0 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.786+0000 7f16f259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16f4138a50 0x7f16f413c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 10 monclient(hunting): get_auth_request con 0x7f16f4138a50 auth_method 0 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16f4137960 0x7f16f4136950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): get_auth_request con 0x7f16f4137960 auth_method 0 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 10 monclient(hunting): handle_auth_done global_id 23345 payload 824 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16f4138a50 msgr2=0x7f16f413c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16f4138a50 0x7f16f413c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16f4137960 msgr2=0x7f16f4136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f16f4137960 0x7f16f4136950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 monclient: found mon.noname-e 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f4122010 con 0x7f16f40a5bb0 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f259c640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16f40a5bb0 0x7f16f41d0910 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f16e400d800 tx=0x7f16e400dcc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 client.?.objecter ms_handle_connect 0x7f16f40a5bb0 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e400bb30 con 0x7f16f40a5bb0 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: dump: 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.810 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e4019070 con 0x7f16f40a5bb0 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 5 monclient: authenticate success, global_id 23345 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 monclient: _renew_subs 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f413e370 con 0x7f16f40a5bb0 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e400bd10 con 0x7f16f40a5bb0 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: dump: 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f0d99640 4 set_mon_vals no callback set 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 client.23345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 monclient: _renew_subs 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f4121d30 con 0x7f16f40a5bb0 2024-04-19T00:53:59.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 1 librados: init done 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 0x7f16f413cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f16f406d140 con 0x7f16f4136e90 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 0x7f16f413cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient: get_auth_request con 0x7f16f4136e90 auth_method 0 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f16e400b6c0 con 0x7f16f40a5bb0 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_done global_id 74220 payload 824 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16d4038e10 0x7f16d403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16f413f110 con 0x7f16f4136e90 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f16e400f040 con 0x7f16f40a5bb0 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 client.23345.objecter ms_dispatch 0x7f16f4000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 3 client.23345.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 3 client.23345.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 20 client.23345.objecter dump_active .. 0 homeless 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f1d9b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 0x7f16f413cc80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16e800f480 tx=0x7f16e800f940 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 client.23345.objecter ms_handle_connect 0x7f16f4136e90 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 client.23345.objecter resend_mon_ops 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e8019020 con 0x7f16f4136e90 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:59.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: dump: 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.816 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16d4038e10 0x7f16d403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 10 monclient: get_auth_request con 0x7f16d4038e10 auth_method 0 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f16e8014670 con 0x7f16f4136e90 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 msgr2=0x7f16f413cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 0x7f16f413cc80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16e800f480 tx=0x7f16e800f940 comp rx=0 tx=0).stop 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 msgr2=0x7f16f413cc80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f4136e90 0x7f16f413cc80 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f16e800f480 tx=0x7f16e800f940 comp rx=0 tx=0).stop 2024-04-19T00:53:59.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16e2ffd640 10 client.23345.objecter ms_handle_connect 0x7f16d4038e10 2024-04-19T00:53:59.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.790+0000 7f16f2d9d640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16d4038e10 0x7f16d403b2d0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0ae000 tx=0x7f16ec0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:53:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 0x7f16f409c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f16f413d5d0 con 0x7f16f413d1c0 2024-04-19T00:53:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 0x7f16f409c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient: get_auth_request con 0x7f16f413d1c0 auth_method 0 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 10 monclient(hunting): handle_auth_done global_id 74229 payload 824 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 1 -- 172.21.15.43:0/2499456661 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16f4076190 con 0x7f16f413d1c0 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 client.23345.objecter ms_handle_connect 0x7f16f413d1c0 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 client.23345.objecter resend_mon_ops 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f1d9b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 0x7f16f409c1c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16e800f480 tx=0x7f16e80046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16e8019040 con 0x7f16f413d1c0 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 monclient: dump: 2024-04-19T00:53:59.820 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f16e8004020 con 0x7f16f413d1c0 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 msgr2=0x7f16f409c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 0x7f16f409c1c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16e800f480 tx=0x7f16e80046d0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 msgr2=0x7f16f409c1c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16e2ffd640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f16f413d1c0 0x7f16f409c1c0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16e800f480 tx=0x7f16e80046d0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:59.822 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19411, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:53:59.823 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.824 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.825 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:53:59.826 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:53:59.827 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:53:59.831 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:53:59.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.833 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:53:59.835 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.836 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:53:59.838 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.842 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.843 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.844 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:53:59.845 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:53:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 10 librados: watch_flush enter 2024-04-19T00:53:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 10 librados: watch_flush exit 2024-04-19T00:53:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 20 client.23345.objecter shutdown clearing up homeless session... 2024-04-19T00:53:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 10 client.23345.objecter successfully canceled tick 2024-04-19T00:53:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16d4038e10 msgr2=0x7f16d403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16d4038e10 0x7f16d403b2d0 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0ae000 tx=0x7f16ec0a7070 comp rx=0 tx=0).stop 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 10 monclient: shutdown 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16f40a5bb0 msgr2=0x7f16f41d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 --2- 172.21.15.43:0/2499456661 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16f40a5bb0 0x7f16f41d0910 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f16e400d800 tx=0x7f16e400dcc0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f2d9d640 1 -- 172.21.15.43:0/2499456661 reap_dead start 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f2d9d640 1 -- 172.21.15.43:0/2499456661 reap_dead start 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 shutdown_connections 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 >> 172.21.15.43:0/2499456661 conn(0x7f16f40a7cb0 msgr2=0x7f16f406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 shutdown_connections 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 -- 172.21.15.43:0/2499456661 wait complete. 2024-04-19T00:53:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.798+0000 7f16f8f5b640 1 librados: shutdown 2024-04-19T00:53:59.849 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:53:59.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:53:59.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:53:59.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.962+0000 7f0686143640 10 monclient: get_monmap_and_config 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.962+0000 7f0686143640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.962+0000 7f0686143640 10 monclient: monmap: 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:53:59.968611+0000 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:53:59.968611+0000 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.966 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:53:59.967 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.962+0000 7f0686143640 1 Processor -- start 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.962+0000 7f0686143640 1 -- start start 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.962+0000 7f0686143640 10 monclient: init 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient: _add_conns ranks=[3,9,1,7,8,10,2,5,4,6,0] 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a1fc0 0x7f06800a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): picked mon.noname-g con 0x7f06800a1fc0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06800a1280 0x7f06800a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): picked mon.noname-a con 0x7f06800a1280 addr v2:172.21.15.43:3300/0 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f06800a43c0 addr v1:172.21.15.161:6789/0 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06800a05e0 con 0x7f06800a43c0 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f068006c1a0 con 0x7f06800a1280 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f06800a08c0 con 0x7f06800a1fc0 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient(hunting): authenticate will time out at 1510.096591s 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06800a1280 0x7f06800a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f06800a1280 0x7f06800a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42884/0 (socket says 172.21.15.43:42884) 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 -- 172.21.15.43:0/4257009206 learned_addr learned my addr 172.21.15.43:0/4257009206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): get_auth_request con 0x7f06800a1280 auth_method 0 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient(hunting): handle_auth_done global_id 64584 payload 274 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 -- 172.21.15.43:0/4257009206 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a1fc0 msgr2=0x7f06800a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067f7fe640 1 --2- 172.21.15.43:0/4257009206 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a1fc0 0x7f06800a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 --2- 172.21.15.43:0/4257009206 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a1fc0 0x7f06800a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 -- 172.21.15.43:0/4257009206 >> v1:172.21.15.161:6789/0 conn(0x7f06800a43c0 legacy=0x7f068013fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 monclient: found mon.noname-a 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:53:59.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 -- 172.21.15.43:0/4257009206 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06800a0020 con 0x7f06800a1280 2024-04-19T00:53:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067effd640 1 --2- 172.21.15.43:0/4257009206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06800a1280 0x7f06800a3e80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9103c2ed57d84172 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 1 -- 172.21.15.43:0/4257009206 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06740b0070 con 0x7f06800a1280 2024-04-19T00:53:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:53:59.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: dump: 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.982 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 1 -- 172.21.15.43:0/4257009206 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740aa810 con 0x7f06800a1280 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 1 -- 172.21.15.43:0/4257009206 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06740a9300 con 0x7f06800a1280 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 5 monclient: authenticate success, global_id 64584 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient: get_monmap_and_config success 2024-04-19T00:53:59.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 4 set_mon_vals no callback set 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f067e7fc640 10 monclient: dump: 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.984 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 10 monclient: shutdown 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 -- 172.21.15.43:0/4257009206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06800a1280 msgr2=0x7f06800a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 --2- 172.21.15.43:0/4257009206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06800a1280 0x7f06800a3e80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f067401c3c0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.966+0000 7f0686143640 1 -- 172.21.15.43:0/4257009206 shutdown_connections 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 --2- 172.21.15.43:0/4257009206 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06800a1280 0x7f06800a3e80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 --2- 172.21.15.43:0/4257009206 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a1fc0 0x7f06800a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- 172.21.15.43:0/4257009206 >> 172.21.15.43:0/4257009206 conn(0x7f06800a85c0 msgr2=0x7f06800a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- 172.21.15.43:0/4257009206 shutdown_connections 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- 172.21.15.43:0/4257009206 wait complete. 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: build_initial_monmap 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: monmap: 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:53:59.986 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 librados: starting msgr at 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 librados: starting objecter 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 Processor -- start 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- start start 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 librados: setting wanted keys 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 librados: calling monclient init 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: init 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: _reopen_session rank -1 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: _add_conns ranks=[7,6,3,5,8,4,2,0,1] 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06800a1280 0x7f06801d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): picked mon.noname-i con 0x7f06800a1280 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06800a1fc0 0x7f06801d1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): picked mon.noname-d con 0x7f06800a1fc0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a43c0 0x7f06801265e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): picked mon.noname-g con 0x7f06800a43c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0680143a70 con 0x7f06800a43c0 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0680143bf0 con 0x7f06800a1fc0 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0680143d70 con 0x7f06800a1280 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): _renew_subs 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): authenticate will time out at 1510.101678s 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06800a1fc0 0x7f06801d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06800a1fc0 0x7f06801d1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33272/0 (socket says 172.21.15.43:33272) 2024-04-19T00:53:59.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 -- 172.21.15.43:0/3694724737 learned_addr learned my addr 172.21.15.43:0/3694724737 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): get_auth_request con 0x7f06800a1fc0 auth_method 0 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06800a1280 0x7f06801d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): get_auth_request con 0x7f06800a1280 auth_method 0 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067ffff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a43c0 0x7f06801265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067ffff640 10 monclient(hunting): get_auth_request con 0x7f06800a43c0 auth_method 0 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient(hunting): handle_auth_done global_id 74238 payload 824 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06800a1280 msgr2=0x7f06801d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f06800a1280 0x7f06801d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a43c0 msgr2=0x7f06801265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06800a43c0 0x7f06801265e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient: _finish_hunting 0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 monclient: found mon.noname-d 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0680128e10 con 0x7f06800a1fc0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067effd640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06800a1fc0 0x7f06801d1480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f06740191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 client.?.objecter ms_handle_connect 0x7f06800a1fc0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06740b9070 con 0x7f06800a1fc0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: dump: 2024-04-19T00:53:59.992 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.993 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: _finish_auth 0 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: _check_auth_tickets 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740044b0 con 0x7f06800a1fc0 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06740b0040 con 0x7f06800a1fc0 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 5 monclient: authenticate success, global_id 74238 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067d7fa640 4 set_mon_vals no callback set 2024-04-19T00:53:59.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 10 monclient: dump: 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:53:59.995 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: _renew_subs 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068012a5a0 con 0x7f06800a1fc0 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 client.74238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: _renew_subs 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f06800732f0 con 0x7f06800a1fc0 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 librados: init done 2024-04-19T00:53:59.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 0x7f0680126b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f06800a1c00 con 0x7f068012a0a0 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0674004040 con 0x7f06800a1fc0 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 0x7f0680126b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient: get_auth_request con 0x7f068012a0a0 auth_method 0 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:53:59.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:53:59.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.970+0000 7f0667fff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f066c038e10 0x7f066c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:53:59.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f06740e2650 con 0x7f06800a1fc0 2024-04-19T00:53:59.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f067f7fe640 10 monclient(hunting): handle_auth_done global_id 23354 payload 824 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 client.74238.objecter ms_dispatch 0x7f0680000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 3 client.74238.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 3 client.74238.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f067f7fe640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0680128b60 con 0x7f068012a0a0 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 20 client.74238.objecter dump_active .. 0 homeless 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 client.74238.objecter ms_handle_connect 0x7f068012a0a0 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 client.74238.objecter resend_mon_ops 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 0x7f0680126b20 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f067000fd40 tx=0x7f067000d750 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f067ffff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f066c038e10 0x7f066c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f067ffff640 10 monclient: get_auth_request con 0x7f066c038e10 auth_method 0 2024-04-19T00:53:59.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f067000a9f0 con 0x7f068012a0a0 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 monclient: dump: 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.000 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f067000a580 con 0x7f068012a0a0 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 msgr2=0x7f0680126b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 0x7f0680126b20 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f067000fd40 tx=0x7f067000d750 comp rx=0 tx=0).stop 2024-04-19T00:54:00.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 msgr2=0x7f0680126b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f068012a0a0 0x7f0680126b20 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f067ffff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f066c038e10 0x7f066c03b2d0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f0680141190 tx=0x7f0678008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.974+0000 7f0667fff640 10 client.74238.objecter ms_handle_connect 0x7f066c038e10 2024-04-19T00:54:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f0686143640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f0686143640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f0686143640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f068006d140 con 0x7f0680127060 2024-04-19T00:54:00.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient: get_auth_request con 0x7f0680127060 auth_method 0 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 msgr2=0x7f0680127470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 msgr2=0x7f0680127470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:00.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:53:59.978+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:00.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient: get_auth_request con 0x7f0680127060 auth_method 0 2024-04-19T00:54:00.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 10 monclient(hunting): handle_auth_done global_id 23372 payload 824 2024-04-19T00:54:00.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 1 -- 172.21.15.43:0/3694724737 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f06801279b0 con 0x7f0680127060 2024-04-19T00:54:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067f7fe640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f067000fd40 tx=0x7f0670004a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 client.74238.objecter ms_handle_connect 0x7f0680127060 2024-04-19T00:54:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 client.74238.objecter resend_mon_ops 2024-04-19T00:54:00.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f067001d290 con 0x7f0680127060 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 monclient: dump: 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.187 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9445 (secure 0 0 0) 0x7f067001d040 con 0x7f0680127060 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 msgr2=0x7f0680127470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f067000fd40 tx=0x7f0670004a50 comp rx=0 tx=0).stop 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 msgr2=0x7f0680127470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0667fff640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0680127060 0x7f0680127470 secure :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f067000fd40 tx=0x7f0670004a50 comp rx=0 tx=0).stop 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 583694, 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:00.189 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.190 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:00.192 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.193 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.194 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:00.195 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.196 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:00.197 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.198 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.199 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.200 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.201 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.202 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.203 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.206 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:00.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.210 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.211 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.212 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:00.212 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 10 librados: watch_flush enter 2024-04-19T00:54:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 10 librados: watch_flush exit 2024-04-19T00:54:00.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 20 client.74238.objecter shutdown clearing up homeless session... 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 10 client.74238.objecter successfully canceled tick 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f066c038e10 msgr2=0x7f066c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f066c038e10 0x7f066c03b2d0 secure :-1 s=READY pgs=3503 cs=0 l=1 rev1=1 crypto rx=0x7f0680141190 tx=0x7f0678008040 comp rx=0 tx=0).stop 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 10 monclient: shutdown 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06800a1fc0 msgr2=0x7f06801d1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f0686143640 1 --2- 172.21.15.43:0/3694724737 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f06800a1fc0 0x7f06801d1480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0674009870 tx=0x7f06740191c0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067ffff640 1 -- 172.21.15.43:0/3694724737 reap_dead start 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.182+0000 7f067ffff640 1 -- 172.21.15.43:0/3694724737 reap_dead start 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.186+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 shutdown_connections 2024-04-19T00:54:00.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.186+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 >> 172.21.15.43:0/3694724737 conn(0x7f06800a85c0 msgr2=0x7f06800a72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.186+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 shutdown_connections 2024-04-19T00:54:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.186+0000 7f0686143640 1 -- 172.21.15.43:0/3694724737 wait complete. 2024-04-19T00:54:00.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.186+0000 7f0686143640 1 librados: shutdown 2024-04-19T00:54:00.216 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:54:00.216 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:54:00.216 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:54:00.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient: get_monmap_and_config 2024-04-19T00:54:00.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient: monmap: 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:00.351516+0000 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:00.351516+0000 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.349 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:00.350 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 Processor -- start 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 -- start start 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient: init 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient: _add_conns ranks=[9,2,10,4,0,1,7,5,8,3,6] 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb170138a50 0x7fb170138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): picked mon.noname-a con 0x7fb170138a50 addr v2:172.21.15.43:3300/0 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1700a5bb0 0x7fb170139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): picked mon.noname-b con 0x7fb1700a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb170137960 0x7fb170137e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): picked mon.noname-f con 0x7fb170137960 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fb1700a1020 con 0x7fb170138a50 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb1701398b0 con 0x7fb170137960 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb1700a0780 con 0x7fb1700a5bb0 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb1750ad640 10 monclient(hunting): authenticate will time out at 1510.478702s 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb170138a50 0x7fb170138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1700a5bb0 0x7fb170139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb170138a50 0x7fb170138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42888/0 (socket says 172.21.15.43:42888) 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 -- 172.21.15.43:0/1390031513 learned_addr learned my addr 172.21.15.43:0/1390031513 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): get_auth_request con 0x7fb170138a50 auth_method 0 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): get_auth_request con 0x7fb1700a5bb0 auth_method 0 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16f59e640 1 --2- 172.21.15.43:0/1390031513 >> v2:172.21.15.161:3300/0 conn(0x7fb170137960 0x7fb170137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16f59e640 10 monclient(hunting): get_auth_request con 0x7fb170137960 auth_method 0 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_done global_id 64593 payload 274 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 -- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1700a5bb0 msgr2=0x7fb170139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1700a5bb0 0x7fb170139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 -- 172.21.15.43:0/1390031513 >> v2:172.21.15.161:3300/0 conn(0x7fb170137960 msgr2=0x7fb170137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1390031513 >> v2:172.21.15.161:3300/0 conn(0x7fb170137960 0x7fb170137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 monclient: found mon.noname-a 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.346+0000 7fb16ed9d640 1 -- 172.21.15.43:0/1390031513 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1700a0d40 con 0x7fb170138a50 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb170138a50 0x7fb170138e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a7160 tx=0x7fb16001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b747d18cbb0082f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 1 -- 172.21.15.43:0/1390031513 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1600b2070 con 0x7fb170138a50 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: dump: 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.358 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 1 -- 172.21.15.43:0/1390031513 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1600ab7b0 con 0x7fb170138a50 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 5 monclient: authenticate success, global_id 64593 2024-04-19T00:54:00.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 1 -- 172.21.15.43:0/1390031513 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1600abe50 con 0x7fb170138a50 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 4 set_mon_vals no callback set 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb16dd9b640 10 monclient: dump: 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.360 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 10 monclient: shutdown 2024-04-19T00:54:00.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 -- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb170138a50 msgr2=0x7fb170138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb170138a50 0x7fb170138e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a7160 tx=0x7fb16001c3b0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 -- 172.21.15.43:0/1390031513 shutdown_connections 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb1700a5bb0 0x7fb170139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1390031513 >> v2:172.21.15.161:3300/0 conn(0x7fb170137960 0x7fb170137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1390031513 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb170138a50 0x7fb170138e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 -- 172.21.15.43:0/1390031513 >> 172.21.15.43:0/1390031513 conn(0x7fb1700a7cb0 msgr2=0x7fb1700a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 -- 172.21.15.43:0/1390031513 shutdown_connections 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 -- 172.21.15.43:0/1390031513 wait complete. 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 10 monclient: monmap: 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:00.362 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.363 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 librados: starting msgr at 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 librados: starting objecter 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 Processor -- start 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 -- start start 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 librados: setting wanted keys 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 1 librados: calling monclient init 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.350+0000 7fb1750ad640 10 monclient: init 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient: _add_conns ranks=[8,6,2,3,7,0,4,1,5] 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb1700a5bb0 0x7fb1701d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): picked mon.noname-e con 0x7fb1700a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb170137960 0x7fb170136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): picked mon.noname-d con 0x7fb170137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb170138a50 0x7fb17013c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): picked mon.noname-b con 0x7fb170138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb1700a2100 con 0x7fb170138a50 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb1700a1f80 con 0x7fb170137960 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb170142810 con 0x7fb1700a5bb0 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): authenticate will time out at 1510.484176s 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb170138a50 0x7fb17013c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb170137960 0x7fb170136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb170137960 0x7fb170136a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33274/0 (socket says 172.21.15.43:33274) 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 -- 172.21.15.43:0/1472231687 learned_addr learned my addr 172.21.15.43:0/1472231687 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): get_auth_request con 0x7fb170137960 auth_method 0 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb170138a50 0x7fb17013c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40738/0 (socket says 172.21.15.43:40738) 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): get_auth_request con 0x7fb170138a50 auth_method 0 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb1700a5bb0 0x7fb1701d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): get_auth_request con 0x7fb1700a5bb0 auth_method 0 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient(hunting): handle_auth_done global_id 74247 payload 824 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb1700a5bb0 msgr2=0x7fb1701d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb1700a5bb0 0x7fb1701d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb170138a50 msgr2=0x7fb17013c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb170138a50 0x7fb17013c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 monclient: found mon.noname-d 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb170122410 con 0x7fb170137960 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb170138a50 0x7fb17013c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:54:00.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16e59c640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb170137960 0x7fb170136a70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb16400e900 tx=0x7fb16400edc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 client.?.objecter ms_handle_connect 0x7fb170137960 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb164017070 con 0x7fb170137960 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: dump: 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.376 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16400ce10 con 0x7fb170137960 2024-04-19T00:54:00.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1640138e0 con 0x7fb170137960 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 5 monclient: authenticate success, global_id 74247 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient: _renew_subs 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17009ee60 con 0x7fb170137960 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: dump: 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.378 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16cd99640 4 set_mon_vals no callback set 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 client.74247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient: _renew_subs 2024-04-19T00:54:00.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fb170121b70 con 0x7fb170137960 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 librados: init done 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 0x7fb17013cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb17006d140 con 0x7fb170136fb0 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb16401a070 con 0x7fb170137960 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb15c038e10 0x7fb15c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7fb16400f040 con 0x7fb170137960 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 0x7fb17013cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 client.74247.objecter ms_dispatch 0x7fb170000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 3 client.74247.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 3 client.74247.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:54:00.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient: get_auth_request con 0x7fb170136fb0 auth_method 0 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 20 client.74247.objecter dump_active .. 0 homeless 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb15c038e10 0x7fb15c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 10 monclient: get_auth_request con 0x7fb15c038e10 auth_method 0 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_done global_id 64414 payload 824 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1700008e0 con 0x7fb170136fb0 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 client.74247.objecter ms_handle_connect 0x7fb170136fb0 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 0x7fb17013cd40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a72d0 tx=0x7fb16001abf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 client.74247.objecter resend_mon_ops 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 client.74247.objecter ms_handle_connect 0x7fb15c038e10 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb16f59e640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb15c038e10 0x7fb15c03b2d0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fb1701378a0 tx=0x7fb168009820 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1600bb030 con 0x7fb170136fb0 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.354+0000 7fb1577fe640 10 monclient: dump: 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb1600b3040 con 0x7fb170136fb0 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 msgr2=0x7fb17013cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 0x7fb17013cd40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a72d0 tx=0x7fb16001abf0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 msgr2=0x7fb17013cd40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.358+0000 7fb1577fe640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170136fb0 0x7fb17013cd40 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb1700a72d0 tx=0x7fb16001abf0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb1750ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 0x7fb17013d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb1750ad640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb170121800 con 0x7fb170121d70 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 0x7fb17013d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient: get_auth_request con 0x7fb170121d70 auth_method 0 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 10 monclient(hunting): handle_auth_done global_id 64423 payload 824 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 1 -- 172.21.15.43:0/1472231687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb17013efe0 con 0x7fb170121d70 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb1577fe640 10 client.74247.objecter ms_handle_connect 0x7fb170121d70 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 10 client.74247.objecter resend_mon_ops 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.362+0000 7fb16ed9d640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 0x7fb17013d280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb16001cc50 tx=0x7fb160019150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb1600bb040 con 0x7fb170121d70 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 10 monclient: dump: 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fb1600c1810 con 0x7fb170121d70 2024-04-19T00:54:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 msgr2=0x7fb17013d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 0x7fb17013d280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb16001cc50 tx=0x7fb160019150 comp rx=0 tx=0).stop 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 msgr2=0x7fb17013d280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1577fe640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb170121d70 0x7fb17013d280 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb16001cc50 tx=0x7fb160019150 comp rx=0 tx=0).stop 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:00.387 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 166633, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.388 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.389 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:00.391 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.392 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:00.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.395 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.397 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:00.398 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.399 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.401 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.404 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.406 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.407 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.408 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.409 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:00.410 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 10 librados: watch_flush enter 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 10 librados: watch_flush exit 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 20 client.74247.objecter shutdown clearing up homeless session... 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 10 client.74247.objecter successfully canceled tick 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb15c038e10 msgr2=0x7fb15c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb15c038e10 0x7fb15c03b2d0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 crypto rx=0x7fb1701378a0 tx=0x7fb168009820 comp rx=0 tx=0).stop 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 10 monclient: shutdown 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb170137960 msgr2=0x7fb170136a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 --2- 172.21.15.43:0/1472231687 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb170137960 0x7fb170136a70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb16400e900 tx=0x7fb16400edc0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb16f59e640 1 -- 172.21.15.43:0/1472231687 reap_dead start 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb16f59e640 1 -- 172.21.15.43:0/1472231687 reap_dead start 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 shutdown_connections 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 >> 172.21.15.43:0/1472231687 conn(0x7fb1700a7cb0 msgr2=0x7fb1700a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 shutdown_connections 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 -- 172.21.15.43:0/1472231687 wait complete. 2024-04-19T00:54:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.366+0000 7fb1750ad640 1 librados: shutdown 2024-04-19T00:54:00.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:54:00.414 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:54:00.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient: get_monmap_and_config 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient: monmap: 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:00.534690+0000 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:00.534690+0000 2024-04-19T00:54:00.532 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:00.533 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:00.534 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 Processor -- start 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 -- start start 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient: init 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient: _add_conns ranks=[6,4,2,8,9,7,3,1,5,10,0] 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac138a50 0x7f8eac138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): picked mon.noname-d con 0x7f8eac138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac0a5bb0 0x7f8eac139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): picked mon.noname-c con 0x7f8eac0a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 0x7f8eac137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): picked mon.noname-b con 0x7f8eac137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8eac0a1020 con 0x7f8eac137960 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8eac1398b0 con 0x7f8eac0a5bb0 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8eac0a0780 con 0x7f8eac138a50 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb28ea640 10 monclient(hunting): authenticate will time out at 1510.661758s 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 0x7f8eac137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac138a50 0x7f8eac138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 0x7f8eac137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40754/0 (socket says 172.21.15.43:40754) 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 -- 172.21.15.43:0/4210743208 learned_addr learned my addr 172.21.15.43:0/4210743208 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): get_auth_request con 0x7f8eac137960 auth_method 0 2024-04-19T00:54:00.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac0a5bb0 0x7f8eac139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): get_auth_request con 0x7f8eac138a50 auth_method 0 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): get_auth_request con 0x7f8eac0a5bb0 auth_method 0 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_done global_id 74225 payload 274 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 -- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac138a50 msgr2=0x7f8eac138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac138a50 0x7f8eac138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 -- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac0a5bb0 msgr2=0x7f8eac139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac0a5bb0 0x7f8eac139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 monclient: found mon.noname-b 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.530+0000 7f8eb0e87640 1 -- 172.21.15.43:0/4210743208 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eac0a0d40 con 0x7f8eac137960 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 0x7f8eac137dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009e80 tx=0x7f8ea401c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83296469985b5514 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 1 -- 172.21.15.43:0/4210743208 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea40090d0 con 0x7f8eac137960 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: dump: 2024-04-19T00:54:00.542 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.543 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 1 -- 172.21.15.43:0/4210743208 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea40a5b50 con 0x7f8eac137960 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 5 monclient: authenticate success, global_id 74225 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 4 set_mon_vals no callback set 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 1 -- 172.21.15.43:0/4210743208 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea40a5d40 con 0x7f8eac137960 2024-04-19T00:54:00.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaaffd640 10 monclient: dump: 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.545 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: shutdown 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 msgr2=0x7f8eac137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 0x7f8eac137dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009e80 tx=0x7f8ea401c8e0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- 172.21.15.43:0/4210743208 shutdown_connections 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8eac137960 0x7f8eac137dc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac0a5bb0 0x7f8eac139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/4210743208 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac138a50 0x7f8eac138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- 172.21.15.43:0/4210743208 >> 172.21.15.43:0/4210743208 conn(0x7f8eac0a7cb0 msgr2=0x7f8eac0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- 172.21.15.43:0/4210743208 shutdown_connections 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- 172.21.15.43:0/4210743208 wait complete. 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: monmap: 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:00.547 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 librados: starting msgr at 2024-04-19T00:54:00.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 librados: starting objecter 2024-04-19T00:54:00.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 Processor -- start 2024-04-19T00:54:00.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- start start 2024-04-19T00:54:00.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 librados: setting wanted keys 2024-04-19T00:54:00.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 librados: calling monclient init 2024-04-19T00:54:00.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: init 2024-04-19T00:54:00.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: _add_conns ranks=[6,7,4,5,2,8,3,1,0] 2024-04-19T00:54:00.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac0a5bb0 0x7f8eac0a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): picked mon.noname-d con 0x7f8eac0a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8eac137960 0x7f8eac09da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): picked mon.noname-i con 0x7f8eac137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac138a50 0x7f8eac09dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): picked mon.noname-c con 0x7f8eac138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8eac0a2100 con 0x7f8eac138a50 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8eac0a1f80 con 0x7f8eac0a5bb0 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8eac142810 con 0x7f8eac137960 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient(hunting): authenticate will time out at 1510.666928s 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac138a50 0x7f8eac09dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac138a50 0x7f8eac09dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:45010/0 (socket says 172.21.15.43:45010) 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 -- 172.21.15.43:0/58643454 learned_addr learned my addr 172.21.15.43:0/58643454 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): get_auth_request con 0x7f8eac138a50 auth_method 0 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac0a5bb0 0x7f8eac0a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): get_auth_request con 0x7f8eac0a5bb0 auth_method 0 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient(hunting): handle_auth_done global_id 64426 payload 824 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8eac137960 msgr2=0x7f8eac09da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8eac137960 0x7f8eac09da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac0a5bb0 msgr2=0x7f8eac0a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8eac0a5bb0 0x7f8eac0a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 monclient: found mon.noname-c 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eac09fc90 con 0x7f8eac138a50 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb0e87640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac138a50 0x7f8eac09dfb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40af040 tx=0x7f8ea4019680 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 client.?.objecter ms_handle_connect 0x7f8eac138a50 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea40bc020 con 0x7f8eac138a50 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: dump: 2024-04-19T00:54:00.560 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.561 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 5 monclient: authenticate success, global_id 64426 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ea4003e10 con 0x7f8eac138a50 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: _renew_subs 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eac09ee60 con 0x7f8eac138a50 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea40090d0 con 0x7f8eac138a50 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eb28ea640 10 client.64426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8eaa7fc640 4 set_mon_vals no callback set 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.534+0000 7f8ea8ff9640 10 monclient: dump: 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.563 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 10 monclient: _renew_subs 2024-04-19T00:54:00.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eac09bcc0 con 0x7f8eac138a50 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 1 librados: init done 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 0x7f8eac1358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8eac06d140 con 0x7f8eac1354c0 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8ea40bf020 con 0x7f8eac138a50 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e98038e40 0x7f8e9803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f8ea40e4330 con 0x7f8eac138a50 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 client.64426.objecter ms_dispatch 0x7f8eac000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 3 client.64426.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:54:00.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 3 client.64426.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 0x7f8eac1358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient: get_auth_request con 0x7f8eac1354c0 auth_method 0 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 20 client.64426.objecter dump_active .. 0 homeless 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eab7fe640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e98038e40 0x7f8e9803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eab7fe640 10 monclient: get_auth_request con 0x7f8e98038e40 auth_method 0 2024-04-19T00:54:00.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 10 monclient(hunting): handle_auth_done global_id 19290 payload 824 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8eac1d0c90 con 0x7f8eac1354c0 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eabfff640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 0x7f8eac1358a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0011890 tx=0x7f8ea0011d50 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 client.64426.objecter ms_handle_connect 0x7f8eac1354c0 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 client.64426.objecter resend_mon_ops 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 client.64426.objecter ms_handle_connect 0x7f8e98038e40 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8eab7fe640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e98038e40 0x7f8e9803b300 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7f8eac06e8c0 tx=0x7f8e9c0095c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea0019070 con 0x7f8eac1354c0 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 monclient: dump: 2024-04-19T00:54:00.567 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.568 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ea0004240 con 0x7f8eac1354c0 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 msgr2=0x7f8eac1358a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 0x7f8eac1358a0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0011890 tx=0x7f8ea0011d50 comp rx=0 tx=0).stop 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 msgr2=0x7f8eac1358a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.538+0000 7f8ea8ff9640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac1354c0 0x7f8eac1358a0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eb28ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 0x7f8eac12eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eb28ea640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8eac0a52b0 con 0x7f8eac135de0 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 0x7f8eac12eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient: get_auth_request con 0x7f8eac135de0 auth_method 0 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 10 monclient(hunting): handle_auth_done global_id 19299 payload 824 2024-04-19T00:54:00.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 1 -- 172.21.15.43:0/58643454 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8eac076190 con 0x7f8eac135de0 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8eabfff640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 0x7f8eac12eae0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0011890 tx=0x7f8ea00046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 client.64426.objecter ms_handle_connect 0x7f8eac135de0 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 client.64426.objecter resend_mon_ops 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8ea0019040 con 0x7f8eac135de0 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 monclient: dump: 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.571 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f8ea001e450 con 0x7f8eac135de0 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:00.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 msgr2=0x7f8eac12eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 0x7f8eac12eae0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0011890 tx=0x7f8ea00046d0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 msgr2=0x7f8eac12eae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.546+0000 7f8ea8ff9640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8eac135de0 0x7f8eac12eae0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:00.573 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 584034, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:54:00.574 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.575 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:00.577 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:00.578 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.579 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.581 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:00.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.583 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:00.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.590 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.594 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:00.595 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.596 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.597 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.598 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:00.599 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:00.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 10 librados: watch_flush enter 2024-04-19T00:54:00.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 10 librados: watch_flush exit 2024-04-19T00:54:00.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 20 client.64426.objecter shutdown clearing up homeless session... 2024-04-19T00:54:00.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 10 client.64426.objecter successfully canceled tick 2024-04-19T00:54:00.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e98038e40 msgr2=0x7f8e9803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e98038e40 0x7f8e9803b300 secure :-1 s=READY pgs=3505 cs=0 l=1 rev1=1 crypto rx=0x7f8eac06e8c0 tx=0x7f8e9c0095c0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 10 monclient: shutdown 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac138a50 msgr2=0x7f8eac09dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 --2- 172.21.15.43:0/58643454 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8eac138a50 0x7f8eac09dfb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40af040 tx=0x7f8ea4019680 comp rx=0 tx=0).stop 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb0e87640 1 -- 172.21.15.43:0/58643454 reap_dead start 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb0e87640 1 -- 172.21.15.43:0/58643454 reap_dead start 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 shutdown_connections 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 >> 172.21.15.43:0/58643454 conn(0x7f8eac0a7cb0 msgr2=0x7f8eac06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 shutdown_connections 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 -- 172.21.15.43:0/58643454 wait complete. 2024-04-19T00:54:00.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.550+0000 7f8eb28ea640 1 librados: shutdown 2024-04-19T00:54:00.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:54:00.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:54:00.603 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:54:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient: get_monmap_and_config 2024-04-19T00:54:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient: monmap: 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:00.723273+0000 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:00.723273+0000 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:00.722 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 Processor -- start 2024-04-19T00:54:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 -- start start 2024-04-19T00:54:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient: init 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient: _add_conns ranks=[2,3,5,8,1,0,9,7,10,4,6] 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): picked mon.noname-b con 0x7f00c0138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00c00a5b80 0x7f00c01395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): picked mon.noname-g con 0x7f00c00a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01358b0 0x7f00c0135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): picked mon.noname-h con 0x7f00c01358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00c00a4410 con 0x7f00c0138cd0 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f00c0139b30 con 0x7f00c00a5b80 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f00c00a3dc0 con 0x7f00c01358b0 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00c58f4640 10 monclient(hunting): authenticate will time out at 1510.850271s 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40764/0 (socket says 172.21.15.43:40764) 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 -- 172.21.15.43:0/3814444862 learned_addr learned my addr 172.21.15.43:0/3814444862 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): get_auth_request con 0x7f00c0138cd0 auth_method 0 2024-04-19T00:54:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00bf7fe640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01358b0 0x7f00c0135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00bf7fe640 10 monclient(hunting): get_auth_request con 0x7f00c01358b0 auth_method 0 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00be7fc640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00c00a5b80 0x7f00c01395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00be7fc640 10 monclient(hunting): get_auth_request con 0x7f00c00a5b80 auth_method 0 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00be7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient(hunting): handle_auth_done global_id 74234 payload 274 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 -- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01358b0 msgr2=0x7f00c0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01358b0 0x7f00c0135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 -- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00c00a5b80 msgr2=0x7f00c01395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00c00a5b80 0x7f00c01395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 monclient: found mon.noname-b 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.718+0000 7f00beffd640 1 -- 172.21.15.43:0/3814444862 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c00a40a0 con 0x7f00c0138cd0 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00beffd640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01390b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00b00098a0 tx=0x7f00b001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f057268691ebbf0b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 1 -- 172.21.15.43:0/3814444862 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00b00b0070 con 0x7f00c0138cd0 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: dump: 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 1 -- 172.21.15.43:0/3814444862 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b00aa810 con 0x7f00c0138cd0 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 5 monclient: authenticate success, global_id 74234 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 1 -- 172.21.15.43:0/3814444862 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00b00a9300 con 0x7f00c0138cd0 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 4 set_mon_vals no callback set 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bdffb640 10 monclient: dump: 2024-04-19T00:54:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.733 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: shutdown 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- 172.21.15.43:0/3814444862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 msgr2=0x7f00c01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01390b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00b00098a0 tx=0x7f00b001c3e0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- 172.21.15.43:0/3814444862 shutdown_connections 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01358b0 0x7f00c0135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00c00a5b80 0x7f00c01395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3814444862 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01390b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- 172.21.15.43:0/3814444862 >> 172.21.15.43:0/3814444862 conn(0x7f00c00ab2f0 msgr2=0x7f00c00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- 172.21.15.43:0/3814444862 shutdown_connections 2024-04-19T00:54:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- 172.21.15.43:0/3814444862 wait complete. 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: monmap: 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:00.742 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 librados: starting msgr at 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 librados: starting objecter 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 Processor -- start 2024-04-19T00:54:00.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- start start 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 librados: setting wanted keys 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 librados: calling monclient init 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: init 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient: _add_conns ranks=[1,4,2,6,7,8,0,5,3] 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c00a5b80 0x7f00c0134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): picked mon.noname-f con 0x7f00c00a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f00c01358b0 0x7f00c0135230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): picked mon.noname-c con 0x7f00c01358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): picked mon.noname-b con 0x7f00c0138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00c00a2010 con 0x7f00c00a5b80 2024-04-19T00:54:00.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00c00a1e90 con 0x7f00c0138cd0 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f00c0142aa0 con 0x7f00c01358b0 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00c58f4640 10 monclient(hunting): authenticate will time out at 1510.855646s 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:40768/0 (socket says 172.21.15.43:40768) 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 1 -- 172.21.15.43:0/3040686596 learned_addr learned my addr 172.21.15.43:0/3040686596 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): get_auth_request con 0x7f00c0138cd0 auth_method 0 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00be7fc640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f00c01358b0 0x7f00c0135230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00be7fc640 10 monclient(hunting): get_auth_request con 0x7f00c01358b0 auth_method 0 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00be7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00be7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00beffd640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c00a5b80 0x7f00c0134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.722+0000 7f00be7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00be7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00be7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): get_auth_request con 0x7f00c00a5b80 auth_method 0 2024-04-19T00:54:00.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 10 monclient(hunting): handle_auth_done global_id 74243 payload 824 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f00c01358b0 msgr2=0x7f00c0135230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f00c01358b0 0x7f00c0135230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c00a5b80 msgr2=0x7f00c0134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c00a5b80 0x7f00c0134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 monclient: found mon.noname-b 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c00a3590 con 0x7f00c0138cd0 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00c00a5b80 0x7f00c0134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:54:00.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bf7fe640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01d0d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00b8007e40 tx=0x7f00b800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 client.?.objecter ms_handle_connect 0x7f00c0138cd0 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00b8014890 con 0x7f00c0138cd0 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: dump: 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.748 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b8019070 con 0x7f00c0138cd0 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00b8013380 con 0x7f00c0138cd0 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: dump: 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.750 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00bd7fa640 4 set_mon_vals no callback set 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 5 monclient: authenticate success, global_id 74243 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 monclient: _renew_subs 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c00a3040 con 0x7f00c0138cd0 2024-04-19T00:54:00.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 client.74243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 monclient: _renew_subs 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c00a0810 con 0x7f00c0138cd0 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 1 librados: init done 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 0x7f00c01d1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f00c006d140 con 0x7f00c01d1280 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f00b8014420 con 0x7f00c0138cd0 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00ac038e10 0x7f00ac03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 0x7f00c01d1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f00b8049bf0 con 0x7f00c0138cd0 2024-04-19T00:54:00.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 client.74243.objecter ms_dispatch 0x7f00c0000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 3 client.74243.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 3 client.74243.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient: get_auth_request con 0x7f00c01d1280 auth_method 0 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 20 client.74243.objecter dump_active .. 0 homeless 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00be7fc640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00ac038e10 0x7f00ac03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00be7fc640 10 monclient: get_auth_request con 0x7f00ac038e10 auth_method 0 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 10 monclient(hunting): handle_auth_done global_id 74174 payload 824 2024-04-19T00:54:00.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00c01d1ba0 con 0x7f00c01d1280 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 client.74243.objecter ms_handle_connect 0x7f00ac038e10 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00be7fc640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00ac038e10 0x7f00ac03b2d0 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7f00c009e0d0 tx=0x7f00b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 client.74243.objecter ms_handle_connect 0x7f00c01d1280 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 client.74243.objecter resend_mon_ops 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00beffd640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 0x7f00c01d1660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009870 tx=0x7f00b00191e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00b00ba030 con 0x7f00c01d1280 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.726+0000 7f00a7fff640 10 monclient: dump: 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.754 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00b0003620 con 0x7f00c01d1280 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 msgr2=0x7f00c01d1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 0x7f00c01d1660 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009870 tx=0x7f00b00191e0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 msgr2=0x7f00c01d1660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.730+0000 7f00a7fff640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c01d1280 0x7f00c01d1660 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f00b0009870 tx=0x7f00b00191e0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00c58f4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 0x7f00c013c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00c58f4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f00c0068240 con 0x7f00c013bcc0 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 0x7f00c013c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient: get_auth_request con 0x7f00c013bcc0 auth_method 0 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 10 monclient(hunting): handle_auth_done global_id 74183 payload 824 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 1 -- 172.21.15.43:0/3040686596 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00c00693b0 con 0x7f00c013bcc0 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00beffd640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 0x7f00c013c0d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00b00098a0 tx=0x7f00b00a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00a7fff640 10 client.74243.objecter ms_handle_connect 0x7f00c013bcc0 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.734+0000 7f00a7fff640 10 client.74243.objecter resend_mon_ops 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00b00ba040 con 0x7f00c013bcc0 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 10 monclient: dump: 2024-04-19T00:54:00.757 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.758 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f00b0003e20 con 0x7f00c013bcc0 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 msgr2=0x7f00c013c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 0x7f00c013c0d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f00b00098a0 tx=0x7f00b00a8000 comp rx=0 tx=0).stop 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 msgr2=0x7f00c013c0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00a7fff640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f00c013bcc0 0x7f00c013c0d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.759 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 20355, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:00.760 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.761 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.762 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:00.763 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.764 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.765 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.766 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:00.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.770 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:00.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.774 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.775 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.776 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:00.777 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.778 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.779 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.780 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.781 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:00.782 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 10 librados: watch_flush enter 2024-04-19T00:54:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 10 librados: watch_flush exit 2024-04-19T00:54:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 20 client.74243.objecter shutdown clearing up homeless session... 2024-04-19T00:54:00.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 10 client.74243.objecter successfully canceled tick 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00ac038e10 msgr2=0x7f00ac03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00ac038e10 0x7f00ac03b2d0 secure :-1 s=READY pgs=3506 cs=0 l=1 rev1=1 crypto rx=0x7f00c009e0d0 tx=0x7f00b4008040 comp rx=0 tx=0).stop 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 10 monclient: shutdown 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 msgr2=0x7f00c01d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 --2- 172.21.15.43:0/3040686596 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00c0138cd0 0x7f00c01d0d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00b8007e40 tx=0x7f00b800e4b0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00bf7fe640 1 -- 172.21.15.43:0/3040686596 reap_dead start 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00bf7fe640 1 -- 172.21.15.43:0/3040686596 reap_dead start 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 shutdown_connections 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 >> 172.21.15.43:0/3040686596 conn(0x7f00c00ab2f0 msgr2=0x7f00c01368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 shutdown_connections 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 -- 172.21.15.43:0/3040686596 wait complete. 2024-04-19T00:54:00.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.738+0000 7f00c58f4640 1 librados: shutdown 2024-04-19T00:54:00.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:54:00.786 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:54:00.786 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient: get_monmap_and_config 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient: monmap: 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:00.914968+0000 2024-04-19T00:54:00.912 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:00.914968+0000 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.913 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.914 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.914 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:00.914 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:00.914 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 Processor -- start 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 -- start start 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient: init 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient: _add_conns ranks=[8,10,0,9,7,3,4,5,6,2,1] 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f385c138cd0 0x7f385c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): picked mon.noname-e con 0x7f385c138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f385c0a5b80 0x7f385c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): picked mon.noname-f con 0x7f385c0a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f385c1358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c0a3dc0 con 0x7f385c1358b0 2024-04-19T00:54:00.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f385c0a4410 con 0x7f385c0a5b80 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f385c139b30 con 0x7f385c138cd0 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3862f88640 10 monclient(hunting): authenticate will time out at 1511.042098s 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3861525640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f385c1358b0 0x7f385c135d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:48968/0 (socket says 172.21.15.43:48968) 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3861525640 1 -- 172.21.15.43:0/1818928337 learned_addr learned my addr 172.21.15.43:0/1818928337 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 1 --2- 172.21.15.43:0/1818928337 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f385c138cd0 0x7f385c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): get_auth_request con 0x7f385c138cd0 auth_method 0 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 1 --2- 172.21.15.43:0/1818928337 >> v2:172.21.15.161:3300/0 conn(0x7f385c0a5b80 0x7f385c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): get_auth_request con 0x7f385c0a5b80 auth_method 0 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385b7fe640 1 -- 172.21.15.43:0/1818928337 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175895805 0 0) 0x7f385c0a3dc0 con 0x7f385c1358b0 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385b7fe640 10 monclient(hunting): _negotiate 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385b7fe640 10 monclient(hunting): my global_id is 64602 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385b7fe640 1 -- 172.21.15.43:0/1818928337 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3848003530 con 0x7f385c1358b0 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient(hunting): handle_auth_done global_id 23390 payload 274 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 1 -- 172.21.15.43:0/1818928337 >> v2:172.21.15.161:3300/0 conn(0x7f385c0a5b80 msgr2=0x7f385c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 1 --2- 172.21.15.43:0/1818928337 >> v2:172.21.15.161:3300/0 conn(0x7f385c0a5b80 0x7f385c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 1 -- 172.21.15.43:0/1818928337 >> v1:172.21.15.43:6789/0 conn(0x7f385c1358b0 legacy=0x7f385c135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 1 monclient: found mon.noname-e 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:54:00.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.910+0000 7f3860d24640 1 -- 172.21.15.43:0/1818928337 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c0a40a0 con 0x7f385c138cd0 2024-04-19T00:54:00.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 1 --2- 172.21.15.43:0/1818928337 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f385c138cd0 0x7f385c1390b0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097c0 tx=0x7f384c01c300 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5ab26ef7065ddc07 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 1 -- 172.21.15.43:0/1818928337 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f384c0b0070 con 0x7f385c138cd0 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: dump: 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.927 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 1 -- 172.21.15.43:0/1818928337 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384c0aa810 con 0x7f385c138cd0 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 1 -- 172.21.15.43:0/1818928337 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f384c0a9300 con 0x7f385c138cd0 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 5 monclient: authenticate success, global_id 23390 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:00.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 4 set_mon_vals no callback set 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f385b7fe640 10 monclient: dump: 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.929 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: shutdown 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- 172.21.15.43:0/1818928337 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f385c138cd0 msgr2=0x7f385c1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 --2- 172.21.15.43:0/1818928337 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f385c138cd0 0x7f385c1390b0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097c0 tx=0x7f384c01c300 comp rx=0 tx=0).stop 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- 172.21.15.43:0/1818928337 shutdown_connections 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 --2- 172.21.15.43:0/1818928337 >> v2:172.21.15.161:3300/0 conn(0x7f385c0a5b80 0x7f385c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 --2- 172.21.15.43:0/1818928337 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f385c138cd0 0x7f385c1390b0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- 172.21.15.43:0/1818928337 >> 172.21.15.43:0/1818928337 conn(0x7f385c0ab2f0 msgr2=0x7f385c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- 172.21.15.43:0/1818928337 shutdown_connections 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- 172.21.15.43:0/1818928337 wait complete. 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: build_initial_monmap 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: monmap: 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:00.931 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 librados: starting msgr at 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 librados: starting objecter 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 Processor -- start 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- start start 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 librados: setting wanted keys 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 librados: calling monclient init 2024-04-19T00:54:00.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: init 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient: _add_conns ranks=[0,3,4,8,1,6,5,2,7] 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f385c0a5b80 0x7f385c1d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): picked mon.noname-a con 0x7f385c0a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f385c1358b0 0x7f385c13ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): picked mon.noname-g con 0x7f385c1358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f385c138cd0 0x7f385c1404b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): picked mon.noname-c con 0x7f385c138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f385c143ab0 con 0x7f385c0a5b80 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f385c0a20a0 con 0x7f385c1358b0 2024-04-19T00:54:00.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f385c143c30 con 0x7f385c138cd0 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3862f88640 10 monclient(hunting): authenticate will time out at 1511.047706s 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f385c138cd0 0x7f385c1404b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f385c0a5b80 0x7f385c1d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f385c138cd0 0x7f385c1404b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:40992/0 (socket says 172.21.15.43:40992) 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): get_auth_request con 0x7f385c0a5b80 auth_method 0 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 1 -- 172.21.15.43:0/3998066978 learned_addr learned my addr 172.21.15.43:0/3998066978 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 10 monclient(hunting): get_auth_request con 0x7f385c138cd0 auth_method 0 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3861525640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.914+0000 7f3860d24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3861525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3861525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3861525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f385c1358b0 0x7f385c13ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): get_auth_request con 0x7f385c1358b0 auth_method 0 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 10 monclient(hunting): handle_auth_done global_id 64611 payload 824 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f385c138cd0 msgr2=0x7f385c1404b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f385c138cd0 0x7f385c1404b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f385c1358b0 msgr2=0x7f385c13ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f385c1358b0 0x7f385c13ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 10 monclient: _finish_hunting 0 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 monclient: found mon.noname-a 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c1409f0 con 0x7f385c0a5b80 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3860d24640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f385c0a5b80 0x7f385c1d0f40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f384c0ab000 tx=0x7f384c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 client.?.objecter ms_handle_connect 0x7f385c0a5b80 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f384c0b0040 con 0x7f385c0a5b80 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: dump: 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.937 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: _finish_auth 0 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: _check_auth_tickets 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384c004040 con 0x7f385c0a5b80 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f384c003620 con 0x7f385c0a5b80 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385affd640 4 set_mon_vals no callback set 2024-04-19T00:54:00.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: dump: 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.939 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 5 monclient: authenticate success, global_id 64611 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 monclient: _renew_subs 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c13e910 con 0x7f385c0a5b80 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 client.64611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 monclient: _renew_subs 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c0a3040 con 0x7f385c0a5b80 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 1 librados: init done 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 0x7f385c13ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f385c13c620 con 0x7f385c13a950 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f384c0aab80 con 0x7f385c0a5b80 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 0x7f385c13ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient: get_auth_request con 0x7f385c13a950 auth_method 0 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f383c038e40 0x7f383c03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(371..371 src has 1..371) ==== 6170+0+0 (secure 0 0 0) 0x7f384c0e2450 con 0x7f385c0a5b80 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 client.64611.objecter ms_dispatch 0x7f385c000ba0 osd_map(371..371 src has 1..371) 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 3 client.64611.objecter handle_osd_map got epochs [371,371] > 0 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 3 client.64611.objecter handle_osd_map decoding full epoch 371 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 20 client.64611.objecter dump_active .. 0 homeless 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3861525640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f383c038e40 0x7f383c03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3861525640 10 monclient: get_auth_request con 0x7f383c038e40 auth_method 0 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 10 monclient(hunting): handle_auth_done global_id 74194 payload 824 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f385c0693b0 con 0x7f385c13a950 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 client.64611.objecter ms_handle_connect 0x7f385c13a950 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 client.64611.objecter resend_mon_ops 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f385bfff640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 0x7f385c13ad30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f385c0a7360 tx=0x7f38440115a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3844003e10 con 0x7f385c13a950 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:54:00.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: dump: 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.943 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 client.64611.objecter ms_handle_connect 0x7f383c038e40 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f3861525640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f383c038e40 0x7f383c03b300 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f3854005890 tx=0x7f3854009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f38440039a0 con 0x7f385c13a950 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 msgr2=0x7f385c13ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 0x7f385c13ad30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f385c0a7360 tx=0x7f38440115a0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 msgr2=0x7f385c13ad30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.918+0000 7f38597fa640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13a950 0x7f385c13ad30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 0x7f385c09da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f385c06d140 con 0x7f385c13b270 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 0x7f385c09da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient: get_auth_request con 0x7f385c13b270 auth_method 0 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:00.946 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 10 monclient(hunting): handle_auth_done global_id 74203 payload 824 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 1 -- 172.21.15.43:0/3998066978 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f385c140e10 con 0x7f385c13b270 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f385bfff640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 0x7f385c09da80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3844000f90 tx=0x7f3844004a50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 client.64611.objecter ms_handle_connect 0x7f385c13b270 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 client.64611.objecter resend_mon_ops 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f38440046d0 con 0x7f385c13b270 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 monclient: dump: 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f38440046d0 con 0x7f385c13b270 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 msgr2=0x7f385c09da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 0x7f385c09da80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3844000f90 tx=0x7f3844004a50 comp rx=0 tx=0).stop 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 msgr2=0x7f385c09da80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f38597fa640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f385c13b270 0x7f385c09da80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 66, 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 20551, 2024-04-19T00:54:00.949 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.950 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.951 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.952 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:00.953 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:00.954 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:00.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.956 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:00.957 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:00.958 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.959 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.960 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.961 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.962 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:00.963 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.964 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.965 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.966 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:00.967 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:00.968 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:00.969 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:00.970 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:00.971 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 10 librados: watch_flush enter 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 10 librados: watch_flush exit 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 20 client.64611.objecter shutdown clearing up homeless session... 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 10 client.64611.objecter successfully canceled tick 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f383c038e40 msgr2=0x7f383c03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f383c038e40 0x7f383c03b300 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f3854005890 tx=0x7f3854009040 comp rx=0 tx=0).stop 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 10 monclient: shutdown 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f385c0a5b80 msgr2=0x7f385c1d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:00.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 --2- 172.21.15.43:0/3998066978 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f385c0a5b80 0x7f385c1d0f40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f384c0ab000 tx=0x7f384c019100 comp rx=0 tx=0).stop 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3861525640 1 -- 172.21.15.43:0/3998066978 reap_dead start 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3861525640 1 -- 172.21.15.43:0/3998066978 reap_dead start 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 shutdown_connections 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 >> 172.21.15.43:0/3998066978 conn(0x7f385c0ab2f0 msgr2=0x7f385c1368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 shutdown_connections 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 -- 172.21.15.43:0/3998066978 wait complete. 2024-04-19T00:54:00.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:00.930+0000 7f3862f88640 1 librados: shutdown 2024-04-19T00:54:00.974 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:54:00.975 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['e', 'f'] 2024-04-19T00:54:00.975 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['g'] 2024-04-19T00:54:00.975 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-04-19T00:54:00.975 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-04-19T00:54:00.975 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-04-19T00:54:00.975 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:54:00.992 INFO:tasks.ceph.mon.e:Stopped 2024-04-19T00:54:00.992 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-04-19T00:54:00.992 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-04-19T00:54:00.993 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-04-19T00:54:00.993 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:54:01.013 INFO:tasks.ceph.mon.f:Stopped 2024-04-19T00:54:01.013 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-04-19T00:54:01.014 INFO:tasks.ceph.mon.g:Sent signal 19 2024-04-19T00:54:01.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:54:16.015 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-04-19T00:54:16.015 INFO:tasks.ceph.mon.g:Sent signal 18 2024-04-19T00:54:16.016 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-04-19T00:54:16.016 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient: get_monmap_and_config 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient: build_initial_monmap 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient: monmap: 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:16.151967+0000 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:16.151967+0000 2024-04-19T00:54:16.149 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:16.150 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:16.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 Processor -- start 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 -- start start 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient: init 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient: _add_conns ranks=[8,2,4,10,7,1,5,9,0,3,6] 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 0x7effa8136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): picked mon.noname-e con 0x7effa809e4c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): picked mon.noname-b con 0x7effa809e8a0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): picked mon.noname-c con 0x7effa809eca0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7effa80a1c50 con 0x7effa809e8a0 2024-04-19T00:54:16.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7effa80a1e20 con 0x7effa809eca0 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7effa806c760 con 0x7effa809e4c0 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effaf0e1640 10 monclient(hunting): authenticate will time out at 1526.279545s 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7efface7d640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 msgr2=0x7effa8136e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7efface7d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 0x7effa8136e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36174/0 (socket says 172.21.15.43:36174) 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 learned_addr learned my addr 172.21.15.43:0/3811482901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7effa809eca0 auth_method 0 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:16.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7effa809e8a0 auth_method 0 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:16.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.146+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:16.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 msgr2=0x7effa8136e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:16.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 0x7effa8136e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:16.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:16.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:16.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7effa809eca0 auth_method 0 2024-04-19T00:54:16.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7effa809e8a0 auth_method 0 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:16.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:16.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.350+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:16.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 msgr2=0x7effa8136e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:16.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 0x7effa8136e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:16.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:16.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:16.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7effa809eca0 auth_method 0 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7effa809e8a0 auth_method 0 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:16.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:16.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:16.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:16.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:16.754+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:17.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 msgr2=0x7effa8136e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 0x7effa8136e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7effa809eca0 auth_method 0 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7effa809e8a0 auth_method 0 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:17.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:17.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:17.558+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:19.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): tick 2024-04-19T00:54:19.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 msgr2=0x7effa8136e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7effa809e4c0 0x7effa8136e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 msgr2=0x7effa80a3e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7effa809eca0 0x7effa80a3e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 msgr2=0x7effa80a3940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7effa809e8a0 0x7effa80a3940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient: _add_conns ranks=[1,3,5,6,9,10,7,0] 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7eff94003830 addr v1:172.21.15.161:6789/0 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-g con 0x7eff94007380 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:19.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-h con 0x7eff94009e70 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa80a5510 con 0x7eff94003830 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7effa80a1e20 con 0x7eff94007380 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.146+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7effa806c760 con 0x7eff94009e70 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa67fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.161:6789/0 conn(0x7eff94003830 legacy=0x7eff94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff94007380 auth_method 0 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:19.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7eff94009e70 auth_method 0 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:19.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.150+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:19.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.161:6789/0 conn(0x7eff94003830 legacy=0x7eff94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7eff94009e70 auth_method 0 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff94007380 auth_method 0 2024-04-19T00:54:19.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:19.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:19.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.354+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:19.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.161:6789/0 conn(0x7eff94003830 legacy=0x7eff94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff94007380 auth_method 0 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7eff94009e70 auth_method 0 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:19.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:19.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:19.758+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:20.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.161:6789/0 conn(0x7eff94003830 legacy=0x7eff94005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:20.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:20.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:20.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7eff94009e70 auth_method 0 2024-04-19T00:54:20.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:20.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff94007380 auth_method 0 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:20.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:20.562+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): tick 2024-04-19T00:54:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:54:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:54:22.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7eff94009e70 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 msgr2=0x7eff94009810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff94007380 0x7eff94009810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.161:6789/0 conn(0x7eff94003830 legacy=0x7eff94005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 reap_dead start 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 reap_dead start 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient: _add_conns ranks=[6,0,10,9,7] 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-d con 0x7eff9400cbd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff9400d050 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:22.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-f con 0x7eff94009850 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9400dc00 con 0x7eff9400d050 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7effa80a1e20 con 0x7eff94009850 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7effa806c760 con 0x7eff9400cbd0 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effa67fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400cbd0 auth_method 0 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:22.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:22.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:22.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:22.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.154+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:22.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:22.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:22.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:22.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400cbd0 auth_method 0 2024-04-19T00:54:22.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:22.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.358+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:22.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400cbd0 auth_method 0 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:22.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:22.762+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:23.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:23.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400cbd0 auth_method 0 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:23.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:23.566+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): tick 2024-04-19T00:54:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:54:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:54:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:54:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 msgr2=0x7eff9400c330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:25.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7eff9400cbd0 0x7eff9400c330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.43:6789/0 conn(0x7eff9400d050 legacy=0x7eff94003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient: _add_conns ranks=[7,9] 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-i con 0x7eff9400e200 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-a con 0x7eff9400ec50 addr v2:172.21.15.43:3300/0 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7effa80a1e20 con 0x7eff9400ec50 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7effa806c760 con 0x7eff9400e200 2024-04-19T00:54:25.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa67fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff9400ec50 auth_method 0 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400e200 auth_method 0 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:25.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:25.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.150+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:25.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff9400ec50 auth_method 0 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400e200 auth_method 0 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:25.355 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:25.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:25.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:25.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:25.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.354+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff9400ec50 auth_method 0 2024-04-19T00:54:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:25.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400e200 auth_method 0 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:25.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:25.758+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:26.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff9400ec50 auth_method 0 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 10 monclient(hunting): get_auth_request con 0x7eff9400e200 auth_method 0 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:26.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:54:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:54:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:54:26.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:26.562+0000 7efface7d640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:28.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): tick 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 monclient(hunting): continuing hunt 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 msgr2=0x7eff9400e5e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7eff9400e200 0x7eff9400e5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 msgr2=0x7eff94013160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.43:3300/0 conn(0x7eff9400ec50 0x7eff94013160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient: _add_conns ranks=[0,10,4,7,6,9,3,2,5,8,1] 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 reap_dead start 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff9400d050 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:28.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-f con 0x7eff94009850 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff9400cb70 0x7eff9400c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): picked mon.noname-c con 0x7eff9400cb70 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa80a5510 con 0x7eff9400d050 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7effa80a1e20 con 0x7eff94009850 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 1 -- 172.21.15.43:0/3811482901 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7effa806c760 con 0x7eff9400cb70 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa67fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa7fff640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff9400cb70 0x7eff9400c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa7fff640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7eff9400cb70 auth_method 0 2024-04-19T00:54:28.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient(hunting): handle_auth_done global_id 64453 payload 274 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 msgr2=0x7eff9400d4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 >> v1:172.21.15.43:6789/0 conn(0x7eff9400d050 legacy=0x7eff94003c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 monclient: found mon.noname-c 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa77fe640 1 -- 172.21.15.43:0/3811482901 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4153119031 0 0) 0x7effa80a5510 con 0x7eff9400d050 2024-04-19T00:54:28.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:54:28.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effad67e640 1 -- 172.21.15.43:0/3811482901 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9400e200 con 0x7eff9400cb70 2024-04-19T00:54:28.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.150+0000 7effa77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:54:28.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effad67e640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff9400cb70 0x7eff9400c330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7effa80a5480 tx=0x7effa001e500 comp rx=0 tx=0).ready entity=mon.4 client_cookie=76aed0f34376c2df server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 1 -- 172.21.15.43:0/3811482901 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7effa0003e10 con 0x7eff9400cb70 2024-04-19T00:54:28.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:54:28.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: dump: 2024-04-19T00:54:28.160 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.160 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.161 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 1 -- 172.21.15.43:0/3811482901 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa000b0d0 con 0x7eff9400cb70 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 5 monclient: authenticate success, global_id 64453 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 4 set_mon_vals no callback set 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 1 -- 172.21.15.43:0/3811482901 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7effa0009700 con 0x7eff9400cb70 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:28.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effa77fe640 10 monclient: dump: 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.163 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.164 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.164 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 10 monclient: shutdown 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 -- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff9400cb70 msgr2=0x7eff9400c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff9400cb70 0x7eff9400c330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7effa80a5480 tx=0x7effa001e500 comp rx=0 tx=0).stop 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 -- 172.21.15.43:0/3811482901 shutdown_connections 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 --2- 172.21.15.43:0/3811482901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7eff9400cb70 0x7eff9400c330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 --2- 172.21.15.43:0/3811482901 >> v2:172.21.15.161:3300/0 conn(0x7eff94009850 0x7eff9400d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 -- 172.21.15.43:0/3811482901 >> 172.21.15.43:0/3811482901 conn(0x7effa80a85c0 msgr2=0x7effa80a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 -- 172.21.15.43:0/3811482901 shutdown_connections 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.158+0000 7effaf0e1640 1 -- 172.21.15.43:0/3811482901 wait complete. 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient: monmap: 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:28.166 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 librados: starting msgr at 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 librados: starting objecter 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 Processor -- start 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 -- start start 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 librados: setting wanted keys 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 librados: calling monclient init 2024-04-19T00:54:28.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient: init 2024-04-19T00:54:28.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient: _add_conns ranks=[1,3,2,8,7,5,4,6,0] 2024-04-19T00:54:28.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff94009850 0x7effa81cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): picked mon.noname-f con 0x7eff94009850 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:28.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9400cb70 0x7effa81cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): picked mon.noname-g con 0x7eff9400cb70 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 0x7effa80718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): picked mon.noname-b con 0x7eff9400d050 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7effa8136fe0 con 0x7eff94009850 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7effa8136e60 con 0x7eff9400d050 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7effa8137160 con 0x7eff9400cb70 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effaf0e1640 10 monclient(hunting): authenticate will time out at 1538.294869s 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7efface7d640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff94009850 msgr2=0x7effa81cd750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7efface7d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff94009850 0x7effa81cd750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 0x7effa80718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 0x7effa80718b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35060/0 (socket says 172.21.15.43:35060) 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 -- 172.21.15.43:0/464465553 learned_addr learned my addr 172.21.15.43:0/464465553 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): get_auth_request con 0x7eff9400d050 auth_method 0 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa7fff640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9400cb70 0x7effa81cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa7fff640 10 monclient(hunting): get_auth_request con 0x7eff9400cb70 auth_method 0 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient(hunting): handle_auth_done global_id 74252 payload 824 2024-04-19T00:54:28.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 -- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9400cb70 msgr2=0x7effa81cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9400cb70 0x7effa81cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 -- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff94009850 msgr2=0x7effa81cd750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff94009850 0x7effa81cd750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 monclient: found mon.noname-b 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 -- 172.21.15.43:0/464465553 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa8125ae0 con 0x7eff9400d050 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa7fff640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9400cb70 0x7effa81cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effad67e640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 0x7effa80718b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7effa001e330 tx=0x7effa00082a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa57fa640 10 client.?.objecter ms_handle_connect 0x7eff9400d050 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.162+0000 7effa57fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:28.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7effa0003e40 con 0x7eff9400d050 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: dump: 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.175 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 5 monclient: authenticate success, global_id 74252 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 monclient: _renew_subs 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa00bf070 con 0x7eff9400d050 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa8124e20 con 0x7eff9400d050 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7effa0009c40 con 0x7eff9400d050 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 client.74252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa6ffd640 4 set_mon_vals no callback set 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 monclient: dump: 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.177 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 monclient: _renew_subs 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7effa8125100 con 0x7eff9400d050 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 1 librados: init done 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa806bda0 con 0x7eff9400d050 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7effa0015040 con 0x7eff9400d050 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff9403e600 0x7eff94040ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(373..373 src has 1..373) ==== 6157+0+0 (secure 0 0 0) 0x7effa0012040 con 0x7eff9400d050 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 client.74252.objecter ms_dispatch 0x7effa8000ba0 osd_map(373..373 src has 1..373) 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 3 client.74252.objecter handle_osd_map got epochs [373,373] > 0 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 3 client.74252.objecter handle_osd_map decoding full epoch 373 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 20 client.74252.objecter dump_active .. 0 homeless 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7efface7d640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff9403e600 0x7eff94040ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7efface7d640 10 monclient: get_auth_request con 0x7eff9403e600 auth_method 0 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7effa57fa640 10 client.74252.objecter ms_handle_connect 0x7eff9403e600 2024-04-19T00:54:28.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.166+0000 7efface7d640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff9403e600 0x7eff94040ac0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7eff9c007370 tx=0x7eff9c0037c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.170+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7effa8125ae0 con 0x7eff9400d050 2024-04-19T00:54:28.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.170+0000 7effa57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.170+0000 7effa57fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:28.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effaf0e1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:54:28.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effaf0e1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:54:28.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effaf0e1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:28.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7effa81236c0 con 0x7eff9400d050 2024-04-19T00:54:28.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effa57fa640 1 -- 172.21.15.43:0/464465553 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3180 (secure 0 0 0) 0x7effa000d2f0 con 0x7eff9400d050 2024-04-19T00:54:28.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effa57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:54:28.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.422+0000 7effa57fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:28.427 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":78,"quorum":[0,2,3,4,5,6,7],"quorum_names":["a","b","g","c","h","d","i"],"quorum_leader_name":"g","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:54:28.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 10 librados: watch_flush enter 2024-04-19T00:54:28.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 10 librados: watch_flush exit 2024-04-19T00:54:28.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 20 client.74252.objecter shutdown clearing up homeless session... 2024-04-19T00:54:28.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 10 client.74252.objecter successfully canceled tick 2024-04-19T00:54:28.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff9403e600 msgr2=0x7eff94040ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff9403e600 0x7eff94040ac0 secure :-1 s=READY pgs=3508 cs=0 l=1 rev1=1 crypto rx=0x7eff9c007370 tx=0x7eff9c0037c0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 10 monclient: shutdown 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 msgr2=0x7effa80718b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 0x7effa80718b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7effa001e330 tx=0x7effa00082a0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 shutdown_connections 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7eff9403e600 0x7eff94040ac0 unknown :-1 s=CLOSED pgs=3508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7eff9400d050 0x7effa80718b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7eff9400cb70 0x7effa81cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 --2- 172.21.15.43:0/464465553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff94009850 0x7effa81cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 >> 172.21.15.43:0/464465553 conn(0x7effa80a85c0 msgr2=0x7effa81346e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 shutdown_connections 2024-04-19T00:54:28.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 -- 172.21.15.43:0/464465553 wait complete. 2024-04-19T00:54:28.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.426+0000 7effaf0e1640 1 librados: shutdown 2024-04-19T00:54:28.447 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-04-19T00:54:28.447 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-04-19T00:54:28.447 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:54:28.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.570+0000 7f1d779c6640 10 monclient: get_monmap_and_config 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.570+0000 7f1d779c6640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.570+0000 7f1d779c6640 10 monclient: monmap: 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:28.577526+0000 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:28.577526+0000 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:28.576 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.586 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 Processor -- start 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 -- start start 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient: init 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient: _add_conns ranks=[0,8,3,4,10,1,5,9,6,2,7] 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d70138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1d70139540 0x7f1d7013dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): picked mon.noname-e con 0x7f1d70139540 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d70140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): picked mon.noname-g con 0x7f1d700a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d700a0d40 con 0x7f1d70138a50 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d700a1020 con 0x7f1d700a5bb0 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f1d700a0780 con 0x7f1d70139540 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 10 monclient(hunting): authenticate will time out at 1538.705379s 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1d70139540 msgr2=0x7f1d7013dfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1d70139540 0x7f1d7013dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75762640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f1d70138a50 0x7f1d70138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49126/0 (socket says 172.21.15.43:49126) 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75762640 1 -- 172.21.15.43:0/3489550883 learned_addr learned my addr 172.21.15.43:0/3489550883 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951427094 0 0) 0x7f1d700a0d40 con 0x7f1d70138a50 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): _negotiate 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): my global_id is 64629 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d5c003530 con 0x7f1d70138a50 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d70140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): get_auth_request con 0x7f1d700a5bb0 auth_method 0 2024-04-19T00:54:28.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3548944173 0 0) 0x7f1d5c003530 con 0x7f1d70138a50 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1d70139540 msgr2=0x7f1d7013dfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1d70139540 0x7f1d7013dfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 msgr2=0x7f1d70140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d70140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d700a0a60 con 0x7f1d70138a50 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 5 monclient: authenticate success, global_id 64629 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d70140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f1d6c001970 con 0x7f1d70138a50 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: dump: 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.590 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 >> v1:172.21.15.43:6789/0 conn(0x7f1d70138a50 legacy=0x7f1d70138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient: _add_conns ranks=[1,3,4,8,6,0,5,2,7] 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d5c004240 0x7f1d5c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): picked mon.f con 0x7f1d5c004240 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:28.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d5c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): picked mon.g con 0x7f1d700a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1d5c0099c0 0x7f1d5c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): picked mon.c con 0x7f1d5c0099c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d700a1020 con 0x7f1d5c004240 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d700a0780 con 0x7f1d700a5bb0 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f1d5c00c480 con 0x7f1d5c0099c0 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d67fff640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d779c6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75762640 1 -- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d5c004240 msgr2=0x7f1d5c004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75762640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d5c004240 0x7f1d5c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1d5c0099c0 0x7f1d5c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): get_auth_request con 0x7f1d5c0099c0 auth_method 0 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d5c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 10 monclient(hunting): get_auth_request con 0x7f1d700a5bb0 auth_method 0 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.593 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d74f61640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient(hunting): handle_auth_done global_id 64629 payload 293 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 -- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 msgr2=0x7f1d5c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d700a5bb0 0x7f1d5c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 -- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d5c004240 msgr2=0x7f1d5c004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d5c004240 0x7f1d5c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 monclient: found mon.c 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 1 -- 172.21.15.43:0/3489550883 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5c003530 con 0x7f1d5c0099c0 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.574+0000 7f1d75f63640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1d5c0099c0 0x7f1d5c00bdb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d680073e0 tx=0x7f1d6801cea0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=ed112d76513e8d10 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d680b2070 con 0x7f1d5c0099c0 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: dump: 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.595 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d680a5df0 con 0x7f1d5c0099c0 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 4 set_mon_vals no callback set 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 1 -- 172.21.15.43:0/3489550883 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d68009c80 con 0x7f1d5c0099c0 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d67fff640 10 monclient: dump: 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.597 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: shutdown 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1d5c0099c0 msgr2=0x7f1d5c00bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 --2- 172.21.15.43:0/3489550883 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1d5c0099c0 0x7f1d5c00bdb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d680073e0 tx=0x7f1d6801cea0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75f63640 1 -- 172.21.15.43:0/3489550883 reap_dead start 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- 172.21.15.43:0/3489550883 shutdown_connections 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- 172.21.15.43:0/3489550883 >> 172.21.15.43:0/3489550883 conn(0x7f1d700a7cb0 msgr2=0x7f1d700a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- 172.21.15.43:0/3489550883 shutdown_connections 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- 172.21.15.43:0/3489550883 wait complete. 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: monmap: 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.599 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 librados: starting msgr at 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 librados: starting objecter 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 Processor -- start 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- start start 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 librados: setting wanted keys 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 librados: calling monclient init 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: init 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient: _add_conns ranks=[0,1,7,3,8,5,4,2,6] 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701d0730 0x7f1d70136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): picked mon.noname-a con 0x7f1d701d0730 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:54:28.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d70137300 0x7f1d701cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): picked mon.noname-f con 0x7f1d70137300 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d701cd770 0x7f1d701cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): picked mon.noname-i con 0x7f1d701cd770 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d701410d0 con 0x7f1d701d0730 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d70140f50 con 0x7f1d70137300 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1d70141250 con 0x7f1d701cd770 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d779c6640 10 monclient(hunting): authenticate will time out at 1538.711172s 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701d0730 0x7f1d70136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d74f61640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d70137300 msgr2=0x7f1d701cd230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701d0730 0x7f1d70136dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58060/0 (socket says 172.21.15.43:58060) 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 -- 172.21.15.43:0/3799732839 learned_addr learned my addr 172.21.15.43:0/3799732839 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): get_auth_request con 0x7f1d701d0730 auth_method 0 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d74f61640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d70137300 0x7f1d701cd230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d701cd770 0x7f1d701cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient(hunting): handle_auth_done global_id 64638 payload 824 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d701cd770 msgr2=0x7f1d701cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d701cd770 0x7f1d701cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d70137300 msgr2=0x7f1d701cd230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d70137300 0x7f1d701cd230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 monclient: found mon.noname-a 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d70137780 con 0x7f1d701d0730 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d75762640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701d0730 0x7f1d70136dc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c001170 tx=0x7f1d6c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d65ffb640 10 client.?.objecter ms_handle_connect 0x7f1d701d0730 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d65ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d6c018070 con 0x7f1d701d0730 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.578+0000 7f1d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: dump: 2024-04-19T00:54:28.603 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.604 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 5 monclient: authenticate success, global_id 64638 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 monclient: _renew_subs 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d6c004d20 con 0x7f1d701d0730 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7013d7d0 con 0x7f1d701d0730 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d6c007540 con 0x7f1d701d0730 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 client.64638.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d677fe640 4 set_mon_vals no callback set 2024-04-19T00:54:28.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: dump: 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.606 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 monclient: _renew_subs 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d701231b0 con 0x7f1d701d0730 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 1 librados: init done 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 0x7f1d700a3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7006d140 con 0x7f1d701233f0 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 0x7f1d700a3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient: get_auth_request con 0x7f1d701233f0 auth_method 0 2024-04-19T00:54:28.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1d6c0070d0 con 0x7f1d701d0730 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 10 monclient(hunting): handle_auth_done global_id 64647 payload 824 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d701d0b10 con 0x7f1d701233f0 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d54038e10 0x7f1d5403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(374..374 src has 1..374) ==== 6170+0+0 (secure 0 0 0) 0x7f1d6c048790 con 0x7f1d701d0730 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 client.64638.objecter ms_dispatch 0x7f1d70000ba0 osd_map(374..374 src has 1..374) 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 3 client.64638.objecter handle_osd_map got epochs [374,374] > 0 2024-04-19T00:54:28.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 3 client.64638.objecter handle_osd_map decoding full epoch 374 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d74f61640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 0x7f1d700a3280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d7013bb00 tx=0x7f1d6000d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 20 client.64638.objecter dump_active .. 0 homeless 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 client.64638.objecter ms_handle_connect 0x7f1d701233f0 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 client.64638.objecter resend_mon_ops 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d60016020 con 0x7f1d701233f0 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: dump: 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.609 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d54038e10 0x7f1d5403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d75f63640 10 monclient: get_auth_request con 0x7f1d54038e10 auth_method 0 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d60003b80 con 0x7f1d701233f0 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:28.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 msgr2=0x7f1d700a3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 0x7f1d700a3280 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d7013bb00 tx=0x7f1d6000d960 comp rx=0 tx=0).stop 2024-04-19T00:54:28.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 msgr2=0x7f1d700a3280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:28.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701233f0 0x7f1d700a3280 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1d7013bb00 tx=0x7f1d6000d960 comp rx=0 tx=0).stop 2024-04-19T00:54:28.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d65ffb640 10 client.64638.objecter ms_handle_connect 0x7f1d54038e10 2024-04-19T00:54:28.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.582+0000 7f1d75f63640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d54038e10 0x7f1d5403b2d0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f1d680af000 tx=0x7f1d6800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 0x7f1d700a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7013da30 con 0x7f1d700a37c0 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 0x7f1d700a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient: get_auth_request con 0x7f1d700a37c0 auth_method 0 2024-04-19T00:54:28.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 10 monclient(hunting): handle_auth_done global_id 64656 payload 824 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 1 -- 172.21.15.43:0/3799732839 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d70076190 con 0x7f1d700a37c0 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d74f61640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 0x7f1d700a3bd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1d7013bb00 tx=0x7f1d60005a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 client.64638.objecter ms_handle_connect 0x7f1d700a37c0 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 client.64638.objecter resend_mon_ops 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d6001de60 con 0x7f1d700a37c0 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 monclient: dump: 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.615 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9421 (secure 0 0 0) 0x7f1d6001d0d0 con 0x7f1d700a37c0 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 msgr2=0x7f1d700a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 0x7f1d700a3bd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1d7013bb00 tx=0x7f1d60005a80 comp rx=0 tx=0).stop 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 msgr2=0x7f1d700a3bd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d65ffb640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d700a37c0 0x7f1d700a3bd0 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1d7013bb00 tx=0x7f1d60005a80 comp rx=0 tx=0).stop 2024-04-19T00:54:28.616 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 194862, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:28.617 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.618 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.619 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:28.620 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.621 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:28.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:28.624 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:28.625 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:28.626 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.628 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:28.629 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.630 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:28.632 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:28.636 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.638 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:28.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:28.640 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 10 librados: watch_flush enter 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 10 librados: watch_flush exit 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 20 client.64638.objecter shutdown clearing up homeless session... 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 10 client.64638.objecter successfully canceled tick 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d54038e10 msgr2=0x7f1d5403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d54038e10 0x7f1d5403b2d0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f1d680af000 tx=0x7f1d6800b040 comp rx=0 tx=0).stop 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 10 monclient: shutdown 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701d0730 msgr2=0x7f1d70136dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 --2- 172.21.15.43:0/3799732839 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d701d0730 0x7f1d70136dc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c001170 tx=0x7f1d6c00cb10 comp rx=0 tx=0).stop 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d75f63640 1 -- 172.21.15.43:0/3799732839 reap_dead start 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d75f63640 1 -- 172.21.15.43:0/3799732839 reap_dead start 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 shutdown_connections 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 >> 172.21.15.43:0/3799732839 conn(0x7f1d700a7cb0 msgr2=0x7f1d70137b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 shutdown_connections 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 -- 172.21.15.43:0/3799732839 wait complete. 2024-04-19T00:54:28.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.590+0000 7f1d779c6640 1 librados: shutdown 2024-04-19T00:54:28.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:28.643 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-04-19T00:54:28.644 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:54:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient: get_monmap_and_config 2024-04-19T00:54:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient: monmap: 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:28.766519+0000 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:28.766519+0000 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.764 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.765 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.765 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:28.765 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:28.765 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 Processor -- start 2024-04-19T00:54:28.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 -- start start 2024-04-19T00:54:28.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient: init 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient: _add_conns ranks=[10,8,0,4,9,5,7,6,3,2,1] 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc78138cd0 0x7efc781390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): picked mon.noname-f con 0x7efc78138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efc780a5b80 0x7efc781395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): picked mon.noname-e con 0x7efc780a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc781358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc780a3dc0 con 0x7efc781358b0 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efc780a4410 con 0x7efc78138cd0 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7efc78139b30 con 0x7efc780a5b80 2024-04-19T00:54:28.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 10 monclient(hunting): authenticate will time out at 1538.893708s 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efc780a5b80 msgr2=0x7efc781395f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efc780a5b80 0x7efc781395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7659c640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7efc78138cd0 msgr2=0x7efc781390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc78138cd0 0x7efc781390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7efc781358b0 0x7efc78135d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49130/0 (socket says 172.21.15.43:49130) 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 -- 172.21.15.43:0/3219586204 learned_addr learned my addr 172.21.15.43:0/3219586204 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3166320095 0 0) 0x7efc780a3dc0 con 0x7efc781358b0 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): _negotiate 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): my global_id is 64665 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc64003530 con 0x7efc781358b0 2024-04-19T00:54:28.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4043616492 0 0) 0x7efc64003530 con 0x7efc781358b0 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efc780a5b80 msgr2=0x7efc781395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efc780a5b80 0x7efc781395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 >> v2:172.21.15.161:3300/0 conn(0x7efc78138cd0 msgr2=0x7efc781390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 --2- 172.21.15.43:0/3219586204 >> v2:172.21.15.161:3300/0 conn(0x7efc78138cd0 0x7efc781390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc780a40a0 con 0x7efc781358b0 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7efc70001cf0 con 0x7efc781358b0 2024-04-19T00:54:28.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: dump: 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.774 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 >> v1:172.21.15.43:6789/0 conn(0x7efc781358b0 legacy=0x7efc78135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient: _add_conns ranks=[1,3,7,6,5,2,8,0,4] 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc64004210 0x7efc64004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): picked mon.f con 0x7efc64004210 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efc64004f00 0x7efc640099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): picked mon.g con 0x7efc64004f00 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efc64009f00 0x7efc6400c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): picked mon.i con 0x7efc64009f00 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efc780a4410 con 0x7efc64004210 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efc78139b30 con 0x7efc64004f00 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7efc6400cb60 con 0x7efc64009f00 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7559a640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7659c640 1 -- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc64004210 msgr2=0x7efc64004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7659c640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc64004210 0x7efc64004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efc64009f00 0x7efc6400c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efc64004f00 0x7efc640099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): get_auth_request con 0x7efc64009f00 auth_method 0 2024-04-19T00:54:28.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): get_auth_request con 0x7efc64004f00 auth_method 0 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient(hunting): handle_auth_done global_id 64665 payload 293 2024-04-19T00:54:28.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 -- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efc64004f00 msgr2=0x7efc640099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efc64004f00 0x7efc640099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 -- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc64004210 msgr2=0x7efc64004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc64004210 0x7efc64004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 monclient: found mon.i 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 1 -- 172.21.15.43:0/3219586204 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc64003530 con 0x7efc64009f00 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc76d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 5 monclient: authenticate success, global_id 64665 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc7cb6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:28.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.762+0000 7efc75d9b640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efc64004f00 0x7efc640099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc76d9d640 1 -- 172.21.15.43:0/3219586204 reap_dead start 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc76d9d640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efc64009f00 0x7efc6400c2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc7000a640 tx=0x7efc7001d000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=20a76e609de1ee30 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc68009500 con 0x7efc64009f00 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: dump: 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.779 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc700a9070 con 0x7efc64009f00 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 1 -- 172.21.15.43:0/3219586204 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc7000a260 con 0x7efc64009f00 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:54:28.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7559a640 10 monclient: dump: 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.781 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 4 set_mon_vals no callback set 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 10 monclient: shutdown 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 -- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efc64009f00 msgr2=0x7efc6400c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efc64009f00 0x7efc6400c2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc7000a640 tx=0x7efc7001d000 comp rx=0 tx=0).stop 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 -- 172.21.15.43:0/3219586204 shutdown_connections 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 --2- 172.21.15.43:0/3219586204 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efc64009f00 0x7efc6400c2f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 -- 172.21.15.43:0/3219586204 >> 172.21.15.43:0/3219586204 conn(0x7efc780ab2f0 msgr2=0x7efc780ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 -- 172.21.15.43:0/3219586204 shutdown_connections 2024-04-19T00:54:28.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 1 -- 172.21.15.43:0/3219586204 wait complete. 2024-04-19T00:54:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.766+0000 7efc7cb6d640 10 monclient: monmap: 2024-04-19T00:54:28.790 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.791 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 librados: starting msgr at 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 librados: starting objecter 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 Processor -- start 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- start start 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 librados: setting wanted keys 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 librados: calling monclient init 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: init 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: _add_conns ranks=[5,0,1,3,4,8,7,2,6] 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efc64009f00 0x7efc7812aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): picked mon.noname-h con 0x7efc64009f00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc7812b000 0x7efc78132260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): picked mon.noname-a con 0x7efc7812b000 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc7812b3e0 0x7efc781327a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): picked mon.noname-f con 0x7efc7812b3e0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efc78135cc0 con 0x7efc7812b000 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efc780a20a0 con 0x7efc7812b3e0 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7efc78135e40 con 0x7efc64009f00 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): authenticate will time out at 1538.901189s 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc7812b000 0x7efc78132260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc76d9d640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc7812b3e0 msgr2=0x7efc781327a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc76d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc7812b3e0 0x7efc781327a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc7812b000 0x7efc78132260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58100/0 (socket says 172.21.15.43:58100) 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 -- 172.21.15.43:0/2020438764 learned_addr learned my addr 172.21.15.43:0/2020438764 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): get_auth_request con 0x7efc7812b000 auth_method 0 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efc64009f00 0x7efc7812aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): get_auth_request con 0x7efc64009f00 auth_method 0 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient(hunting): handle_auth_done global_id 64674 payload 824 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efc64009f00 msgr2=0x7efc7812aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efc64009f00 0x7efc7812aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc7812b3e0 msgr2=0x7efc781327a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc7812b3e0 0x7efc781327a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 monclient: found mon.noname-a 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc78134940 con 0x7efc7812b000 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efc64009f00 0x7efc7812aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc75d9b640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc7812b000 0x7efc78132260 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e640 tx=0x7efc6800eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 client.?.objecter ms_handle_connect 0x7efc7812b000 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:28.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc68016020 con 0x7efc7812b000 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: dump: 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.796 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc68003ba0 con 0x7efc7812b000 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 5 monclient: authenticate success, global_id 64674 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: _renew_subs 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc78133e00 con 0x7efc7812b000 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc68003d50 con 0x7efc7812b000 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc74d99640 4 set_mon_vals no callback set 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 client.64674.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 monclient: dump: 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.798 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: _renew_subs 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc781344c0 con 0x7efc7812b000 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 librados: init done 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7efc68019020 con 0x7efc7812b000 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 0x7efc781330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efc7806d140 con 0x7efc78132ce0 2024-04-19T00:54:28.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 0x7efc781330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient: get_auth_request con 0x7efc78132ce0 auth_method 0 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efc58038e10 0x7efc5803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(374..374 src has 1..374) ==== 6170+0+0 (secure 0 0 0) 0x7efc6800f040 con 0x7efc7812b000 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 client.64674.objecter ms_dispatch 0x7efc78000ba0 osd_map(374..374 src has 1..374) 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 3 client.64674.objecter handle_osd_map got epochs [374,374] > 0 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 3 client.64674.objecter handle_osd_map decoding full epoch 374 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 10 monclient(hunting): handle_auth_done global_id 74261 payload 824 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc780693b0 con 0x7efc78132ce0 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 20 client.64674.objecter dump_active .. 0 homeless 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc76d9d640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efc58038e10 0x7efc5803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc76d9d640 10 monclient: get_auth_request con 0x7efc58038e10 auth_method 0 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 client.64674.objecter ms_handle_connect 0x7efc78132ce0 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc7659c640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 0x7efc781330c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d700 tx=0x7efc6000dbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.770+0000 7efc6effd640 10 client.64674.objecter resend_mon_ops 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc60003e30 con 0x7efc78132ce0 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 10 monclient: dump: 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.801 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efc60017070 con 0x7efc78132ce0 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 msgr2=0x7efc781330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 0x7efc781330c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d700 tx=0x7efc6000dbc0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 msgr2=0x7efc781330c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78132ce0 0x7efc781330c0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d700 tx=0x7efc6000dbc0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc6effd640 10 client.64674.objecter ms_handle_connect 0x7efc58038e10 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.774+0000 7efc76d9d640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efc58038e10 0x7efc5803b2d0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7efc6400cd30 tx=0x7efc70001f70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7cb6d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7cb6d640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 0x7efc78137b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7cb6d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efc78068240 con 0x7efc78137750 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 0x7efc78137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient: get_auth_request con 0x7efc78137750 auth_method 0 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 10 monclient(hunting): handle_auth_done global_id 74270 payload 824 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 1 -- 172.21.15.43:0/2020438764 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efc7809bf00 con 0x7efc78137750 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 client.64674.objecter ms_handle_connect 0x7efc78137750 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 client.64674.objecter resend_mon_ops 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc7659c640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 0x7efc78137b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc60003940 tx=0x7efc600059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efc6001cd20 con 0x7efc78137750 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 monclient: dump: 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.804 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9590 (secure 0 0 0) 0x7efc6001cd20 con 0x7efc78137750 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 msgr2=0x7efc78137b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 0x7efc78137b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc60003940 tx=0x7efc600059e0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 msgr2=0x7efc78137b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:28.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.778+0000 7efc6effd640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efc78137750 0x7efc78137b30 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efc60003940 tx=0x7efc600059e0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 48399, 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:28.806 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.807 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.808 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:28.809 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.810 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:28.811 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:28.813 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:28.815 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:28.817 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:28.819 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:28.821 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:28.825 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.826 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:54:28.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:28.828 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 10 librados: watch_flush enter 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 10 librados: watch_flush exit 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 20 client.64674.objecter shutdown clearing up homeless session... 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 10 client.64674.objecter successfully canceled tick 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efc58038e10 msgr2=0x7efc5803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efc58038e10 0x7efc5803b2d0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7efc6400cd30 tx=0x7efc70001f70 comp rx=0 tx=0).stop 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 10 monclient: shutdown 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc7812b000 msgr2=0x7efc78132260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 --2- 172.21.15.43:0/2020438764 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efc7812b000 0x7efc78132260 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e640 tx=0x7efc6800eb00 comp rx=0 tx=0).stop 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc76d9d640 1 -- 172.21.15.43:0/2020438764 reap_dead start 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc76d9d640 1 -- 172.21.15.43:0/2020438764 reap_dead start 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 shutdown_connections 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 >> 172.21.15.43:0/2020438764 conn(0x7efc780ab2f0 msgr2=0x7efc780a5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 shutdown_connections 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 -- 172.21.15.43:0/2020438764 wait complete. 2024-04-19T00:54:28.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.782+0000 7efc7cb6d640 1 librados: shutdown 2024-04-19T00:54:28.831 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:28.831 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-04-19T00:54:28.832 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:54:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient: get_monmap_and_config 2024-04-19T00:54:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient: monmap: 2024-04-19T00:54:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.952 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:28.955102+0000 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:28.955102+0000 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.953 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.954 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:28.954 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:28.954 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 Processor -- start 2024-04-19T00:54:28.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 -- start start 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient: init 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient: _add_conns ranks=[8,9,0,5,3,4,1,7,2,6,10] 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f70a0138a50 0x7f70a0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): picked mon.noname-e con 0x7f70a0138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70a00a5bb0 0x7f70a0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): picked mon.noname-a con 0x7f70a00a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a0137960 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a00a0d40 con 0x7f70a0137960 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f70a00a1020 con 0x7f70a00a5bb0 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f70a01398b0 con 0x7f70a0138a50 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a4a36640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f70a0138a50 msgr2=0x7f70a0138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a6c9a640 10 monclient(hunting): authenticate will time out at 1539.083004s 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f70a4a36640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f70a0138a50 0x7f70a0138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70a00a5bb0 0x7f70a0139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70a00a5bb0 0x7f70a0139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58116/0 (socket says 172.21.15.43:58116) 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 -- 172.21.15.43:0/839062608 learned_addr learned my addr 172.21.15.43:0/839062608 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): get_auth_request con 0x7f70a00a5bb0 auth_method 0 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient(hunting): handle_auth_done global_id 64683 payload 274 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 -- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f70a0138a50 msgr2=0x7f70a0138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 --2- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f70a0138a50 0x7f70a0138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 -- 172.21.15.43:0/839062608 >> v1:172.21.15.43:6789/0 conn(0x7f70a0137960 legacy=0x7f70a0137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 monclient: found mon.noname-a 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709f7fe640 1 -- 172.21.15.43:0/839062608 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2598358821 0 0) 0x7f70a00a0d40 con 0x7f70a0137960 2024-04-19T00:54:28.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:54:28.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709ffff640 1 -- 172.21.15.43:0/839062608 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00a0a60 con 0x7f70a00a5bb0 2024-04-19T00:54:28.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.950+0000 7f709f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709ffff640 1 --2- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a00a5bb0 0x7f70a0139370 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7094009870 tx=0x7f709401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445d089f5e4f27ae server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 1 -- 172.21.15.43:0/839062608 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70940b0070 con 0x7f70a00a5bb0 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: dump: 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.967 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 1 -- 172.21.15.43:0/839062608 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70940aa810 con 0x7f70a00a5bb0 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 5 monclient: authenticate success, global_id 64683 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 1 -- 172.21.15.43:0/839062608 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70940aae30 con 0x7f70a00a5bb0 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 4 set_mon_vals no callback set 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f709f7fe640 10 monclient: dump: 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.969 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.970 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 10 monclient: shutdown 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 -- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a00a5bb0 msgr2=0x7f70a0139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a00a5bb0 0x7f70a0139370 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7094009870 tx=0x7f709401c440 comp rx=0 tx=0).stop 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 -- 172.21.15.43:0/839062608 shutdown_connections 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a00a5bb0 0x7f70a0139370 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/839062608 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f70a0138a50 0x7f70a0138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 -- 172.21.15.43:0/839062608 >> 172.21.15.43:0/839062608 conn(0x7f70a00a7cb0 msgr2=0x7f70a00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 -- 172.21.15.43:0/839062608 shutdown_connections 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 -- 172.21.15.43:0/839062608 wait complete. 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 10 monclient: build_initial_monmap 2024-04-19T00:54:28.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 10 monclient: monmap: 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:28.972 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 librados: starting msgr at 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 librados: starting objecter 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 Processor -- start 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 -- start start 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 librados: setting wanted keys 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 1 librados: calling monclient init 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.954+0000 7f70a6c9a640 10 monclient: init 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient: _add_conns ranks=[3,4,6,5,2,8,7,1,0] 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a00a5bb0 0x7f70a012eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): picked mon.noname-g con 0x7f70a00a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:28.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a0137960 0x7f70a012f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): picked mon.noname-c con 0x7f70a0137960 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70a0138a50 0x7f70a01d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): picked mon.noname-d con 0x7f70a0138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70a0143bb0 con 0x7f70a00a5bb0 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f70a0143a30 con 0x7f70a0137960 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f70a0143d30 con 0x7f70a0138a50 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): authenticate will time out at 1539.088376s 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70a0138a50 0x7f70a01d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a0137960 0x7f70a012f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70a0138a50 0x7f70a01d0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:49524/0 (socket says 172.21.15.43:49524) 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a0137960 0x7f70a012f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50332/0 (socket says 172.21.15.43:50332) 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 -- 172.21.15.43:0/3181750985 learned_addr learned my addr 172.21.15.43:0/3181750985 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): get_auth_request con 0x7f70a0138a50 auth_method 0 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): get_auth_request con 0x7f70a0137960 auth_method 0 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a00a5bb0 0x7f70a012eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): get_auth_request con 0x7f70a00a5bb0 auth_method 0 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient(hunting): handle_auth_done global_id 74274 payload 824 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a0137960 msgr2=0x7f70a012f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a0137960 0x7f70a012f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a00a5bb0 msgr2=0x7f70a012eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a00a5bb0 0x7f70a012eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient: _finish_hunting 0 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 monclient: found mon.noname-d 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00a5420 con 0x7f70a0138a50 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a0137960 0x7f70a012f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a5237640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70a0138a50 0x7f70a01d0d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f709800c760 tx=0x7f709800cc20 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 client.?.objecter ms_handle_connect 0x7f70a0138a50 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7098003bb0 con 0x7f70a0138a50 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: dump: 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.977 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.978 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7098003d70 con 0x7f70a0138a50 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 5 monclient: authenticate success, global_id 74274 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient: _renew_subs 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a009fc90 con 0x7f70a0138a50 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f709800a640 con 0x7f70a0138a50 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:54:28.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709effd640 4 set_mon_vals no callback set 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 client.74274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: dump: 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.980 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient: _renew_subs 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a0135bb0 con 0x7f70a0138a50 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 librados: init done 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 0x7f70a01d1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f70a0136200 con 0x7f70a01d1260 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f709800a7e0 con 0x7f70a0138a50 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 0x7f70a01d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient: get_auth_request con 0x7f70a01d1260 auth_method 0 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7084038e10 0x7f708403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(374..374 src has 1..374) ==== 6170+0+0 (secure 0 0 0) 0x7f7098049210 con 0x7f70a0138a50 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 client.74274.objecter ms_dispatch 0x7f70a0000ba0 osd_map(374..374 src has 1..374) 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 3 client.74274.objecter handle_osd_map got epochs [374,374] > 0 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 3 client.74274.objecter handle_osd_map decoding full epoch 374 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 10 monclient(hunting): handle_auth_done global_id 64471 payload 824 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 20 client.74274.objecter dump_active .. 0 homeless 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70a01d1b80 con 0x7f70a01d1260 2024-04-19T00:54:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 client.74274.objecter ms_handle_connect 0x7f70a01d1260 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 client.74274.objecter resend_mon_ops 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f70a4a36640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 0x7f70a01d1640 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f708c003990 tx=0x7f708c00db90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7084038e10 0x7f708403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 10 monclient: get_auth_request con 0x7f7084038e10 auth_method 0 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f708c018070 con 0x7f70a01d1260 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: dump: 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f708c005a50 con 0x7f70a01d1260 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 msgr2=0x7f70a01d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 0x7f70a01d1640 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f708c003990 tx=0x7f708c00db90 comp rx=0 tx=0).stop 2024-04-19T00:54:28.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 msgr2=0x7f70a01d1640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:28.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a01d1260 0x7f70a01d1640 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f708c003990 tx=0x7f708c00db90 comp rx=0 tx=0).stop 2024-04-19T00:54:28.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709d7fa640 10 client.74274.objecter ms_handle_connect 0x7f7084038e10 2024-04-19T00:54:28.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.958+0000 7f709ffff640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7084038e10 0x7f708403b2d0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f70940ad000 tx=0x7f70940a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:28.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 0x7f70a009c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f70a006d140 con 0x7f70a009c4b0 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 0x7f70a009c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient: get_auth_request con 0x7f70a009c4b0 auth_method 0 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 10 monclient(hunting): handle_auth_done global_id 64480 payload 824 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 1 -- 172.21.15.43:0/3181750985 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70a0076190 con 0x7f70a009c4b0 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a4a36640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 0x7f70a009c8c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f708c003990 tx=0x7f708c0059e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 client.74274.objecter ms_handle_connect 0x7f70a009c4b0 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 client.74274.objecter resend_mon_ops 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f708c018030 con 0x7f70a009c4b0 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 monclient: dump: 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7f708c003e00 con 0x7f70a009c4b0 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 msgr2=0x7f70a009c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 0x7f70a009c8c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f708c003990 tx=0x7f708c0059e0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 msgr2=0x7f70a009c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f709d7fa640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f70a009c4b0 0x7f70a009c8c0 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f708c003990 tx=0x7f708c0059e0 comp rx=0 tx=0).stop 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 195238, 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:28.990 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.991 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.992 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:28.993 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:28.994 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:28.995 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:28.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:28.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.000 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.002 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.005 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.006 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.007 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.009 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.010 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:29.011 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 10 librados: watch_flush enter 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 10 librados: watch_flush exit 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 20 client.74274.objecter shutdown clearing up homeless session... 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 10 client.74274.objecter successfully canceled tick 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7084038e10 msgr2=0x7f708403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7084038e10 0x7f708403b2d0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f70940ad000 tx=0x7f70940a6000 comp rx=0 tx=0).stop 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 10 monclient: shutdown 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70a0138a50 msgr2=0x7f70a01d0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 --2- 172.21.15.43:0/3181750985 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70a0138a50 0x7f70a01d0d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f709800c760 tx=0x7f709800cc20 comp rx=0 tx=0).stop 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a5237640 1 -- 172.21.15.43:0/3181750985 reap_dead start 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a5237640 1 -- 172.21.15.43:0/3181750985 reap_dead start 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 shutdown_connections 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 >> 172.21.15.43:0/3181750985 conn(0x7f70a00a7cb0 msgr2=0x7f70a006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 shutdown_connections 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 -- 172.21.15.43:0/3181750985 wait complete. 2024-04-19T00:54:29.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:28.966+0000 7f70a6c9a640 1 librados: shutdown 2024-04-19T00:54:29.014 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:29.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-04-19T00:54:29.015 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:54:29.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient: get_monmap_and_config 2024-04-19T00:54:29.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient: monmap: 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:29.143494+0000 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:29.143494+0000 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.141 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.142 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.142 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.142 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.142 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:29.142 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:29.142 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 Processor -- start 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 -- start start 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient: init 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient: _add_conns ranks=[3,4,8,2,10,6,1,9,0,5,7] 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): picked mon.noname-g con 0x7f7fa0138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 0x7f7fa01395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): picked mon.noname-c con 0x7f7fa00a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7fa01358b0 0x7f7fa0135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): picked mon.noname-e con 0x7f7fa01358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa00a4410 con 0x7f7fa0138cd0 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f7fa0139b30 con 0x7f7fa00a5b80 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7fa00a3dc0 con 0x7f7fa01358b0 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa672c640 10 monclient(hunting): authenticate will time out at 1539.270787s 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa4cc9640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7fa01358b0 msgr2=0x7f7fa0135d30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7fa4cc9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7fa01358b0 0x7f7fa0135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 0x7f7fa01395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 0x7f7fa01395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50364/0 (socket says 172.21.15.43:50364) 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 -- 172.21.15.43:0/3171664405 learned_addr learned my addr 172.21.15.43:0/3171664405 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): get_auth_request con 0x7f7fa00a5b80 auth_method 0 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9ffff640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9ffff640 10 monclient(hunting): get_auth_request con 0x7f7fa0138cd0 auth_method 0 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient(hunting): handle_auth_done global_id 64489 payload 274 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 -- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7fa01358b0 msgr2=0x7f7fa0135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7fa01358b0 0x7f7fa0135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 -- 172.21.15.43:0/3171664405 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fa0138cd0 msgr2=0x7f7fa01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 monclient: found mon.noname-c 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:54:29.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.138+0000 7f7f9f7fe640 1 -- 172.21.15.43:0/3171664405 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa00a40a0 con 0x7f7fa00a5b80 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9f7fe640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 0x7f7fa01395f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7f880098a0 tx=0x7f7f8801c400 comp rx=0 tx=0).ready entity=mon.4 client_cookie=adf0d7a305f97dba server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 1 -- 172.21.15.43:0/3171664405 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f880b0070 con 0x7f7fa00a5b80 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: dump: 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.155 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.156 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: _finish_auth 0 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: _check_auth_tickets 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 1 -- 172.21.15.43:0/3171664405 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f880aa810 con 0x7f7fa00a5b80 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 1 -- 172.21.15.43:0/3171664405 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f880a9300 con 0x7f7fa00a5b80 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 5 monclient: authenticate success, global_id 64489 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 4 set_mon_vals no callback set 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7f9effd640 10 monclient: dump: 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.157 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: shutdown 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 -- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 msgr2=0x7f7fa01395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 0x7f7fa01395f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7f880098a0 tx=0x7f7f8801c400 comp rx=0 tx=0).stop 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 -- 172.21.15.43:0/3171664405 shutdown_connections 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7fa01358b0 0x7f7fa0135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7fa00a5b80 0x7f7fa01395f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 --2- 172.21.15.43:0/3171664405 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 -- 172.21.15.43:0/3171664405 >> 172.21.15.43:0/3171664405 conn(0x7f7fa00ab2f0 msgr2=0x7f7fa00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 -- 172.21.15.43:0/3171664405 shutdown_connections 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 -- 172.21.15.43:0/3171664405 wait complete. 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: monmap: 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.159 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 librados: starting msgr at 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 librados: starting objecter 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 Processor -- start 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 -- start start 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 librados: setting wanted keys 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 1 librados: calling monclient init 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: init 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.142+0000 7f7fa672c640 10 monclient: _add_conns ranks=[7,1,2,0,3,4,6,5,8] 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7fa00a5b80 0x7f7fa01d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): picked mon.noname-i con 0x7f7fa00a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fa01358b0 0x7f7fa01d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): picked mon.noname-f con 0x7f7fa01358b0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): picked mon.noname-b con 0x7f7fa0138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:29.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa00a2010 con 0x7f7fa01358b0 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7fa00a1e90 con 0x7f7fa0138cd0 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7fa0142aa0 con 0x7f7fa00a5b80 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): authenticate will time out at 1539.276072s 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9f7fe640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fa01358b0 msgr2=0x7f7fa01d1070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fa01358b0 0x7f7fa01d1070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01d15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35100/0 (socket says 172.21.15.43:35100) 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 -- 172.21.15.43:0/689322432 learned_addr learned my addr 172.21.15.43:0/689322432 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): get_auth_request con 0x7f7fa0138cd0 auth_method 0 2024-04-19T00:54:29.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7fa00a5b80 0x7f7fa01d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): get_auth_request con 0x7f7fa00a5b80 auth_method 0 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient(hunting): handle_auth_done global_id 74279 payload 824 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7fa00a5b80 msgr2=0x7f7fa01d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7fa00a5b80 0x7f7fa01d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fa01358b0 msgr2=0x7f7fa01d1070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7fa01358b0 0x7f7fa01d1070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient: _finish_hunting 0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 monclient: found mon.noname-b 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa00a3590 con 0x7f7fa0138cd0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa4cc9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01d15b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800be60 tx=0x7f7f9800bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 client.?.objecter ms_handle_connect 0x7f7fa0138cd0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f9800cd20 con 0x7f7fa0138cd0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: dump: 2024-04-19T00:54:29.164 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.165 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: _finish_auth 0 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: _check_auth_tickets 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f98010030 con 0x7f7fa0138cd0 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f98015cc0 con 0x7f7fa0138cd0 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 5 monclient: authenticate success, global_id 74279 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient: _renew_subs 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa00a3040 con 0x7f7fa0138cd0 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9dffb640 4 set_mon_vals no callback set 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:29.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 client.74279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: dump: 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.167 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient: _renew_subs 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa013e6a0 con 0x7f7fa0138cd0 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 librados: init done 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 0x7f7fa0134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7fa006d140 con 0x7f7fa01348a0 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7f980153e0 con 0x7f7fa0138cd0 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 0x7f7fa0134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient: get_auth_request con 0x7f7fa01348a0 auth_method 0 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f80038e10 0x7f7f8003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(374..374 src has 1..374) ==== 6170+0+0 (secure 0 0 0) 0x7f7f980498c0 con 0x7f7fa0138cd0 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 client.74279.objecter ms_dispatch 0x7f7fa0000ba0 osd_map(374..374 src has 1..374) 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 3 client.74279.objecter handle_osd_map got epochs [374,374] > 0 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 3 client.74279.objecter handle_osd_map decoding full epoch 374 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_done global_id 74283 payload 824 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7fa00693b0 con 0x7f7fa01348a0 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 20 client.74279.objecter dump_active .. 0 homeless 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9f7fe640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f80038e10 0x7f7f8003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 client.74279.objecter ms_handle_connect 0x7f7fa01348a0 2024-04-19T00:54:29.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 client.74279.objecter resend_mon_ops 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9f7fe640 10 monclient: get_auth_request con 0x7f7f80038e10 auth_method 0 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9ffff640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 0x7f7fa0134c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000f410 tx=0x7f7f9000f8d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f90019020 con 0x7f7fa01348a0 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: dump: 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.170 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f90014670 con 0x7f7fa01348a0 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 msgr2=0x7f7fa0134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 0x7f7fa0134c80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000f410 tx=0x7f7f9000f8d0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 msgr2=0x7f7fa0134c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:29.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01348a0 0x7f7fa0134c80 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000f410 tx=0x7f7f9000f8d0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9cff9640 10 client.74279.objecter ms_handle_connect 0x7f7f80038e10 2024-04-19T00:54:29.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.146+0000 7f7f9f7fe640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f80038e10 0x7f7f8003b2d0 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f7f880ad000 tx=0x7f7f88008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:29.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 0x7f7fa009da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7fa0068240 con 0x7f7fa01351c0 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 0x7f7fa009da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient: get_auth_request con 0x7f7fa01351c0 auth_method 0 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 10 monclient(hunting): handle_auth_done global_id 74292 payload 824 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 1 -- 172.21.15.43:0/689322432 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7fa0076190 con 0x7f7fa01351c0 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9ffff640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 0x7f7fa009da80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000c10 tx=0x7f7f900046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 client.74279.objecter ms_handle_connect 0x7f7fa01351c0 2024-04-19T00:54:29.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 client.74279.objecter resend_mon_ops 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7f90019040 con 0x7f7fa01351c0 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 monclient: dump: 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.175 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9420 (secure 0 0 0) 0x7f7f90004030 con 0x7f7fa01351c0 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 msgr2=0x7f7fa009da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 0x7f7fa009da80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000c10 tx=0x7f7f900046d0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 msgr2=0x7f7fa009da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7f9cff9640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7fa01351c0 0x7f7fa009da80 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000c10 tx=0x7f7f900046d0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.176 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 48767, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.177 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.178 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.179 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:29.180 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.181 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.183 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:29.185 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.188 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.190 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.193 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.194 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.195 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.196 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.197 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.198 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.199 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:54:29.199 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.199 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.199 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.199 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:29.199 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 10 librados: watch_flush enter 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 10 librados: watch_flush exit 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 20 client.74279.objecter shutdown clearing up homeless session... 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 10 client.74279.objecter successfully canceled tick 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f80038e10 msgr2=0x7f7f8003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7f80038e10 0x7f7f8003b2d0 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f7f880ad000 tx=0x7f7f88008040 comp rx=0 tx=0).stop 2024-04-19T00:54:29.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 10 monclient: shutdown 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fa0138cd0 msgr2=0x7f7fa01d15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 --2- 172.21.15.43:0/689322432 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7fa0138cd0 0x7f7fa01d15b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7f9800be60 tx=0x7f7f9800bf50 comp rx=0 tx=0).stop 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa4cc9640 1 -- 172.21.15.43:0/689322432 reap_dead start 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa4cc9640 1 -- 172.21.15.43:0/689322432 reap_dead start 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 shutdown_connections 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 >> 172.21.15.43:0/689322432 conn(0x7f7fa00ab2f0 msgr2=0x7f7fa01368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 shutdown_connections 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 -- 172.21.15.43:0/689322432 wait complete. 2024-04-19T00:54:29.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.154+0000 7f7fa672c640 1 librados: shutdown 2024-04-19T00:54:29.203 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:29.203 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon g 2024-04-19T00:54:29.203 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:54:29.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient: get_monmap_and_config 2024-04-19T00:54:29.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient: monmap: 2024-04-19T00:54:29.324 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.324 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.324 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:29.327260+0000 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:29.327260+0000 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:29.325 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 Processor -- start 2024-04-19T00:54:29.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 -- start start 2024-04-19T00:54:29.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient: init 2024-04-19T00:54:29.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient: _add_conns ranks=[4,10,2,1,6,0,5,7,9,8,3] 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 0x7f07dc1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): picked mon.noname-c con 0x7f07dc138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07dc0a5b80 0x7f07dc1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): picked mon.noname-f con 0x7f07dc0a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc1358b0 0x7f07dc135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): picked mon.noname-b con 0x7f07dc1358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f07dc0a4410 con 0x7f07dc0a5b80 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc139b30 con 0x7f07dc1358b0 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.328 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f07dc0a3dc0 con 0x7f07dc138cd0 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07e18c0640 10 monclient(hunting): authenticate will time out at 1539.454372s 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 0x7f07dc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 0x7f07dc1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50366/0 (socket says 172.21.15.43:50366) 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc1358b0 0x7f07dc135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 1 -- 172.21.15.43:0/3713079938 learned_addr learned my addr 172.21.15.43:0/3713079938 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): get_auth_request con 0x7f07dc138cd0 auth_method 0 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07da7fc640 1 -- 172.21.15.43:0/3713079938 >> v2:172.21.15.161:3300/0 conn(0x7f07dc0a5b80 msgr2=0x7f07dc1395f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): get_auth_request con 0x7f07dc1358b0 auth_method 0 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07da7fc640 1 --2- 172.21.15.43:0/3713079938 >> v2:172.21.15.161:3300/0 conn(0x7f07dc0a5b80 0x7f07dc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient(hunting): handle_auth_done global_id 74288 payload 274 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 -- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 msgr2=0x7f07dc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 --2- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 0x7f07dc1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 -- 172.21.15.43:0/3713079938 >> v2:172.21.15.161:3300/0 conn(0x7f07dc0a5b80 msgr2=0x7f07dc1395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 --2- 172.21.15.43:0/3713079938 >> v2:172.21.15.161:3300/0 conn(0x7f07dc0a5b80 0x7f07dc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:54:29.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 monclient: found mon.noname-b 2024-04-19T00:54:29.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:29.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07db7fe640 1 -- 172.21.15.43:0/3713079938 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc0a40a0 con 0x7f07dc1358b0 2024-04-19T00:54:29.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.322+0000 7f07daffd640 1 --2- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 0x7f07dc1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07db7fe640 1 --2- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc1358b0 0x7f07dc135d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07d40097c0 tx=0x7f07d401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3487f47cc1ba66 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 1 -- 172.21.15.43:0/3713079938 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07d40b0070 con 0x7f07dc1358b0 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: dump: 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.338 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: _finish_auth 0 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:29.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 5 monclient: authenticate success, global_id 74288 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 1 -- 172.21.15.43:0/3713079938 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d40aa810 con 0x7f07dc1358b0 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 1 -- 172.21.15.43:0/3713079938 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07d40a9300 con 0x7f07dc1358b0 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 4 set_mon_vals no callback set 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07d9ffb640 10 monclient: dump: 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.340 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 10 monclient: shutdown 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 -- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc1358b0 msgr2=0x7f07dc135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 --2- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc1358b0 0x7f07dc135d40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f07d40097c0 tx=0x7f07d401c320 comp rx=0 tx=0).stop 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 -- 172.21.15.43:0/3713079938 shutdown_connections 2024-04-19T00:54:29.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 --2- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc1358b0 0x7f07dc135d40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 --2- 172.21.15.43:0/3713079938 >> v2:172.21.15.161:3300/0 conn(0x7f07dc0a5b80 0x7f07dc1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 --2- 172.21.15.43:0/3713079938 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc138cd0 0x7f07dc1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 -- 172.21.15.43:0/3713079938 >> 172.21.15.43:0/3713079938 conn(0x7f07dc0ab2f0 msgr2=0x7f07dc0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 -- 172.21.15.43:0/3713079938 shutdown_connections 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 -- 172.21.15.43:0/3713079938 wait complete. 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 10 monclient: monmap: 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.342 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 librados: starting msgr at 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 librados: starting objecter 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 Processor -- start 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 -- start start 2024-04-19T00:54:29.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 librados: setting wanted keys 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 1 librados: calling monclient init 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 10 monclient: init 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.326+0000 7f07e18c0640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient: _add_conns ranks=[8,4,2,5,0,1,3,7,6] 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f07dc0a5b80 0x7f07dc1d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): picked mon.noname-e con 0x7f07dc0a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc1358b0 0x7f07dc13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): picked mon.noname-c con 0x7f07dc1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc138cd0 0x7f07dc13f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): picked mon.noname-b con 0x7f07dc138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc0a2010 con 0x7f07dc138cd0 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f07dc0a1e90 con 0x7f07dc1358b0 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f07dc142aa0 con 0x7f07dc0a5b80 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): authenticate will time out at 1539.460086s 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f07dc0a5b80 msgr2=0x7f07dc1d0a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f07dc0a5b80 0x7f07dc1d0a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc1358b0 0x7f07dc13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc138cd0 0x7f07dc13f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc138cd0 0x7f07dc13f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35114/0 (socket says 172.21.15.43:35114) 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 -- 172.21.15.43:0/1715235658 learned_addr learned my addr 172.21.15.43:0/1715235658 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): get_auth_request con 0x7f07dc1358b0 auth_method 0 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): get_auth_request con 0x7f07dc138cd0 auth_method 0 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient(hunting): handle_auth_done global_id 74297 payload 824 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f07dc0a5b80 msgr2=0x7f07dc1d0a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f07dc0a5b80 0x7f07dc1d0a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc1358b0 msgr2=0x7f07dc13ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc1358b0 0x7f07dc13ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 monclient: found mon.noname-b 2024-04-19T00:54:29.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc13b5b0 con 0x7f07dc138cd0 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f07dc1358b0 0x7f07dc13ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07db7fe640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc138cd0 0x7f07dc13f320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07d40097c0 tx=0x7f07d4019120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 client.?.objecter ms_handle_connect 0x7f07dc138cd0 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07d40b0040 con 0x7f07dc138cd0 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: dump: 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.347 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: _finish_auth 0 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07d40044b0 con 0x7f07dc138cd0 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 5 monclient: authenticate success, global_id 74297 2024-04-19T00:54:29.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient: _renew_subs 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc13a2b0 con 0x7f07dc138cd0 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07d40aac30 con 0x7f07dc138cd0 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: dump: 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.349 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 client.74297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d97fa640 4 set_mon_vals no callback set 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient: _renew_subs 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc0a3590 con 0x7f07dc138cd0 2024-04-19T00:54:29.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 librados: init done 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 0x7f07dc133af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc06d140 con 0x7f07dc133710 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f07d4004040 con 0x7f07dc138cd0 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f07bc038e30 0x7f07bc03b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(375..375 src has 1..375) ==== 6170+0+0 (secure 0 0 0) 0x7f07d40b9070 con 0x7f07dc138cd0 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 0x7f07dc133af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 client.74297.objecter ms_dispatch 0x7f07dc000ba0 osd_map(375..375 src has 1..375) 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 3 client.74297.objecter handle_osd_map got epochs [375,375] > 0 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 3 client.74297.objecter handle_osd_map decoding full epoch 375 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient: get_auth_request con 0x7f07dc133710 auth_method 0 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 20 client.74297.objecter dump_active .. 0 homeless 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f07bc038e30 0x7f07bc03b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 10 monclient: get_auth_request con 0x7f07bc038e30 auth_method 0 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 10 monclient(hunting): handle_auth_done global_id 19362 payload 824 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07dc0693b0 con 0x7f07dc133710 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07daffd640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 0x7f07dc133af0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0072a0 tx=0x7f07cc00c490 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 client.74297.objecter ms_handle_connect 0x7f07dc133710 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 client.74297.objecter resend_mon_ops 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07cc0090d0 con 0x7f07dc133710 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07da7fc640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f07bc038e30 0x7f07bc03b2f0 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f07c4006e50 tx=0x7f07c4006da0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:54:29.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 monclient: dump: 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.353 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.330+0000 7f07d3fff640 10 client.74297.objecter ms_handle_connect 0x7f07bc038e30 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07cc019070 con 0x7f07dc133710 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 msgr2=0x7f07dc133af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 0x7f07dc133af0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0072a0 tx=0x7f07cc00c490 comp rx=0 tx=0).stop 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 msgr2=0x7f07dc133af0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:29.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.334+0000 7f07d3fff640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc133710 0x7f07dc133af0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07e18c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07e18c0640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 0x7f07dc12cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07e18c0640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc068240 con 0x7f07dc134030 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 0x7f07dc12cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient: get_auth_request con 0x7f07dc134030 auth_method 0 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 10 monclient(hunting): handle_auth_done global_id 19371 payload 824 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.338+0000 7f07daffd640 1 -- 172.21.15.43:0/1715235658 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07dc13f860 con 0x7f07dc134030 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07daffd640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 0x7f07dc12cb50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f07cc018000 tx=0x7f07cc004ac0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 client.74297.objecter ms_handle_connect 0x7f07dc134030 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 client.74297.objecter resend_mon_ops 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f07cc014bc0 con 0x7f07dc134030 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 monclient: dump: 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.357 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9253 (secure 0 0 0) 0x7f07cc014750 con 0x7f07dc134030 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:29.358 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 msgr2=0x7f07dc12cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 0x7f07dc12cb50 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f07cc018000 tx=0x7f07cc004ac0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 msgr2=0x7f07dc12cb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07d3fff640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f07dc134030 0x7f07dc12cb50 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 612829, 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:29.359 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.360 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.361 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:29.362 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.363 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:29.364 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.365 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.366 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:29.368 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.369 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.370 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:29.372 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.373 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:29.374 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.378 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.380 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.380 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:29.380 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:29.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 10 librados: watch_flush enter 2024-04-19T00:54:29.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 10 librados: watch_flush exit 2024-04-19T00:54:29.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 20 client.74297.objecter shutdown clearing up homeless session... 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 10 client.74297.objecter successfully canceled tick 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f07bc038e30 msgr2=0x7f07bc03b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f07bc038e30 0x7f07bc03b2f0 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f07c4006e50 tx=0x7f07c4006da0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 10 monclient: shutdown 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc138cd0 msgr2=0x7f07dc13f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 --2- 172.21.15.43:0/1715235658 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f07dc138cd0 0x7f07dc13f320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07d40097c0 tx=0x7f07d4019120 comp rx=0 tx=0).stop 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07db7fe640 1 -- 172.21.15.43:0/1715235658 reap_dead start 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07db7fe640 1 -- 172.21.15.43:0/1715235658 reap_dead start 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 shutdown_connections 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 >> 172.21.15.43:0/1715235658 conn(0x7f07dc0ab2f0 msgr2=0x7f07dc0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 shutdown_connections 2024-04-19T00:54:29.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 -- 172.21.15.43:0/1715235658 wait complete. 2024-04-19T00:54:29.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.342+0000 7f07e18c0640 1 librados: shutdown 2024-04-19T00:54:29.383 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:29.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-04-19T00:54:29.383 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:54:29.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient: get_monmap_and_config 2024-04-19T00:54:29.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient: monmap: 2024-04-19T00:54:29.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:29.503157+0000 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:29.503157+0000 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.501 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.502 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:29.502 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:29.502 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 1 Processor -- start 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 1 -- start start 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient: init 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient: _add_conns ranks=[0,1,9,3,7,2,8,6,5,4,10] 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc618138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:54:29.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc6180a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc6181397c0 0x7fc6181419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient(hunting): picked mon.noname-a con 0x7fc6181397c0 addr v2:172.21.15.43:3300/0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6180a3dc0 con 0x7fc618138cd0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6180a40a0 con 0x7fc6180a5b80 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fc6180a4410 con 0x7fc6181397c0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61ea7e640 10 monclient(hunting): authenticate will time out at 1539.630300s 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc6181397c0 0x7fc6181419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc617fff640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7fc6180a5b80 legacy=0x7fc61813e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc6181397c0 0x7fc6181419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58132/0 (socket says 172.21.15.43:58132) 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 -- 172.21.15.43:0/3705404062 learned_addr learned my addr 172.21.15.43:0/3705404062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): get_auth_request con 0x7fc6181397c0 auth_method 0 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61c81a640 1 --1- 172.21.15.43:0/3705404062 >> v1:172.21.15.43:6789/0 conn(0x7fc618138cd0 0x7fc6181390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:49150/0 (socket says 172.21.15.43:49150) 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient(hunting): handle_auth_done global_id 64701 payload 274 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 -- 172.21.15.43:0/3705404062 >> v1:172.21.15.161:6789/0 conn(0x7fc6180a5b80 legacy=0x7fc61813e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 -- 172.21.15.43:0/3705404062 >> v1:172.21.15.43:6789/0 conn(0x7fc618138cd0 legacy=0x7fc6181390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient: _finish_hunting 0 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 monclient: found mon.noname-a 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.498+0000 7fc61d01b640 1 -- 172.21.15.43:0/3705404062 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6180a2ae0 con 0x7fc6181397c0 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61d01b640 1 --2- 172.21.15.43:0/3705404062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6181397c0 0x7fc6181419a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc6180a4590 tx=0x7fc60801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd6a460422afebb server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 1 -- 172.21.15.43:0/3705404062 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6180a20a0 con 0x7fc6181397c0 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: dump: 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.509 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: _finish_auth 0 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: _check_auth_tickets 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 1 -- 172.21.15.43:0/3705404062 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6080090d0 con 0x7fc6181397c0 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 5 monclient: authenticate success, global_id 64701 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 4 set_mon_vals no callback set 2024-04-19T00:54:29.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 1 -- 172.21.15.43:0/3705404062 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6080a5c20 con 0x7fc6181397c0 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc6177fe640 10 monclient: dump: 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.511 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: shutdown 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- 172.21.15.43:0/3705404062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6181397c0 msgr2=0x7fc6181419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 --2- 172.21.15.43:0/3705404062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6181397c0 0x7fc6181419a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc6180a4590 tx=0x7fc60801c8d0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- 172.21.15.43:0/3705404062 shutdown_connections 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 --2- 172.21.15.43:0/3705404062 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fc6181397c0 0x7fc6181419a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- 172.21.15.43:0/3705404062 >> 172.21.15.43:0/3705404062 conn(0x7fc6180ab2f0 msgr2=0x7fc6180ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- 172.21.15.43:0/3705404062 shutdown_connections 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- 172.21.15.43:0/3705404062 wait complete. 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: monmap: 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.513 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 librados: starting msgr at 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 librados: starting objecter 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 Processor -- start 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- start start 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 librados: setting wanted keys 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 librados: calling monclient init 2024-04-19T00:54:29.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: init 2024-04-19T00:54:29.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient: _add_conns ranks=[8,2,5,4,6,1,7,0,3] 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc6180a5b80 0x7fc61809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): picked mon.noname-e con 0x7fc6180a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc618138cd0 0x7fc61809bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): picked mon.noname-b con 0x7fc618138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181397c0 0x7fc61809c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): picked mon.noname-h con 0x7fc6181397c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc618144a90 con 0x7fc618138cd0 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc618144910 con 0x7fc6181397c0 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fc618144c10 con 0x7fc6180a5b80 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61ea7e640 10 monclient(hunting): authenticate will time out at 1539.635421s 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61c81a640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc6180a5b80 msgr2=0x7fc61809ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61c81a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc6180a5b80 0x7fc61809ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc618138cd0 0x7fc61809bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc618138cd0 0x7fc61809bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35122/0 (socket says 172.21.15.43:35122) 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 -- 172.21.15.43:0/1322982305 learned_addr learned my addr 172.21.15.43:0/1322982305 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): get_auth_request con 0x7fc618138cd0 auth_method 0 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc61d01b640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181397c0 0x7fc61809c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient(hunting): handle_auth_done global_id 74306 payload 824 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc6180a5b80 msgr2=0x7fc61809ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc6180a5b80 0x7fc61809ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181397c0 msgr2=0x7fc61809c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181397c0 0x7fc61809c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient: _finish_hunting 0 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 monclient: found mon.noname-b 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.502+0000 7fc617fff640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61809d9a0 con 0x7fc618138cd0 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc617fff640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc618138cd0 0x7fc61809bfb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc60000b510 tx=0x7fc60000b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 client.?.objecter ms_handle_connect 0x7fc618138cd0 2024-04-19T00:54:29.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc600005070 con 0x7fc618138cd0 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: dump: 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.524 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: _finish_auth 0 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: _check_auth_tickets 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc600015070 con 0x7fc618138cd0 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6000056c0 con 0x7fc618138cd0 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6167fc640 4 set_mon_vals no callback set 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: dump: 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.526 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 5 monclient: authenticate success, global_id 74306 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 monclient: _renew_subs 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc618136f40 con 0x7fc618138cd0 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 client.74306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 monclient: _renew_subs 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:29.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc618136760 con 0x7fc618138cd0 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 1 librados: init done 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 0x7fc6181d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc61809dec0 con 0x7fc61809e660 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc600005250 con 0x7fc618138cd0 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc5f8038e40 0x7fc5f803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(375..375 src has 1..375) ==== 6170+0+0 (secure 0 0 0) 0x7fc600049230 con 0x7fc618138cd0 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 client.74306.objecter ms_dispatch 0x7fc618000ba0 osd_map(375..375 src has 1..375) 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 3 client.74306.objecter handle_osd_map got epochs [375,375] > 0 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 3 client.74306.objecter handle_osd_map decoding full epoch 375 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 20 client.74306.objecter dump_active .. 0 homeless 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 0x7fc6181d0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient: get_auth_request con 0x7fc61809e660 auth_method 0 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61d01b640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc5f8038e40 0x7fc5f803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61d01b640 10 monclient: get_auth_request con 0x7fc5f8038e40 auth_method 0 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 10 monclient(hunting): handle_auth_done global_id 74201 payload 824 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6180a3040 con 0x7fc61809e660 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 client.74306.objecter ms_handle_connect 0x7fc61809e660 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61d01b640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc5f8038e40 0x7fc5f803b300 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7fc6080ae000 tx=0x7fc6080a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 client.74306.objecter resend_mon_ops 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 client.74306.objecter ms_handle_connect 0x7fc5f8038e40 2024-04-19T00:54:29.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc61c81a640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 0x7fc6181d0b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c380 tx=0x7fc61000c840 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc6100047f0 con 0x7fc61809e660 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.506+0000 7fc6157fa640 10 monclient: dump: 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.530 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6100072e0 con 0x7fc61809e660 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 msgr2=0x7fc6181d0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 0x7fc6181d0b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c380 tx=0x7fc61000c840 comp rx=0 tx=0).stop 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 msgr2=0x7fc6181d0b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:29.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.510+0000 7fc6157fa640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc61809e660 0x7fc6181d0b10 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c380 tx=0x7fc61000c840 comp rx=0 tx=0).stop 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61ea7e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61ea7e640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 0x7fc6181d1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61ea7e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc61806d140 con 0x7fc6181d1050 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 0x7fc6181d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient: get_auth_request con 0x7fc6181d1050 auth_method 0 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 10 monclient(hunting): handle_auth_done global_id 74210 payload 824 2024-04-19T00:54:29.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.514+0000 7fc61c81a640 1 -- 172.21.15.43:0/1322982305 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6181d1970 con 0x7fc6181d1050 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61c81a640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 0x7fc6181d1430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c380 tx=0x7fc610000bd0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 client.74306.objecter ms_handle_connect 0x7fc6181d1050 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 client.74306.objecter resend_mon_ops 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc610019cf0 con 0x7fc6181d1050 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 monclient: dump: 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.533 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9250 (secure 0 0 0) 0x7fc610019880 con 0x7fc6181d1050 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 msgr2=0x7fc6181d1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 0x7fc6181d1430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c380 tx=0x7fc610000bd0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 msgr2=0x7fc6181d1430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc6157fa640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc6181d1050 0x7fc6181d1430 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc61000c380 tx=0x7fc610000bd0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 49139, 2024-04-19T00:54:29.535 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.536 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:29.537 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:29.539 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:29.540 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:29.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.545 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.547 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:29.549 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.550 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:29.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.552 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:29.553 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:29.555 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:29.556 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:29.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:29.557 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:29.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:29.557 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:29.557 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 10 librados: watch_flush enter 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 10 librados: watch_flush exit 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 20 client.74306.objecter shutdown clearing up homeless session... 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 10 client.74306.objecter successfully canceled tick 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc5f8038e40 msgr2=0x7fc5f803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc5f8038e40 0x7fc5f803b300 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7fc6080ae000 tx=0x7fc6080a7070 comp rx=0 tx=0).stop 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 10 monclient: shutdown 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc618138cd0 msgr2=0x7fc61809bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 --2- 172.21.15.43:0/1322982305 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc618138cd0 0x7fc61809bfb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc60000b510 tx=0x7fc60000b9d0 comp rx=0 tx=0).stop 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61d01b640 1 -- 172.21.15.43:0/1322982305 reap_dead start 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61d01b640 1 -- 172.21.15.43:0/1322982305 reap_dead start 2024-04-19T00:54:29.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 shutdown_connections 2024-04-19T00:54:29.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 >> 172.21.15.43:0/1322982305 conn(0x7fc6180ab2f0 msgr2=0x7fc618130e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:29.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 shutdown_connections 2024-04-19T00:54:29.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 -- 172.21.15.43:0/1322982305 wait complete. 2024-04-19T00:54:29.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.518+0000 7fc61ea7e640 1 librados: shutdown 2024-04-19T00:54:29.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:29.560 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon i 2024-04-19T00:54:29.560 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient: get_monmap_and_config 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient: build_initial_monmap 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient: monmap: 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:54:29.678903+0000 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:54:29.678903+0000 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:29.676 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:54:29.677 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 Processor -- start 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 -- start start 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient: init 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient: _add_conns ranks=[10,1,8,4,3,0,9,7,2,6,5] 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 0x7fd334138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): picked mon.noname-f con 0x7fd334138a50 addr v2:172.21.15.161:3300/0 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fd3340a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 0x7fd334138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): picked mon.noname-e con 0x7fd334139370 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3340a0d40 con 0x7fd3340a5bb0 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd3340a1020 con 0x7fd334138a50 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd3340a0780 con 0x7fd334139370 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd33b88b640 10 monclient(hunting): authenticate will time out at 1539.806033s 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd339e28640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 msgr2=0x7fd334138070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd339e28640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 0x7fd334138070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd338e26640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7fd3340a5bb0 legacy=0x7fd334137960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd339627640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 msgr2=0x7fd334138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:29.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.674+0000 7fd339627640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 0x7fd334138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:54:29.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.878+0000 7fd339e28640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 msgr2=0x7fd334138070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.878+0000 7fd339e28640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 0x7fd334138070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.878+0000 7fd339627640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 msgr2=0x7fd334138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.878+0000 7fd338e26640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7fd3340a5bb0 legacy=0x7fd334137960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:29.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:29.878+0000 7fd339627640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 0x7fd334138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:54:30.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:30.282+0000 7fd339e28640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 msgr2=0x7fd334138070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:30.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:30.282+0000 7fd339e28640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 0x7fd334138070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:30.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:30.282+0000 7fd338e26640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7fd3340a5bb0 legacy=0x7fd334137960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:30.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:30.282+0000 7fd339627640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 msgr2=0x7fd334138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:30.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:30.282+0000 7fd339627640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 0x7fd334138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:54:31.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:31.086+0000 7fd339e28640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 msgr2=0x7fd334138070 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:54:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:31.086+0000 7fd339e28640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 0x7fd334138070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:31.086+0000 7fd338e26640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7fd3340a5bb0 legacy=0x7fd334137960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:54:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:31.086+0000 7fd339627640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 msgr2=0x7fd334138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:54:31.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:31.086+0000 7fd339627640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 0x7fd334138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:54:32.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): tick 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 monclient(hunting): continuing hunt 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 msgr2=0x7fd334138070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd334139370 0x7fd334138070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 msgr2=0x7fd334138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd334138a50 0x7fd334138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7fd3340a5bb0 legacy=0x7fd334137960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient: _add_conns ranks=[4,5,3,0,6,7,2,9] 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 0x7fd318005c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fd318003830 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:54:32.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd318006280 0x7fd318008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): picked mon.noname-h con 0x7fd318006280 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd318008e00 0x7fd31800b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): picked mon.noname-g con 0x7fd318008e00 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd3340a1020 con 0x7fd318008e00 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd3340a0780 con 0x7fd318003830 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fd31800ba10 con 0x7fd318006280 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd326ffd640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 0x7fd318005c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 0x7fd318005c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:57778/0 (socket says 172.21.15.43:57778) 2024-04-19T00:54:32.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 -- 172.21.15.43:0/3600671554 learned_addr learned my addr 172.21.15.43:0/3600671554 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): get_auth_request con 0x7fd318003830 auth_method 0 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd318006280 0x7fd318008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd318008e00 0x7fd31800b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): get_auth_request con 0x7fd318006280 auth_method 0 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): get_auth_request con 0x7fd318008e00 auth_method 0 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient(hunting): handle_auth_done global_id 64516 payload 274 2024-04-19T00:54:32.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 -- 172.21.15.43:0/3600671554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd318006280 msgr2=0x7fd318008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd318006280 0x7fd318008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 -- 172.21.15.43:0/3600671554 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd318008e00 msgr2=0x7fd31800b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd318008e00 0x7fd31800b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient: _finish_hunting 0 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 monclient: found mon.noname-c 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339e28640 1 -- 172.21.15.43:0/3600671554 reap_dead start 2024-04-19T00:54:32.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.678+0000 7fd339627640 1 -- 172.21.15.43:0/3600671554 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3340a0a60 con 0x7fd318003830 2024-04-19T00:54:32.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd339627640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 0x7fd318005c80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd328009df0 tx=0x7fd32801c870 comp rx=0 tx=0).ready entity=mon.4 client_cookie=10d3d8964511f1eb server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 1 -- 172.21.15.43:0/3600671554 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3280ab020 con 0x7fd318003830 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: dump: 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:32.686 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: _finish_auth 0 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: _check_auth_tickets 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 1 -- 172.21.15.43:0/3600671554 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd328005070 con 0x7fd318003830 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 5 monclient: authenticate success, global_id 64516 2024-04-19T00:54:32.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 10 monclient: get_monmap_and_config success 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 4 set_mon_vals no callback set 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 1 -- 172.21.15.43:0/3600671554 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd328005690 con 0x7fd318003830 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd327fff640 10 monclient: dump: 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:32.688 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:32.689 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 10 monclient: shutdown 2024-04-19T00:54:32.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 -- 172.21.15.43:0/3600671554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 msgr2=0x7fd318005c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 0x7fd318005c80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd328009df0 tx=0x7fd32801c870 comp rx=0 tx=0).stop 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 -- 172.21.15.43:0/3600671554 shutdown_connections 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 --2- 172.21.15.43:0/3600671554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd318003830 0x7fd318005c80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 -- 172.21.15.43:0/3600671554 >> 172.21.15.43:0/3600671554 conn(0x7fd3340a7cb0 msgr2=0x7fd3340a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 -- 172.21.15.43:0/3600671554 shutdown_connections 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.682+0000 7fd33b88b640 1 -- 172.21.15.43:0/3600671554 wait complete. 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: build_initial_monmap 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: monmap: 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:54:32.696 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 librados: starting msgr at 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 librados: starting objecter 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 Processor -- start 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 -- start start 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 librados: setting wanted keys 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 librados: calling monclient init 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: init 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: _reopen_session rank -1 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: _add_conns ranks=[2,0,5,8,6,3,7,1,4] 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd318003830 0x7fd334136d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): picked mon.noname-b con 0x7fd318003830 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3341d0dd0 0x7fd334137270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): picked mon.noname-a con 0x7fd3341d0dd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd334121800 0x7fd334121c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): picked mon.noname-h con 0x7fd334121800 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd3341382f0 con 0x7fd3341d0dd0 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd334138170 con 0x7fd318003830 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fd334138470 con 0x7fd334121800 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): _renew_subs 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient(hunting): authenticate will time out at 1542.817783s 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd318003830 0x7fd334136d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3341d0dd0 0x7fd334137270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd318003830 0x7fd334136d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:57698/0 (socket says 172.21.15.43:57698) 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 -- 172.21.15.43:0/3031914085 learned_addr learned my addr 172.21.15.43:0/3031914085 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): get_auth_request con 0x7fd318003830 auth_method 0 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): get_auth_request con 0x7fd3341d0dd0 auth_method 0 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339e28640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd334121800 0x7fd334121c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339e28640 10 monclient(hunting): get_auth_request con 0x7fd334121800 auth_method 0 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339e28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339e28640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339e28640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient(hunting): handle_auth_done global_id 74315 payload 824 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd334121800 msgr2=0x7fd334121c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd334121800 0x7fd334121c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3341d0dd0 msgr2=0x7fd334137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3341d0dd0 0x7fd334137270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient: _finish_hunting 0 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 monclient: found mon.noname-b 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:54:32.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd334124710 con 0x7fd318003830 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd338e26640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd3341d0dd0 0x7fd334137270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd339627640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd318003830 0x7fd334136d30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd328009d90 tx=0x7fd328019610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 client.?.objecter ms_handle_connect 0x7fd318003830 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3280ab070 con 0x7fd318003830 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: dump: 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:32.702 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: _finish_auth 0 2024-04-19T00:54:32.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3280059e0 con 0x7fd318003830 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd3280a68d0 con 0x7fd318003830 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 5 monclient: authenticate success, global_id 74315 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: _renew_subs 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd33409fc90 con 0x7fd318003830 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd3277fe640 4 set_mon_vals no callback set 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd325ffb640 10 monclient: dump: 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:32.704 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 client.74315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: _renew_subs 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd33409e830 con 0x7fd318003830 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.686+0000 7fd33b88b640 1 librados: init done 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd33b88b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd33b88b640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 0x7fd3340a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd33406d140 con 0x7fd3341221a0 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd3280ae020 con 0x7fd318003830 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd31803a1e0 0x7fd31803c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 0x7fd3340a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient: get_auth_request con 0x7fd3341221a0 auth_method 0 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(378..378 src has 1..378) ==== 6157+0+0 (secure 0 0 0) 0x7fd3280e32a0 con 0x7fd318003830 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 client.74315.objecter ms_dispatch 0x7fd334000ba0 osd_map(378..378 src has 1..378) 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 3 client.74315.objecter handle_osd_map got epochs [378,378] > 0 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 3 client.74315.objecter handle_osd_map decoding full epoch 378 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 20 client.74315.objecter dump_active .. 0 homeless 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd339e28640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd31803a1e0 0x7fd31803c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd339e28640 10 monclient: get_auth_request con 0x7fd31803a1e0 auth_method 0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 10 monclient(hunting): handle_auth_done global_id 74221 payload 824 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3341377b0 con 0x7fd3341221a0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 client.74315.objecter ms_handle_connect 0x7fd3341221a0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 client.74315.objecter resend_mon_ops 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd338e26640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 0x7fd3340a28f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd33409dbc0 tx=0x7fd330007950 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 client.74315.objecter ms_handle_connect 0x7fd31803a1e0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd339e28640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd31803a1e0 0x7fd31803c6a0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fd334139ac0 tx=0x7fd32c007760 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd330003d20 con 0x7fd3341221a0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 monclient: dump: 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:32.708 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:32.709 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd33001b070 con 0x7fd3341221a0 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 msgr2=0x7fd3340a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 0x7fd3340a28f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd33409dbc0 tx=0x7fd330007950 comp rx=0 tx=0).stop 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 msgr2=0x7fd3340a28f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:32.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.690+0000 7fd325ffb640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3341221a0 0x7fd3340a28f0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd33409dbc0 tx=0x7fd330007950 comp rx=0 tx=0).stop 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd33b88b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd33b88b640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 0x7fd3340a3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd33b88b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd334067760 con 0x7fd3340a2e30 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 0x7fd3340a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient: get_auth_request con 0x7fd3340a2e30 auth_method 0 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:54:32.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.698+0000 7fd338e26640 10 monclient(hunting): handle_auth_done global_id 74230 payload 824 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd338e26640 1 -- 172.21.15.43:0/3031914085 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd334122580 con 0x7fd3340a2e30 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd338e26640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 0x7fd3340a3240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd33001a000 tx=0x7fd330005b30 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 client.74315.objecter ms_handle_connect 0x7fd3340a2e30 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 client.74315.objecter resend_mon_ops 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd330016470 con 0x7fd3340a2e30 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 monclient: dump: 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.713 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9250 (secure 0 0 0) 0x7fd330005630 con 0x7fd3340a2e30 2024-04-19T00:54:32.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 msgr2=0x7fd3340a3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 0x7fd3340a3240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd33001a000 tx=0x7fd330005b30 comp rx=0 tx=0).stop 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 msgr2=0x7fd3340a3240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd325ffb640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd3340a2e30 0x7fd3340a3240 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd33001a000 tx=0x7fd330005b30 comp rx=0 tx=0).stop 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:54:32.715 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 78, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 52323, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:54:32.716 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:54:32.717 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:54:32.719 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:54:32.720 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:54:32.721 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.722 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:54:32.724 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.727 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.728 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:54:32.729 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.730 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.731 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.732 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:54:32.733 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:54:32.734 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.735 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:32.736 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:54:32.737 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 10 librados: watch_flush enter 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 10 librados: watch_flush exit 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 20 client.74315.objecter shutdown clearing up homeless session... 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 10 client.74315.objecter successfully canceled tick 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd31803a1e0 msgr2=0x7fd31803c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd31803a1e0 0x7fd31803c6a0 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7fd334139ac0 tx=0x7fd32c007760 comp rx=0 tx=0).stop 2024-04-19T00:54:32.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 10 monclient: shutdown 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd318003830 msgr2=0x7fd334136d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 --2- 172.21.15.43:0/3031914085 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd318003830 0x7fd334136d30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd328009d90 tx=0x7fd328019610 comp rx=0 tx=0).stop 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd339e28640 1 -- 172.21.15.43:0/3031914085 reap_dead start 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd339e28640 1 -- 172.21.15.43:0/3031914085 reap_dead start 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 shutdown_connections 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 >> 172.21.15.43:0/3031914085 conn(0x7fd3340a7cb0 msgr2=0x7fd3341c0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 shutdown_connections 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 -- 172.21.15.43:0/3031914085 wait complete. 2024-04-19T00:54:32.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:54:32.702+0000 7fd33b88b640 1 librados: shutdown 2024-04-19T00:54:32.741 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T00:54:32.741 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T00:54:59.181 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-04-19T00:54:59.183+0000 7fab1f8df640 -1 snap_mapper.add_oid found existing snaps mapped on 301:24e2ac99:test-rados-api-smithi161-28857-80::foo:2, removing 2024-04-19T00:54:59.181 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-04-19T00:54:59.183+0000 7f346bff6640 -1 snap_mapper.add_oid found existing snaps mapped on 301:24e2ac99:test-rados-api-smithi161-28857-80::foo:2, removing 2024-04-19T00:55:30.218 INFO:tasks.workunit.client.0.smithi161.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18095 ms) 2024-04-19T00:55:30.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-19T00:55:30.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (26303 ms) 2024-04-19T00:55:30.218 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (17716 ms) 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (2836 ms) 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (27411 ms) 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3040 ms) 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (3637 ms) 2024-04-19T00:55:30.219 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39963 ms) 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17003 ms) 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4041 ms) 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (2964 ms) 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.220 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (31091 ms) 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (30251 ms) 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (8989 ms) 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4028 ms) 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: waiting for scrubs... 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: done waiting 2024-04-19T00:55:30.221 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25123 ms) 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9979 ms) 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9004 ms) 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8980 ms) 2024-04-19T00:55:30.222 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-04-19T00:56:02.741 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-04-19T00:56:02.742 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-04-19T00:56:02.742 INFO:tasks.ceph.mon.e:Restarting daemon 2024-04-19T00:56:02.742 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-04-19T00:56:02.746 INFO:tasks.ceph.mon.e:Started 2024-04-19T00:56:02.746 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-04-19T00:56:02.746 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-04-19T00:56:02.746 INFO:tasks.ceph.mon.f:Restarting daemon 2024-04-19T00:56:02.746 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-04-19T00:56:02.749 INFO:tasks.ceph.mon.f:Started 2024-04-19T00:56:02.749 INFO:tasks.mon_thrash:Sending STOP to mon g 2024-04-19T00:56:02.749 INFO:tasks.ceph.mon.g:Sent signal 19 2024-04-19T00:56:02.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:56:17.750 INFO:tasks.mon_thrash:Sending CONT to mon g 2024-04-19T00:56:17.751 INFO:tasks.ceph.mon.g:Sent signal 18 2024-04-19T00:56:17.751 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:56:17.751 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:56:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.893+0000 7f452d566640 10 monclient: get_monmap_and_config 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.893+0000 7f452d566640 10 monclient: build_initial_monmap 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.893+0000 7f452d566640 10 monclient: monmap: 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:17.896996+0000 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:17.896996+0000 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:17.895 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.893+0000 7f452d566640 1 Processor -- start 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 1 -- start start 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient: init 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient: _add_conns ranks=[0,5,4,7,1,3,2,8,9,10,6] 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4528135ca0 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4528073000 0x7f452813ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): picked mon.noname-h con 0x7f4528073000 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:17.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f452813b060 0x7f452813d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): picked mon.noname-c con 0x7f452813b060 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4528135800 con 0x7f4528135ca0 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4528071210 con 0x7f452813b060 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4528071d80 con 0x7f4528073000 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f452d566640 10 monclient(hunting): authenticate will time out at 1648.024750s 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4526ffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f4528135ca0 0x7f4528136080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47698/0 (socket says 172.21.15.43:47698) 2024-04-19T00:56:17.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4526ffd640 1 -- 172.21.15.43:0/3509126921 learned_addr learned my addr 172.21.15.43:0/3509126921 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f452813b060 0x7f452813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): get_auth_request con 0x7f452813b060 auth_method 0 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4525ffb640 1 -- 172.21.15.43:0/3509126921 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3804789367 0 0) 0x7f45180014a0 con 0x7f4528135ca0 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4525ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4525ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4525ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:17.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4525ffb640 10 monclient(hunting): my global_id is 64728 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f4525ffb640 1 -- 172.21.15.43:0/3509126921 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4514003530 con 0x7f4528135ca0 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45267fc640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4528073000 0x7f452813ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45267fc640 10 monclient(hunting): get_auth_request con 0x7f4528073000 auth_method 0 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45267fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient(hunting): handle_auth_done global_id 64525 payload 274 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 1 -- 172.21.15.43:0/3509126921 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4528073000 msgr2=0x7f452813ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4528073000 0x7f452813ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 1 -- 172.21.15.43:0/3509126921 >> v1:172.21.15.43:6789/0 conn(0x7f4528135ca0 legacy=0x7f4528136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 1 monclient: found mon.noname-c 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:17.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.897+0000 7f45277fe640 1 -- 172.21.15.43:0/3509126921 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4528072060 con 0x7f452813b060 2024-04-19T00:56:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f452813b060 0x7f452813d450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45200097f0 tx=0x7f452001c3a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3d014dcecd322cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 1 -- 172.21.15.43:0/3509126921 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45200b0070 con 0x7f452813b060 2024-04-19T00:56:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:17.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: dump: 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:17.908 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 5 monclient: authenticate success, global_id 64525 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 1 -- 172.21.15.43:0/3509126921 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45200aa810 con 0x7f452813b060 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 1 -- 172.21.15.43:0/3509126921 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45200aae30 con 0x7f452813b060 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 4 set_mon_vals no callback set 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:17.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4525ffb640 10 monclient: dump: 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:17.910 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: shutdown 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3509126921 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f452813b060 msgr2=0x7f452813d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f452813b060 0x7f452813d450 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45200097f0 tx=0x7f452001c3a0 comp rx=0 tx=0).stop 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3509126921 shutdown_connections 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f452813b060 0x7f452813d450 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:17.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 --2- 172.21.15.43:0/3509126921 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4528073000 0x7f452813ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3509126921 >> 172.21.15.43:0/3509126921 conn(0x7f45280a7ef0 msgr2=0x7f45280a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3509126921 shutdown_connections 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3509126921 wait complete. 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: build_initial_monmap 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: monmap: 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:17.912 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 librados: starting msgr at 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 librados: starting objecter 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 Processor -- start 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- start start 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 librados: setting wanted keys 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 librados: calling monclient init 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: init 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:17.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: _add_conns ranks=[2,6,8,1,7,5,3,4,0] 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 0x7f4528137350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): picked mon.noname-b con 0x7f4528073000 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4528135ca0 0x7f4528137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): picked mon.noname-d con 0x7f4528135ca0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f452813b060 0x7f4528137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): picked mon.noname-e con 0x7f452813b060 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4528070350 con 0x7f4528073000 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f45280701d0 con 0x7f4528135ca0 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f45280704d0 con 0x7f452813b060 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:17.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient(hunting): authenticate will time out at 1648.030612s 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 0x7f4528137350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f452813b060 0x7f4528137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 0x7f4528137350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35744/0 (socket says 172.21.15.43:35744) 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 1 -- 172.21.15.43:0/3073886786 learned_addr learned my addr 172.21.15.43:0/3073886786 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): get_auth_request con 0x7f4528073000 auth_method 0 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): get_auth_request con 0x7f452813b060 auth_method 0 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4528135ca0 0x7f4528137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): get_auth_request con 0x7f4528135ca0 auth_method 0 2024-04-19T00:56:17.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient(hunting): handle_auth_done global_id 114110 payload 824 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 -- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4528135ca0 msgr2=0x7f4528137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:17.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4528135ca0 0x7f4528137890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 -- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 msgr2=0x7f4528137350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 0x7f4528137350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 monclient: found mon.noname-e 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 -- 172.21.15.43:0/3073886786 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45281399e0 con 0x7f452813b060 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f4526ffd640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 0x7f4528137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45267fc640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4528135ca0 0x7f4528137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45277fe640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f452813b060 0x7f4528137dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45280767f0 tx=0x7f4520019140 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 client.?.objecter ms_handle_connect 0x7f452813b060 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45200b0050 con 0x7f452813b060 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:17.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: dump: 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:17.918 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45200b1040 con 0x7f452813b060 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 5 monclient: authenticate success, global_id 114110 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: _renew_subs 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45280713e0 con 0x7f452813b060 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f45200044b0 con 0x7f452813b060 2024-04-19T00:56:17.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f45257fa640 4 set_mon_vals no callback set 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 client.114110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f451f7fe640 10 monclient: dump: 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:17.920 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: _renew_subs 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:17.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f45281392a0 con 0x7f452813b060 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.901+0000 7f452d566640 1 librados: init done 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4520004040 con 0x7f452813b060 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f452d566640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f452d566640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f452d566640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45280008e0 con 0x7f452813b060 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4508038e10 0x7f450803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(439..439 src has 1..439) ==== 6170+0+0 (secure 0 0 0) 0x7f45200bc070 con 0x7f452813b060 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 10 client.114110.objecter ms_dispatch 0x7f4528000ba0 osd_map(439..439 src has 1..439) 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 3 client.114110.objecter handle_osd_map got epochs [439,439] > 0 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 3 client.114110.objecter handle_osd_map decoding full epoch 439 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 20 client.114110.objecter dump_active .. 0 homeless 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f4526ffd640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4508038e10 0x7f450803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:17.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f4526ffd640 10 monclient: get_auth_request con 0x7f4508038e10 auth_method 0 2024-04-19T00:56:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f4526ffd640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4508038e10 0x7f450803b2d0 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7f4518005c10 tx=0x7f4518009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.905+0000 7f451f7fe640 10 client.114110.objecter ms_handle_connect 0x7f4508038e10 2024-04-19T00:56:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.909+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f45281399e0 con 0x7f452813b060 2024-04-19T00:56:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.909+0000 7f451f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:17.909+0000 7f451f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:18.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.133+0000 7f452d566640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:56:18.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.133+0000 7f452d566640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:56:18.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.133+0000 7f452d566640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:18.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.133+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f452809c260 con 0x7f452813b060 2024-04-19T00:56:18.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f451f7fe640 1 -- 172.21.15.43:0/3073886786 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f45200160b0 con 0x7f452813b060 2024-04-19T00:56:18.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f451f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:56:18.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f451f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:18.135 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":84,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:56:18.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 10 librados: watch_flush enter 2024-04-19T00:56:18.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 10 librados: watch_flush exit 2024-04-19T00:56:18.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 20 client.114110.objecter shutdown clearing up homeless session... 2024-04-19T00:56:18.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 10 client.114110.objecter successfully canceled tick 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4508038e10 msgr2=0x7f450803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4508038e10 0x7f450803b2d0 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7f4518005c10 tx=0x7f4518009040 comp rx=0 tx=0).stop 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 10 monclient: shutdown 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f452813b060 msgr2=0x7f4528137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f452813b060 0x7f4528137dd0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f45280767f0 tx=0x7f4520019140 comp rx=0 tx=0).stop 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 shutdown_connections 2024-04-19T00:56:18.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4508038e10 0x7f450803b2d0 unknown :-1 s=CLOSED pgs=3519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f452813b060 0x7f4528137dd0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4528135ca0 0x7f4528137890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 --2- 172.21.15.43:0/3073886786 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4528073000 0x7f4528137350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 >> 172.21.15.43:0/3073886786 conn(0x7f45280a7ef0 msgr2=0x7f45280a3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 shutdown_connections 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 -- 172.21.15.43:0/3073886786 wait complete. 2024-04-19T00:56:18.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.137+0000 7f452d566640 1 librados: shutdown 2024-04-19T00:56:18.155 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:56:18.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:56:18.155 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient: get_monmap_and_config 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient: monmap: 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:18.274664+0000 2024-04-19T00:56:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:18.274664+0000 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:18.272 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 Processor -- start 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 -- start start 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient: init 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient: _add_conns ranks=[4,1,2,8,0,5,6,10,3,7,9] 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f266c09e4c0 0x7f266c136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): picked mon.noname-c con 0x7f266c09e4c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f266c09e8a0 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 0x7f266c0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): picked mon.noname-b con 0x7f266c09eca0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f266c0a1e20 con 0x7f266c09e8a0 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f266c0a1c50 con 0x7f266c09eca0 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f266c06c760 con 0x7f266c09e4c0 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f2671ab4640 10 monclient(hunting): authenticate will time out at 1648.401828s 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 0x7f266c0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f266c09e4c0 0x7f266c136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 0x7f266c0a4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35754/0 (socket says 172.21.15.43:35754) 2024-04-19T00:56:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 -- 172.21.15.43:0/844004053 learned_addr learned my addr 172.21.15.43:0/844004053 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): get_auth_request con 0x7f266c09eca0 auth_method 0 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): get_auth_request con 0x7f266c09e4c0 auth_method 0 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient(hunting): handle_auth_done global_id 74333 payload 274 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 -- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f266c09e4c0 msgr2=0x7f266c136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 --2- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f266c09e4c0 0x7f266c136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 -- 172.21.15.43:0/844004053 >> v1:172.21.15.161:6789/0 conn(0x7f266c09e8a0 legacy=0x7f266c0a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 monclient: found mon.noname-b 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266bfff640 1 -- 172.21.15.43:0/844004053 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266c0a5510 con 0x7f266c09eca0 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.273+0000 7f266b7fe640 1 --2- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f266c09e4c0 0x7f266c136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266bfff640 1 --2- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 0x7f266c0a4050 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2664009e80 tx=0x7f266401c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19524ce56f240677 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 1 -- 172.21.15.43:0/844004053 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26640090d0 con 0x7f266c09eca0 2024-04-19T00:56:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: dump: 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 5 monclient: authenticate success, global_id 74333 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 1 -- 172.21.15.43:0/844004053 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26640a59f0 con 0x7f266c09eca0 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 4 set_mon_vals no callback set 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 1 -- 172.21.15.43:0/844004053 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f26640a5be0 con 0x7f266c09eca0 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266a7fc640 10 monclient: dump: 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: shutdown 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 msgr2=0x7f266c0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 --2- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 0x7f266c0a4050 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2664009e80 tx=0x7f266401c9a0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- 172.21.15.43:0/844004053 shutdown_connections 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 --2- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f266c09eca0 0x7f266c0a4050 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 --2- 172.21.15.43:0/844004053 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f266c09e4c0 0x7f266c136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- 172.21.15.43:0/844004053 >> 172.21.15.43:0/844004053 conn(0x7f266c0a85c0 msgr2=0x7f266c0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- 172.21.15.43:0/844004053 shutdown_connections 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- 172.21.15.43:0/844004053 wait complete. 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: monmap: 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 librados: starting msgr at 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 librados: starting objecter 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 Processor -- start 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- start start 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 librados: setting wanted keys 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 librados: calling monclient init 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: init 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient: _add_conns ranks=[7,3,5,8,2,1,6,4,0] 2024-04-19T00:56:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f266c09e4c0 0x7f266c129890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): picked mon.noname-i con 0x7f266c09e4c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 0x7f266c129dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): picked mon.noname-g con 0x7f266c09e8a0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f266c09eca0 0x7f266c070520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): picked mon.noname-h con 0x7f266c09eca0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f266c1416f0 con 0x7f266c09e8a0 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f266c0a47a0 con 0x7f266c09eca0 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f266c141870 con 0x7f266c09e4c0 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f2671ab4640 10 monclient(hunting): authenticate will time out at 1648.407291s 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 0x7f266c129dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 0x7f266c129dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:57048/0 (socket says 172.21.15.43:57048) 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 1 -- 172.21.15.43:0/3388964810 learned_addr learned my addr 172.21.15.43:0/3388964810 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): get_auth_request con 0x7f266c09e8a0 auth_method 0 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f266c09e4c0 0x7f266c129890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 10 monclient(hunting): get_auth_request con 0x7f266c09e4c0 auth_method 0 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266bfff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f266c09eca0 0x7f266c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266bfff640 10 monclient(hunting): get_auth_request con 0x7f266c09eca0 auth_method 0 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.277+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 10 monclient(hunting): handle_auth_done global_id 19389 payload 824 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f266c09e4c0 msgr2=0x7f266c129890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f266c09e4c0 0x7f266c129890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f266c09eca0 msgr2=0x7f266c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f266c09eca0 0x7f266c070520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 monclient: found mon.noname-g 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266c072300 con 0x7f266c09e8a0 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f266c09e4c0 0x7f266c129890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266affd640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 0x7f266c129dd0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f266000c410 tx=0x7f266000c8d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 client.?.objecter ms_handle_connect 0x7f266c09e8a0 2024-04-19T00:56:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2660011a00 con 0x7f266c09e8a0 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: dump: 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660017070 con 0x7f266c09e8a0 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 5 monclient: authenticate success, global_id 19389 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2660011c60 con 0x7f266c09e8a0 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 monclient: _renew_subs 2024-04-19T00:56:18.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f266c132350 con 0x7f266c09e8a0 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2669ffb640 4 set_mon_vals no callback set 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: dump: 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.299 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.300 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 client.19389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 monclient: _renew_subs 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f266c131ab0 con 0x7f266c09e8a0 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 1 librados: init done 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 0x7f266c070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f266c06d140 con 0x7f266c0724f0 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 0x7f266c070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient: get_auth_request con 0x7f266c0724f0 auth_method 0 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 10 monclient(hunting): handle_auth_done global_id 64737 payload 824 2024-04-19T00:56:18.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f266c06bda0 con 0x7f266c0724f0 2024-04-19T00:56:18.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266b7fe640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 0x7f266c070a60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f265c011cf0 tx=0x7f265c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 client.19389.objecter ms_handle_connect 0x7f266c0724f0 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 client.19389.objecter resend_mon_ops 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2660011590 con 0x7f266c09e8a0 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2658038e40 0x7f265803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f265c019070 con 0x7f266c0724f0 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: dump: 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.305 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.306 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.306 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.306 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.306 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.306 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.306 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(439..439 src has 1..439) ==== 6170+0+0 (secure 0 0 0) 0x7f2660048f90 con 0x7f266c09e8a0 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 client.19389.objecter ms_dispatch 0x7f266c000ba0 osd_map(439..439 src has 1..439) 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 3 client.19389.objecter handle_osd_map got epochs [439,439] > 0 2024-04-19T00:56:18.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 3 client.19389.objecter handle_osd_map decoding full epoch 439 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 20 client.19389.objecter dump_active .. 0 homeless 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f265c0044a0 con 0x7f266c0724f0 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 msgr2=0x7f266c070a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 0x7f266c070a60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f265c011cf0 tx=0x7f265c00f610 comp rx=0 tx=0).stop 2024-04-19T00:56:18.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 msgr2=0x7f266c070a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f2653fff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c0724f0 0x7f266c070a60 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f265c011cf0 tx=0x7f265c00f610 comp rx=0 tx=0).stop 2024-04-19T00:56:18.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266bfff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2658038e40 0x7f265803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.281+0000 7f266bfff640 10 monclient: get_auth_request con 0x7f2658038e40 auth_method 0 2024-04-19T00:56:18.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.285+0000 7f266bfff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2658038e40 0x7f265803b300 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f26640ae000 tx=0x7f26640a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.285+0000 7f2653fff640 10 client.19389.objecter ms_handle_connect 0x7f2658038e40 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 0x7f266c0713b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f266c0718f0 con 0x7f266c070fa0 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 0x7f266c0713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient: get_auth_request con 0x7f266c070fa0 auth_method 0 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 10 monclient(hunting): handle_auth_done global_id 64746 payload 824 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 1 -- 172.21.15.43:0/3388964810 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f266c072070 con 0x7f266c070fa0 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266b7fe640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 0x7f266c0713b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f265c011cf0 tx=0x7f265c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 client.19389.objecter ms_handle_connect 0x7f266c070fa0 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 client.19389.objecter resend_mon_ops 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f265c019040 con 0x7f266c070fa0 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 monclient: dump: 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.314 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f265c01e660 con 0x7f266c070fa0 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 msgr2=0x7f266c0713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 0x7f266c0713b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f265c011cf0 tx=0x7f265c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 msgr2=0x7f266c0713b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2653fff640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f266c070fa0 0x7f266c0713b0 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f265c011cf0 tx=0x7f265c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.316 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 304561, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:18.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.333 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.335 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:18.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.338 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:18.339 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 10 librados: watch_flush enter 2024-04-19T00:56:18.341 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 10 librados: watch_flush exit 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 20 client.19389.objecter shutdown clearing up homeless session... 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 10 client.19389.objecter successfully canceled tick 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2658038e40 msgr2=0x7f265803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2658038e40 0x7f265803b300 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7f26640ae000 tx=0x7f26640a7070 comp rx=0 tx=0).stop 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 10 monclient: shutdown 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 msgr2=0x7f266c129dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 0x7f266c129dd0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f266000c410 tx=0x7f266000c8d0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f266bfff640 1 -- 172.21.15.43:0/3388964810 reap_dead start 2024-04-19T00:56:18.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 shutdown_connections 2024-04-19T00:56:18.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 --2- 172.21.15.43:0/3388964810 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f266c09e8a0 0x7f266c129dd0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 >> 172.21.15.43:0/3388964810 conn(0x7f266c0a85c0 msgr2=0x7f266c12fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 shutdown_connections 2024-04-19T00:56:18.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 -- 172.21.15.43:0/3388964810 wait complete. 2024-04-19T00:56:18.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.289+0000 7f2671ab4640 1 librados: shutdown 2024-04-19T00:56:18.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:18.344 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:56:18.344 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.465+0000 7f8ba5b7a640 10 monclient: get_monmap_and_config 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.465+0000 7f8ba5b7a640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.465+0000 7f8ba5b7a640 10 monclient: monmap: 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:18.469217+0000 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:18.469217+0000 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:18.466 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:18.467 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.465+0000 7f8ba5b7a640 1 Processor -- start 2024-04-19T00:56:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.465+0000 7f8ba5b7a640 1 -- start start 2024-04-19T00:56:18.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.465+0000 7f8ba5b7a640 10 monclient: init 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient: _add_conns ranks=[10,1,8,4,7,0,5,2,3,9,6] 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ba0138cd0 0x7f8ba01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): picked mon.noname-f con 0x7f8ba0138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f8ba00a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 0x7f8ba0135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): picked mon.noname-e con 0x7f8ba01395f0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba00a3dc0 con 0x7f8ba00a5b80 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba00a4410 con 0x7f8ba0138cd0 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8ba00a40a0 con 0x7f8ba01395f0 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient(hunting): authenticate will time out at 1648.596357s 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 0x7f8ba0135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 0x7f8ba0135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56052/0 (socket says 172.21.15.43:56052) 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 -- 172.21.15.43:0/903082752 learned_addr learned my addr 172.21.15.43:0/903082752 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): get_auth_request con 0x7f8ba01395f0 auth_method 0 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9f7fe640 1 --2- 172.21.15.43:0/903082752 >> v2:172.21.15.161:3300/0 conn(0x7f8ba0138cd0 0x7f8ba01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9f7fe640 10 monclient(hunting): get_auth_request con 0x7f8ba0138cd0 auth_method 0 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_done global_id 114119 payload 274 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 -- 172.21.15.43:0/903082752 >> v2:172.21.15.161:3300/0 conn(0x7f8ba0138cd0 msgr2=0x7f8ba01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 --2- 172.21.15.43:0/903082752 >> v2:172.21.15.161:3300/0 conn(0x7f8ba0138cd0 0x7f8ba01390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 -- 172.21.15.43:0/903082752 >> v1:172.21.15.161:6789/0 conn(0x7f8ba00a5b80 legacy=0x7f8ba01358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 monclient: found mon.noname-e 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 -- 172.21.15.43:0/903082752 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba00a2ae0 con 0x7f8ba01395f0 2024-04-19T00:56:18.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9ffff640 1 --2- 172.21.15.43:0/903082752 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 0x7f8ba0135fc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00a7450 tx=0x7f8b9801c8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=9d22c52036ef099f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 1 -- 172.21.15.43:0/903082752 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b980090d0 con 0x7f8ba01395f0 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: dump: 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 1 -- 172.21.15.43:0/903082752 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b980a5b80 con 0x7f8ba01395f0 2024-04-19T00:56:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 1 -- 172.21.15.43:0/903082752 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b980a5d70 con 0x7f8ba01395f0 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 5 monclient: authenticate success, global_id 114119 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 4 set_mon_vals no callback set 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8b9e7fc640 10 monclient: dump: 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 10 monclient: shutdown 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/903082752 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 msgr2=0x7f8ba0135fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/903082752 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 0x7f8ba0135fc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8ba00a7450 tx=0x7f8b9801c8b0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/903082752 shutdown_connections 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/903082752 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8ba01395f0 0x7f8ba0135fc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/903082752 >> v2:172.21.15.161:3300/0 conn(0x7f8ba0138cd0 0x7f8ba01390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/903082752 >> 172.21.15.43:0/903082752 conn(0x7f8ba00ab2f0 msgr2=0x7f8ba00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/903082752 shutdown_connections 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.469+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/903082752 wait complete. 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: monmap: 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 librados: starting msgr at 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 librados: starting objecter 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 Processor -- start 2024-04-19T00:56:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 -- start start 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 librados: setting wanted keys 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 librados: calling monclient init 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: init 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: _add_conns ranks=[6,4,0,8,7,5,3,1,2] 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ba00a5b80 0x7f8ba00a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): picked mon.noname-d con 0x7f8ba00a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba0138cd0 0x7f8ba012e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba0138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ba01395f0 0x7f8ba012ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba01395f0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba0136500 con 0x7f8ba01395f0 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8ba00a20a0 con 0x7f8ba0138cd0 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8ba0136680 con 0x7f8ba00a5b80 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient(hunting): authenticate will time out at 1648.602479s 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ba00a5b80 0x7f8ba00a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ba01395f0 0x7f8ba012ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ba01395f0 0x7f8ba012ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40384/0 (socket says 172.21.15.43:40384) 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 -- 172.21.15.43:0/2181222083 learned_addr learned my addr 172.21.15.43:0/2181222083 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): get_auth_request con 0x7f8ba01395f0 auth_method 0 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ba00a5b80 0x7f8ba00a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40380/0 (socket says 172.21.15.43:40380) 2024-04-19T00:56:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): get_auth_request con 0x7f8ba00a5b80 auth_method 0 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9effd640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba0138cd0 0x7f8ba012e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9effd640 10 monclient(hunting): get_auth_request con 0x7f8ba0138cd0 auth_method 0 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient(hunting): handle_auth_done global_id 64755 payload 824 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ba00a5b80 msgr2=0x7f8ba00a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8ba00a5b80 0x7f8ba00a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba0138cd0 msgr2=0x7f8ba012e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba0138cd0 0x7f8ba012e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 monclient: found mon.noname-a 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba00a0250 con 0x7f8ba01395f0 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9effd640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8ba0138cd0 0x7f8ba012e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9ffff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ba01395f0 0x7f8ba012ee10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009e20 tx=0x7f8b98019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 client.?.objecter ms_handle_connect 0x7f8ba01395f0 2024-04-19T00:56:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b980bc020 con 0x7f8ba01395f0 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: dump: 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 5 monclient: authenticate success, global_id 64755 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b98003bb0 con 0x7f8ba01395f0 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: _renew_subs 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b98003d60 con 0x7f8ba01395f0 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba009eda0 con 0x7f8ba01395f0 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b9dffb640 4 set_mon_vals no callback set 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8b87fff640 10 monclient: dump: 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 client.64755.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: _renew_subs 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba009ff70 con 0x7f8ba01395f0 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.473+0000 7f8ba5b7a640 1 librados: init done 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8b980bf020 con 0x7f8ba01395f0 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8ba5b7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 0x7f8ba009c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba006d140 con 0x7f8ba012f350 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 0x7f8ba009c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b8c038e10 0x7f8b8c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient: get_auth_request con 0x7f8ba012f350 auth_method 0 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(439..439 src has 1..439) ==== 6170+0+0 (secure 0 0 0) 0x7f8b980b3040 con 0x7f8ba01395f0 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 client.64755.objecter ms_dispatch 0x7f8ba0000ba0 osd_map(439..439 src has 1..439) 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 3 client.64755.objecter handle_osd_map got epochs [439,439] > 0 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 3 client.64755.objecter handle_osd_map decoding full epoch 439 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 20 client.64755.objecter dump_active .. 0 homeless 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_done global_id 74342 payload 824 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9effd640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b8c038e10 0x7f8b8c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ba00a3590 con 0x7f8ba012f350 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9effd640 10 monclient: get_auth_request con 0x7f8b8c038e10 auth_method 0 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 client.64755.objecter ms_handle_connect 0x7f8ba012f350 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 client.64755.objecter resend_mon_ops 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9f7fe640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 0x7f8ba009c570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b90010780 tx=0x7f8b90010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b90018020 con 0x7f8ba012f350 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 monclient: dump: 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8b90004240 con 0x7f8ba012f350 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 msgr2=0x7f8ba009c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 0x7f8ba009c570 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b90010780 tx=0x7f8b90010c40 comp rx=0 tx=0).stop 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 msgr2=0x7f8ba009c570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba012f350 0x7f8ba009c570 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b90010780 tx=0x7f8b90010c40 comp rx=0 tx=0).stop 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b87fff640 10 client.64755.objecter ms_handle_connect 0x7f8b8c038e10 2024-04-19T00:56:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.477+0000 7f8b9effd640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b8c038e10 0x7f8b8c03b2d0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f8ba009dcd0 tx=0x7f8b94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.481+0000 7f8ba5b7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.481+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 0x7f8ba009cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.481+0000 7f8ba5b7a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.481+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba0068240 con 0x7f8ba009cab0 2024-04-19T00:56:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 0x7f8ba009cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient: get_auth_request con 0x7f8ba009cab0 auth_method 0 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 10 monclient(hunting): handle_auth_done global_id 74351 payload 824 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 1 -- 172.21.15.43:0/2181222083 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ba012f730 con 0x7f8ba009cab0 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 client.64755.objecter ms_handle_connect 0x7f8ba009cab0 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 client.64755.objecter resend_mon_ops 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9f7fe640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 0x7f8ba009cec0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8b90019000 tx=0x7f8b900046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8b90018040 con 0x7f8ba009cab0 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 monclient: dump: 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f8b90020e50 con 0x7f8ba009cab0 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 msgr2=0x7f8ba009cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 0x7f8ba009cec0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8b90019000 tx=0x7f8b900046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 msgr2=0x7f8ba009cec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b87fff640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8ba009cab0 0x7f8ba009cec0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8b90019000 tx=0x7f8b900046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 158106, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:18.525 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 10 librados: watch_flush enter 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 10 librados: watch_flush exit 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 20 client.64755.objecter shutdown clearing up homeless session... 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 10 client.64755.objecter successfully canceled tick 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b8c038e10 msgr2=0x7f8b8c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8b8c038e10 0x7f8b8c03b2d0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f8ba009dcd0 tx=0x7f8b94009040 comp rx=0 tx=0).stop 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 10 monclient: shutdown 2024-04-19T00:56:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ba01395f0 msgr2=0x7f8ba012ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 --2- 172.21.15.43:0/2181222083 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8ba01395f0 0x7f8ba012ee10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009e20 tx=0x7f8b98019650 comp rx=0 tx=0).stop 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9ffff640 1 -- 172.21.15.43:0/2181222083 reap_dead start 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8b9ffff640 1 -- 172.21.15.43:0/2181222083 reap_dead start 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 shutdown_connections 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 >> 172.21.15.43:0/2181222083 conn(0x7f8ba00ab2f0 msgr2=0x7f8ba0131da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 shutdown_connections 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 -- 172.21.15.43:0/2181222083 wait complete. 2024-04-19T00:56:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.485+0000 7f8ba5b7a640 1 librados: shutdown 2024-04-19T00:56:18.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:18.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:56:18.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient: get_monmap_and_config 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient: monmap: 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:18.651236+0000 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:18.651236+0000 2024-04-19T00:56:18.648 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:18.649 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 Processor -- start 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 -- start start 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient: init 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient: _add_conns ranks=[8,6,5,2,0,10,1,4,7,3,9] 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 0x7fcdac138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): picked mon.noname-e con 0x7fcdac138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): picked mon.noname-d con 0x7fcdac0a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcdac137960 0x7fcdac137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): picked mon.noname-h con 0x7fcdac137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fcdac0a1020 con 0x7fcdac137960 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fcdac1398b0 con 0x7fcdac0a5bb0 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcdac0a0780 con 0x7fcdac138a50 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb2d5f640 10 monclient(hunting): authenticate will time out at 1648.778504s 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 0x7fcdac138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 0x7fcdac138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56060/0 (socket says 172.21.15.43:56060) 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 -- 172.21.15.43:0/3473914857 learned_addr learned my addr 172.21.15.43:0/3473914857 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): get_auth_request con 0x7fcdac138a50 auth_method 0 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40396/0 (socket says 172.21.15.43:40396) 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): get_auth_request con 0x7fcdac0a5bb0 auth_method 0 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb12fc640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcdac137960 0x7fcdac137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_done global_id 114128 payload 274 2024-04-19T00:56:18.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 -- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 msgr2=0x7fcdac139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 -- 172.21.15.43:0/3473914857 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcdac137960 msgr2=0x7fcdac137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcdac137960 0x7fcdac137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 monclient: found mon.noname-e 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb0afb640 1 -- 172.21.15.43:0/3473914857 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac0a0d40 con 0x7fcdac138a50 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb12fc640 10 monclient: get_auth_request con 0x7fcdac137960 auth_method 0 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdb12fc640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcdac137960 0x7fcdac137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.649+0000 7fcdabfff640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 0x7fcdac138e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098a0 tx=0x7fcd9c01c410 comp rx=0 tx=0).ready entity=mon.8 client_cookie=5a8866f3f81e01fe server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 1 -- 172.21.15.43:0/3473914857 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd9c0b0070 con 0x7fcdac138a50 2024-04-19T00:56:18.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: dump: 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.658 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 1 -- 172.21.15.43:0/3473914857 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd9c0044a0 con 0x7fcdac138a50 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 5 monclient: authenticate success, global_id 114128 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 1 -- 172.21.15.43:0/3473914857 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd9c003a90 con 0x7fcdac138a50 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdab7fe640 10 monclient: dump: 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.660 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 4 set_mon_vals no callback set 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: shutdown 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 msgr2=0x7fcdac138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 0x7fcdac138e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098a0 tx=0x7fcd9c01c410 comp rx=0 tx=0).stop 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3473914857 shutdown_connections 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fcdac137960 0x7fcdac137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3473914857 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac138a50 0x7fcdac138e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3473914857 >> 172.21.15.43:0/3473914857 conn(0x7fcdac0a7cb0 msgr2=0x7fcdac0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3473914857 shutdown_connections 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3473914857 wait complete. 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: monmap: 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.662 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 librados: starting msgr at 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 librados: starting objecter 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 Processor -- start 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- start start 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 librados: setting wanted keys 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 librados: calling monclient init 2024-04-19T00:56:18.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: init 2024-04-19T00:56:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient: _add_conns ranks=[7,8,4,6,2,1,3,5,0] 2024-04-19T00:56:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac1d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): picked mon.noname-i con 0x7fcdac0a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac137960 0x7fcdac1369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): picked mon.noname-e con 0x7fcdac137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac138a50 0x7fcdac13c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): picked mon.noname-c con 0x7fcdac138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fcdac0a2100 con 0x7fcdac138a50 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fcdac0a1f80 con 0x7fcdac0a5bb0 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fcdac142810 con 0x7fcdac137960 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb2d5f640 10 monclient(hunting): authenticate will time out at 1648.783579s 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac137960 0x7fcdac1369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb12fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac138a50 0x7fcdac13c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac137960 0x7fcdac1369c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56074/0 (socket says 172.21.15.43:56074) 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 1 -- 172.21.15.43:0/3151470803 learned_addr learned my addr 172.21.15.43:0/3151470803 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 10 monclient(hunting): get_auth_request con 0x7fcdac137960 auth_method 0 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdabfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb12fc640 10 monclient(hunting): get_auth_request con 0x7fcdac138a50 auth_method 0 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac1d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb12fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb12fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.653+0000 7fcdb12fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): get_auth_request con 0x7fcdac0a5bb0 auth_method 0 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 10 monclient(hunting): handle_auth_done global_id 114137 payload 824 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcdac0a5bb0 msgr2=0x7fcdac1d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fcdac0a5bb0 0x7fcdac1d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac138a50 msgr2=0x7fcdac13c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac138a50 0x7fcdac13c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 monclient: found mon.noname-e 2024-04-19T00:56:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac1d18a0 con 0x7fcdac137960 2024-04-19T00:56:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac138a50 0x7fcdac13c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdabfff640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac137960 0x7fcdac1369c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcda000ebb0 tx=0x7fcda000c520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 client.?.objecter ms_handle_connect 0x7fcdac137960 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcda0017070 con 0x7fcdac137960 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: dump: 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.675 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.676 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcda000cc30 con 0x7fcdac137960 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 5 monclient: authenticate success, global_id 114137 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 monclient: _renew_subs 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdac09ee60 con 0x7fcdac137960 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcda000cde0 con 0x7fcdac137960 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 client.114137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: dump: 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.678 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 monclient: _renew_subs 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdac121800 con 0x7fcdac137960 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdaaffd640 4 set_mon_vals no callback set 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 1 librados: init done 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 0x7fcdac13cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fcdac06d140 con 0x7fcdac136f00 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 0x7fcdac13cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fcda001a070 con 0x7fcdac137960 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient: get_auth_request con 0x7fcdac136f00 auth_method 0 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd98038e10 0x7fcd9803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_done global_id 64561 payload 824 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdac13e430 con 0x7fcdac136f00 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(439..439 src has 1..439) ==== 6170+0+0 (secure 0 0 0) 0x7fcda000f040 con 0x7fcdac137960 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 client.114137.objecter ms_dispatch 0x7fcdac000ba0 osd_map(439..439 src has 1..439) 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 3 client.114137.objecter handle_osd_map got epochs [439,439] > 0 2024-04-19T00:56:18.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 3 client.114137.objecter handle_osd_map decoding full epoch 439 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 20 client.114137.objecter dump_active .. 0 homeless 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 client.114137.objecter ms_handle_connect 0x7fcdac136f00 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 0x7fcdac13cc80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009870 tx=0x7fcd9c0191b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 client.114137.objecter resend_mon_ops 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd98038e10 0x7fcd9803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd9c0ba070 con 0x7fcdac136f00 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: dump: 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.681 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 10 monclient: get_auth_request con 0x7fcd98038e10 auth_method 0 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcd9c0b1040 con 0x7fcdac136f00 2024-04-19T00:56:18.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 msgr2=0x7fcdac13cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 0x7fcdac13cc80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009870 tx=0x7fcd9c0191b0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 msgr2=0x7fcdac13cc80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac136f00 0x7fcdac13cc80 secure :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c009870 tx=0x7fcd9c0191b0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcda97fa640 10 client.114137.objecter ms_handle_connect 0x7fcd98038e10 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.657+0000 7fcdb12fc640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd98038e10 0x7fcd9803b2d0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0a7500 tx=0x7fcda40077a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 0x7fcdac13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fcdac13db10 con 0x7fcdac13d1c0 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 0x7fcdac13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient: get_auth_request con 0x7fcdac13d1c0 auth_method 0 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 10 monclient(hunting): handle_auth_done global_id 64570 payload 824 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 1 -- 172.21.15.43:0/3151470803 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcdac076190 con 0x7fcdac13d1c0 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb0afb640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 0x7fcdac13d5d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0097c0 tx=0x7fcd9c0a8000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 client.114137.objecter ms_handle_connect 0x7fcdac13d1c0 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 client.114137.objecter resend_mon_ops 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fcd9c0b0070 con 0x7fcdac13d1c0 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 monclient: dump: 2024-04-19T00:56:18.684 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.685 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7fcd9c0be400 con 0x7fcdac13d1c0 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 msgr2=0x7fcdac13d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 0x7fcdac13d5d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0097c0 tx=0x7fcd9c0a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 msgr2=0x7fcdac13d5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcda97fa640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fcdac13d1c0 0x7fcdac13d5d0 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0097c0 tx=0x7fcd9c0a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:18.686 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 304937, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:18.687 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.688 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.689 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.690 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.691 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.693 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.696 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.697 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.704 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.706 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.707 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.708 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:18.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.708 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.708 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:18.708 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:18.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 10 librados: watch_flush enter 2024-04-19T00:56:18.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 10 librados: watch_flush exit 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 20 client.114137.objecter shutdown clearing up homeless session... 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 10 client.114137.objecter successfully canceled tick 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd98038e10 msgr2=0x7fcd9803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fcd98038e10 0x7fcd9803b2d0 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7fcdac0a7500 tx=0x7fcda40077a0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 10 monclient: shutdown 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac137960 msgr2=0x7fcdac1369c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 --2- 172.21.15.43:0/3151470803 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fcdac137960 0x7fcdac1369c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fcda000ebb0 tx=0x7fcda000c520 comp rx=0 tx=0).stop 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb12fc640 1 -- 172.21.15.43:0/3151470803 reap_dead start 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb12fc640 1 -- 172.21.15.43:0/3151470803 reap_dead start 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 shutdown_connections 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 >> 172.21.15.43:0/3151470803 conn(0x7fcdac0a7cb0 msgr2=0x7fcdac0a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 shutdown_connections 2024-04-19T00:56:18.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 -- 172.21.15.43:0/3151470803 wait complete. 2024-04-19T00:56:18.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.665+0000 7fcdb2d5f640 1 librados: shutdown 2024-04-19T00:56:18.711 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:18.711 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:56:18.711 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:56:18.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient: get_monmap_and_config 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient: monmap: 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:18.847990+0000 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:18.847990+0000 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.845 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.846 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.846 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.846 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.846 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:18.846 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:18.847 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 Processor -- start 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 -- start start 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient: init 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient: _add_conns ranks=[5,4,3,10,9,8,2,1,6,7,0] 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83000a4b10 0x7f83001391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): picked mon.noname-h con 0x7f83000a4b10 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 0x7f83000a5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): picked mon.noname-c con 0x7f83000a4ef0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83001358b0 0x7f8300135d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): picked mon.noname-g con 0x7f83001358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f830006e750 con 0x7f83001358b0 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f830006f250 con 0x7f83000a4ef0 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f830006f530 con 0x7f83000a4b10 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f830852c640 10 monclient(hunting): authenticate will time out at 1648.974773s 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 0x7f83000a5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 0x7f83000a5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50684/0 (socket says 172.21.15.43:50684) 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 -- 172.21.15.43:0/1731191603 learned_addr learned my addr 172.21.15.43:0/1731191603 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): get_auth_request con 0x7f83000a4ef0 auth_method 0 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83001358b0 0x7f8300135d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83000a4b10 0x7f83001391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): get_auth_request con 0x7f83001358b0 auth_method 0 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): get_auth_request con 0x7f83000a4b10 auth_method 0 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f83062c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8306ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient(hunting): handle_auth_done global_id 64579 payload 274 2024-04-19T00:56:18.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 -- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83000a4b10 msgr2=0x7f83001391e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83000a4b10 0x7f83001391e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 -- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83001358b0 msgr2=0x7f8300135d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83001358b0 0x7f8300135d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 monclient: found mon.noname-c 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.845+0000 7f8305ac7640 1 -- 172.21.15.43:0/1731191603 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83000a58b0 con 0x7f83000a4ef0 2024-04-19T00:56:18.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f8305ac7640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 0x7f83000a5370 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f82f00098a0 tx=0x7f82f001c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=f90dabcafce05b48 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 1 -- 172.21.15.43:0/1731191603 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82f00b0070 con 0x7f83000a4ef0 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: dump: 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.859 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.860 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 5 monclient: authenticate success, global_id 64579 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 1 -- 172.21.15.43:0/1731191603 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f00aa810 con 0x7f83000a4ef0 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:18.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 4 set_mon_vals no callback set 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 1 -- 172.21.15.43:0/1731191603 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82f00a9300 con 0x7f83000a4ef0 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f83052c6640 10 monclient: dump: 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.862 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 10 monclient: shutdown 2024-04-19T00:56:18.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 -- 172.21.15.43:0/1731191603 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 msgr2=0x7f83000a5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 0x7f83000a5370 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f82f00098a0 tx=0x7f82f001c440 comp rx=0 tx=0).stop 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 -- 172.21.15.43:0/1731191603 shutdown_connections 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f83001358b0 0x7f8300135d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83000a4ef0 0x7f83000a5370 secure :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f82f00098a0 tx=0x7f82f001c440 comp rx=0 tx=0).stop 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 --2- 172.21.15.43:0/1731191603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83000a4b10 0x7f83001391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 -- 172.21.15.43:0/1731191603 >> 172.21.15.43:0/1731191603 conn(0x7f83000a98c0 msgr2=0x7f83000a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 -- 172.21.15.43:0/1731191603 shutdown_connections 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 1 -- 172.21.15.43:0/1731191603 wait complete. 2024-04-19T00:56:18.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 10 monclient: build_initial_monmap 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.849+0000 7f830852c640 10 monclient: monmap: 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:18.865 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 librados: starting msgr at 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 librados: starting objecter 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 Processor -- start 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- start start 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 librados: setting wanted keys 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 librados: calling monclient init 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: init 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: _add_conns ranks=[1,8,2,7,6,0,5,4,3] 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83000a4b10 0x7f8300134b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): picked mon.noname-f con 0x7f83000a4b10 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:18.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83001358b0 0x7f83001d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): picked mon.noname-e con 0x7f83001358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 0x7f83001d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): picked mon.noname-b con 0x7f83000a0500 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83001429a0 con 0x7f83000a4b10 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8300142b20 con 0x7f83000a0500 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8300142ca0 con 0x7f83001358b0 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): authenticate will time out at 1648.980781s 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 0x7f83001d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 0x7f83001d16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35762/0 (socket says 172.21.15.43:35762) 2024-04-19T00:56:18.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 -- 172.21.15.43:0/307695472 learned_addr learned my addr 172.21.15.43:0/307695472 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): get_auth_request con 0x7f83000a0500 auth_method 0 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83000a4b10 0x7f8300134b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 10 monclient(hunting): get_auth_request con 0x7f83000a4b10 auth_method 0 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient(hunting): handle_auth_done global_id 74360 payload 824 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83001358b0 msgr2=0x7f83001d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83001358b0 0x7f83001d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83000a4b10 msgr2=0x7f8300134b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83000a4b10 0x7f8300134b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient: _finish_hunting 0 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 monclient: found mon.noname-b 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83000a0f80 con 0x7f83000a0500 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8306ac9640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 0x7f83001d16f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f82f800efc0 tx=0x7f82f800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 client.?.objecter ms_handle_connect 0x7f83000a0500 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:18.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82f8018020 con 0x7f83000a0500 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: dump: 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.870 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 5 monclient: authenticate success, global_id 74360 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: _renew_subs 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82f800cd30 con 0x7f83000a0500 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83000a1260 con 0x7f83000a0500 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82f80148f0 con 0x7f83000a0500 2024-04-19T00:56:18.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 monclient: dump: 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.872 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 client.74360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: _renew_subs 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83001355a0 con 0x7f83000a0500 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 librados: init done 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f8304ac5640 4 set_mon_vals no callback set 2024-04-19T00:56:18.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f82f801b020 con 0x7f83000a0500 2024-04-19T00:56:18.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82e4038e40 0x7f82e403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f82f804a240 con 0x7f83000a0500 2024-04-19T00:56:18.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 10 client.74360.objecter ms_dispatch 0x7f8300000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 3 client.74360.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 3 client.74360.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 0x7f83001368c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8300139930 con 0x7f83000a09c0 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f82f6ffd640 20 client.74360.objecter dump_active .. 0 homeless 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82e4038e40 0x7f82e403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.853+0000 7f83062c8640 10 monclient: get_auth_request con 0x7f82e4038e40 auth_method 0 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 0x7f83001368c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient: get_auth_request con 0x7f83000a09c0 auth_method 0 2024-04-19T00:56:18.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 client.74360.objecter ms_handle_connect 0x7f82e4038e40 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f83062c8640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82e4038e40 0x7f82e403b300 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009af0 tx=0x7f82fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 10 monclient(hunting): handle_auth_done global_id 74328 payload 824 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f830006e920 con 0x7f83000a09c0 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 client.74360.objecter ms_handle_connect 0x7f83000a09c0 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f8305ac7640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 0x7f83001368c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009870 tx=0x7f82f00045c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 client.74360.objecter resend_mon_ops 2024-04-19T00:56:18.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82f00b8070 con 0x7f83000a09c0 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 monclient: dump: 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.880 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f82f0009490 con 0x7f83000a09c0 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 msgr2=0x7f83001368c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 0x7f83001368c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009870 tx=0x7f82f00045c0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 msgr2=0x7f83001368c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.857+0000 7f82f6ffd640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83000a09c0 0x7f83001368c0 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009870 tx=0x7f82f00045c0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.861+0000 7f830852c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:18.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.861+0000 7f830852c640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 0x7f8300137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.861+0000 7f830852c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.861+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8300137750 con 0x7f8300136e00 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 0x7f8300137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient: get_auth_request con 0x7f8300136e00 auth_method 0 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 10 monclient(hunting): handle_auth_done global_id 74337 payload 824 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 1 -- 172.21.15.43:0/307695472 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8300069630 con 0x7f8300136e00 2024-04-19T00:56:18.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 client.74360.objecter ms_handle_connect 0x7f8300136e00 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 client.74360.objecter resend_mon_ops 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8305ac7640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 0x7f8300137210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82f00097c0 tx=0x7f82f00a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82f00b8030 con 0x7f8300136e00 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 monclient: dump: 2024-04-19T00:56:18.883 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:18.884 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:18.885 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:18.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f82f00bbd70 con 0x7f8300136e00 2024-04-19T00:56:18.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:18.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:18.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 msgr2=0x7f8300137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 0x7f8300137210 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82f00097c0 tx=0x7f82f00a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:18.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 msgr2=0x7f8300137210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:18.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f82f6ffd640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8300136e00 0x7f8300137210 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f82f00097c0 tx=0x7f82f00a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:18.886 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:18.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:18.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 158478, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.887 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.888 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.889 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:18.890 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:18.891 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:18.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:18.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.898 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.900 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.901 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.902 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:18.903 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:18.904 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.906 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:18.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:18.908 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 10 librados: watch_flush enter 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 10 librados: watch_flush exit 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 20 client.74360.objecter shutdown clearing up homeless session... 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 10 client.74360.objecter successfully canceled tick 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82e4038e40 msgr2=0x7f82e403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82e4038e40 0x7f82e403b300 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009af0 tx=0x7f82fc008040 comp rx=0 tx=0).stop 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 10 monclient: shutdown 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 msgr2=0x7f83001d16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:18.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f8306ac9640 1 -- 172.21.15.43:0/307695472 reap_dead start 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 0x7f83001d16f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f82f800efc0 tx=0x7f82f800c490 comp rx=0 tx=0).stop 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 shutdown_connections 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 --2- 172.21.15.43:0/307695472 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83000a0500 0x7f83001d16f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 >> 172.21.15.43:0/307695472 conn(0x7f83000a98c0 msgr2=0x7f830012ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 shutdown_connections 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 -- 172.21.15.43:0/307695472 wait complete. 2024-04-19T00:56:18.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:18.865+0000 7f830852c640 1 librados: shutdown 2024-04-19T00:56:18.911 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:18.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:56:18.912 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:56:19.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient: get_monmap_and_config 2024-04-19T00:56:19.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient: monmap: 2024-04-19T00:56:19.023 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.023 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.023 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:19.026555+0000 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:19.026555+0000 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:19.024 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:19.025 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 Processor -- start 2024-04-19T00:56:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 -- start start 2024-04-19T00:56:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient: init 2024-04-19T00:56:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient: _add_conns ranks=[10,0,3,9,5,2,6,8,4,1,7] 2024-04-19T00:56:19.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6440a4f50 0x7fb6440a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): picked mon.noname-f con 0x7fb6440a4f50 addr v2:172.21.15.161:3300/0 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6440731a0 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6440a16c0 0x7fb6440a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): picked mon.noname-g con 0x7fb6440a16c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb644070660 con 0x7fb6440731a0 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb6440a5330 con 0x7fb6440a4f50 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb64409bc10 con 0x7fb6440a16c0 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 10 monclient(hunting): authenticate will time out at 1649.153875s 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb6440731a0 0x7fb6440a3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47706/0 (socket says 172.21.15.43:47706) 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 1 -- 172.21.15.43:0/1982711022 learned_addr learned my addr 172.21.15.43:0/1982711022 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:56:19.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1320511042 0 0) 0x7fb638001120 con 0x7fb6440731a0 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): _negotiate 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): my global_id is 64764 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 --2- 172.21.15.43:0/1982711022 >> v2:172.21.15.161:3300/0 conn(0x7fb6440a4f50 0x7fb6440a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6440a16c0 0x7fb6440a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb62c003530 con 0x7fb6440731a0 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): get_auth_request con 0x7fb6440a16c0 auth_method 0 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): get_auth_request con 0x7fb6440a4f50 auth_method 0 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3194505841 0 0) 0x7fb62c003530 con 0x7fb6440731a0 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6440a16c0 msgr2=0x7fb6440a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb6440a16c0 0x7fb6440a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 >> v2:172.21.15.161:3300/0 conn(0x7fb6440a4f50 msgr2=0x7fb6440a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 --2- 172.21.15.43:0/1982711022 >> v2:172.21.15.161:3300/0 conn(0x7fb6440a4f50 0x7fb6440a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb644133900 con 0x7fb6440731a0 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fb638001b40 con 0x7fb6440731a0 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: dump: 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.040 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:56:19.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 >> v1:172.21.15.43:6789/0 conn(0x7fb6440731a0 legacy=0x7fb6440a3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient: _add_conns ranks=[4,6,7,8,3,1,5,0,2] 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb62c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): picked mon.c con 0x7fb62c004240 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb62c004dc0 0x7fb62c005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): picked mon.d con 0x7fb62c004dc0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb62c009af0 0x7fb62c00bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): picked mon.i con 0x7fb62c009af0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fb6440a5330 con 0x7fb62c004240 2024-04-19T00:56:19.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb64409bc10 con 0x7fb62c004dc0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb62c00c890 con 0x7fb62c009af0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64159a640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb62c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb62c004dc0 0x7fb62c005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): get_auth_request con 0x7fb62c004240 auth_method 0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): get_auth_request con 0x7fb62c004dc0 auth_method 0 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient(hunting): handle_auth_done global_id 64764 payload 293 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb62c009af0 0x7fb62c00bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 -- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb62c009af0 msgr2=0x7fb62c00bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb62c009af0 0x7fb62c00bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 -- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb62c004dc0 msgr2=0x7fb62c005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb62c004dc0 0x7fb62c005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 monclient: found mon.c 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb642d9d640 1 -- 172.21.15.43:0/1982711022 reap_dead start 2024-04-19T00:56:19.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 1 -- 172.21.15.43:0/1982711022 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c003530 con 0x7fb62c004240 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb64259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 5 monclient: authenticate success, global_id 64764 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb648931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.025+0000 7fb641d9b640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb62c004dc0 0x7fb62c005240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb62c004810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009fc0 tx=0x7fb63c01cc50 comp rx=0 tx=0).ready entity=mon.4 client_cookie=9fe4769a20bdc7b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb63c0b2070 con 0x7fb62c004240 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: dump: 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.045 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.046 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63c0a5dc0 con 0x7fb62c004240 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 4 set_mon_vals no callback set 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 1 -- 172.21.15.43:0/1982711022 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb63c009c10 con 0x7fb62c004240 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64159a640 10 monclient: dump: 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.047 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: shutdown 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 msgr2=0x7fb62c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb62c004810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009fc0 tx=0x7fb63c01cc50 comp rx=0 tx=0).stop 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- 172.21.15.43:0/1982711022 shutdown_connections 2024-04-19T00:56:19.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 --2- 172.21.15.43:0/1982711022 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb62c004810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- 172.21.15.43:0/1982711022 >> 172.21.15.43:0/1982711022 conn(0x7fb6440a4460 msgr2=0x7fb6440a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- 172.21.15.43:0/1982711022 shutdown_connections 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- 172.21.15.43:0/1982711022 wait complete. 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: monmap: 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.049 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 librados: starting msgr at 2024-04-19T00:56:19.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 librados: starting objecter 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 Processor -- start 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- start start 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 librados: setting wanted keys 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 librados: calling monclient init 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: init 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient: _add_conns ranks=[4,8,6,1,5,2,7,3,0] 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb6441cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): picked mon.noname-c con 0x7fb62c004240 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6441305e0 0x7fb644130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): picked mon.noname-e con 0x7fb6441305e0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb64409d0b0 0x7fb64409d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): picked mon.noname-d con 0x7fb64409d0b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fb6440a3f10 con 0x7fb62c004240 2024-04-19T00:56:19.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb6440a3d90 con 0x7fb64409d0b0 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb6440a4090 con 0x7fb6441305e0 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb648931640 10 monclient(hunting): authenticate will time out at 1649.159609s 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb6441cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6441305e0 0x7fb644130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb6441cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50704/0 (socket says 172.21.15.43:50704) 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 1 -- 172.21.15.43:0/186279113 learned_addr learned my addr 172.21.15.43:0/186279113 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): get_auth_request con 0x7fb62c004240 auth_method 0 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): get_auth_request con 0x7fb6441305e0 auth_method 0 2024-04-19T00:56:19.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb64409d0b0 0x7fb64409d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): get_auth_request con 0x7fb64409d0b0 auth_method 0 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb641d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.029+0000 7fb642d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 10 monclient(hunting): handle_auth_done global_id 114146 payload 824 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb64409d0b0 msgr2=0x7fb64409d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb64409d0b0 0x7fb64409d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 msgr2=0x7fb6441cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb6441cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 monclient: found mon.noname-e 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64406fe80 con 0x7fb6441305e0 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb62c004240 0x7fb6441cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb641d9b640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6441305e0 0x7fb644130a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0040e0 tx=0x7fb638007720 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 client.?.objecter ms_handle_connect 0x7fb6441305e0 2024-04-19T00:56:19.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb638007d20 con 0x7fb6441305e0 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: dump: 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.055 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb638018070 con 0x7fb6441305e0 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb638001b40 con 0x7fb6441305e0 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: dump: 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.056 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb633fff640 4 set_mon_vals no callback set 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 5 monclient: authenticate success, global_id 114146 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 monclient: _renew_subs 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64406f9c0 con 0x7fb6441305e0 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 client.114146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 monclient: _renew_subs 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6440a0f00 con 0x7fb6441305e0 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 1 librados: init done 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 0x7fb6440a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb64406d140 con 0x7fb64409da70 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb63800f930 con 0x7fb6441305e0 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 0x7fb6440a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient: get_auth_request con 0x7fb64409da70 auth_method 0 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb624038e10 0x7fb62403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7fb638010040 con 0x7fb6441305e0 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 client.114146.objecter ms_dispatch 0x7fb644000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 3 client.114146.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 3 client.114146.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 10 monclient(hunting): handle_auth_done global_id 114155 payload 824 2024-04-19T00:56:19.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6441ccd60 con 0x7fb64409da70 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 20 client.114146.objecter dump_active .. 0 homeless 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 client.114146.objecter ms_handle_connect 0x7fb64409da70 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 client.114146.objecter resend_mon_ops 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb64259c640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 0x7fb6440a66f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb63c01cc90 tx=0x7fb63c009750 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb63c0bc070 con 0x7fb64409da70 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: dump: 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb642d9d640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb624038e10 0x7fb62403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb642d9d640 10 monclient: get_auth_request con 0x7fb624038e10 auth_method 0 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb63c0a5d00 con 0x7fb64409da70 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 msgr2=0x7fb6440a66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 0x7fb6440a66f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb63c01cc90 tx=0x7fb63c009750 comp rx=0 tx=0).stop 2024-04-19T00:56:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 msgr2=0x7fb6440a66f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb64409da70 0x7fb6440a66f0 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb63c01cc90 tx=0x7fb63c009750 comp rx=0 tx=0).stop 2024-04-19T00:56:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb642d9d640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb624038e10 0x7fb62403b2d0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7fb64406de10 tx=0x7fb634008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.033+0000 7fb632ffd640 10 client.114146.objecter ms_handle_connect 0x7fb624038e10 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 0x7fb6440a7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb64409f480 con 0x7fb6440a6c30 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 0x7fb6440a7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient: get_auth_request con 0x7fb6440a6c30 auth_method 0 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 10 monclient(hunting): handle_auth_done global_id 114164 payload 824 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 1 -- 172.21.15.43:0/186279113 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb64406dab0 con 0x7fb6440a6c30 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb64259c640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 0x7fb6440a7040 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb63c01ca80 tx=0x7fb63c004a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 client.114146.objecter ms_handle_connect 0x7fb6440a6c30 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 client.114146.objecter resend_mon_ops 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb63c0bc070 con 0x7fb6440a6c30 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 monclient: dump: 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.065 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fb63c0b8480 con 0x7fb6440a6c30 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 msgr2=0x7fb6440a7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 0x7fb6440a7040 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb63c01ca80 tx=0x7fb63c004a50 comp rx=0 tx=0).stop 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 msgr2=0x7fb6440a7040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb632ffd640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6440a6c30 0x7fb6440a7040 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb63c01ca80 tx=0x7fb63c004a50 comp rx=0 tx=0).stop 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:19.067 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16203, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:19.068 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.069 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:19.070 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.071 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:19.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.075 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:19.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.078 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:19.083 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.084 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.085 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:19.086 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:19.087 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:19.088 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:19.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 10 librados: watch_flush enter 2024-04-19T00:56:19.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 10 librados: watch_flush exit 2024-04-19T00:56:19.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 20 client.114146.objecter shutdown clearing up homeless session... 2024-04-19T00:56:19.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 10 client.114146.objecter successfully canceled tick 2024-04-19T00:56:19.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb624038e10 msgr2=0x7fb62403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb624038e10 0x7fb62403b2d0 secure :-1 s=READY pgs=3524 cs=0 l=1 rev1=1 crypto rx=0x7fb64406de10 tx=0x7fb634008040 comp rx=0 tx=0).stop 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 10 monclient: shutdown 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6441305e0 msgr2=0x7fb644130a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 --2- 172.21.15.43:0/186279113 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb6441305e0 0x7fb644130a40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0040e0 tx=0x7fb638007720 comp rx=0 tx=0).stop 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb642d9d640 1 -- 172.21.15.43:0/186279113 reap_dead start 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb642d9d640 1 -- 172.21.15.43:0/186279113 reap_dead start 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 shutdown_connections 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 >> 172.21.15.43:0/186279113 conn(0x7fb6440a4460 msgr2=0x7fb6441321f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 shutdown_connections 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 -- 172.21.15.43:0/186279113 wait complete. 2024-04-19T00:56:19.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.041+0000 7fb648931640 1 librados: shutdown 2024-04-19T00:56:19.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:19.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:56:19.092 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.201+0000 7f82ed86a640 10 monclient: get_monmap_and_config 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.201+0000 7f82ed86a640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient: monmap: 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:19.206427+0000 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:19.206427+0000 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:19.203 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:19.204 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 Processor -- start 2024-04-19T00:56:19.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 -- start start 2024-04-19T00:56:19.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient: init 2024-04-19T00:56:19.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient: _add_conns ranks=[6,9,1,10,4,8,3,7,5,0,2] 2024-04-19T00:56:19.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f82e80a6420 0x7f82e80a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): picked mon.noname-d con 0x7f82e80a6420 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f82e80a6800 0x7f82e80a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): picked mon.noname-a con 0x7f82e80a6800 addr v2:172.21.15.43:3300/0 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f82e8138b50 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82e80a20f0 con 0x7f82e8138b50 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f82e809f340 con 0x7f82e80a6800 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f82e809f540 con 0x7f82e80a6420 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82ed86a640 10 monclient(hunting): authenticate will time out at 1649.333453s 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f82e80a6800 0x7f82e80a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f82e80a6420 0x7f82e80a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f82e80a6800 0x7f82e80a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40386/0 (socket says 172.21.15.43:40386) 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 -- 172.21.15.43:0/2995951264 learned_addr learned my addr 172.21.15.43:0/2995951264 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): get_auth_request con 0x7f82e80a6800 auth_method 0 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): get_auth_request con 0x7f82e80a6420 auth_method 0 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient(hunting): handle_auth_done global_id 64773 payload 274 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 -- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f82e80a6420 msgr2=0x7f82e80a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 --2- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f82e80a6420 0x7f82e80a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 -- 172.21.15.43:0/2995951264 >> v1:172.21.15.161:6789/0 conn(0x7f82e8138b50 legacy=0x7f82e8138fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 monclient: found mon.noname-a 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:19.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.205+0000 7f82e67fc640 1 -- 172.21.15.43:0/2995951264 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e80a07f0 con 0x7f82e80a6800 2024-04-19T00:56:19.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 --2- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e80a6800 0x7f82e80a6c20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009870 tx=0x7f82dc01ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99526217fa050bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 1 -- 172.21.15.43:0/2995951264 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82dc0b2070 con 0x7f82e80a6800 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: dump: 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.216 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 1 -- 172.21.15.43:0/2995951264 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc0ab7b0 con 0x7f82e80a6800 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 1 -- 172.21.15.43:0/2995951264 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82dc0abe50 con 0x7f82e80a6800 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 5 monclient: authenticate success, global_id 64773 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 4 set_mon_vals no callback set 2024-04-19T00:56:19.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e5ffb640 10 monclient: dump: 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.218 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: shutdown 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e80a6800 msgr2=0x7f82e80a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 --2- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e80a6800 0x7f82e80a6c20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009870 tx=0x7f82dc01ce50 comp rx=0 tx=0).stop 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- 172.21.15.43:0/2995951264 shutdown_connections 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 --2- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f82e80a6800 0x7f82e80a6c20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 --2- 172.21.15.43:0/2995951264 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f82e80a6420 0x7f82e80a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- 172.21.15.43:0/2995951264 >> 172.21.15.43:0/2995951264 conn(0x7f82e80ab1b0 msgr2=0x7f82e80ab5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- 172.21.15.43:0/2995951264 shutdown_connections 2024-04-19T00:56:19.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- 172.21.15.43:0/2995951264 wait complete. 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: monmap: 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.220 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 librados: starting msgr at 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 librados: starting objecter 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 Processor -- start 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- start start 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 librados: setting wanted keys 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 librados: calling monclient init 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: init 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient: _add_conns ranks=[7,5,1,4,8,6,2,3,0] 2024-04-19T00:56:19.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f82e80a6420 0x7f82e81d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): picked mon.noname-i con 0x7f82e80a6420 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 0x7f82e8137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): picked mon.noname-h con 0x7f82e80a6800 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8138b50 0x7f82e8138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): picked mon.noname-f con 0x7f82e8138b50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8143a40 con 0x7f82e8138b50 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f82e81438c0 con 0x7f82e80a6800 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f82e8143bc0 con 0x7f82e80a6420 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82ed86a640 10 monclient(hunting): authenticate will time out at 1649.338200s 2024-04-19T00:56:19.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f82e80a6420 0x7f82e81d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 0x7f82e8137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8138b50 0x7f82e8138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 0x7f82e8137dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:40814/0 (socket says 172.21.15.43:40814) 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 -- 172.21.15.43:0/3921519464 learned_addr learned my addr 172.21.15.43:0/3921519464 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): get_auth_request con 0x7f82e80a6800 auth_method 0 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): get_auth_request con 0x7f82e8138b50 auth_method 0 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): get_auth_request con 0x7f82e80a6420 auth_method 0 2024-04-19T00:56:19.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient(hunting): handle_auth_done global_id 74264 payload 824 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f82e80a6420 msgr2=0x7f82e81d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f82e80a6420 0x7f82e81d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8138b50 msgr2=0x7f82e8138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8138b50 0x7f82e8138310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 monclient: found mon.noname-h 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e809dca0 con 0x7f82e80a6800 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e77fe640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8138b50 0x7f82e8138310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82e67fc640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 0x7f82e8137dd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009870 tx=0x7f82dc019190 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82cffff640 10 client.?.objecter ms_handle_connect 0x7f82e80a6800 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82cffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82dc0b2070 con 0x7f82e80a6800 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.209+0000 7f82cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:56:19.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: dump: 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.226 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82dc0abdd0 con 0x7f82e80a6800 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 5 monclient: authenticate success, global_id 74264 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 monclient: _renew_subs 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:56:19.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82e81365e0 con 0x7f82e80a6800 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82dc003620 con 0x7f82e80a6800 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e57fa640 4 set_mon_vals no callback set 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 client.74264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: dump: 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.228 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 monclient: _renew_subs 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f82e8138850 con 0x7f82e80a6800 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 1 librados: init done 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 0x7f82e809be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82e806d140 con 0x7f82e8075f30 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 0x7f82e809be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient: get_auth_request con 0x7f82e8075f30 auth_method 0 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f82dc0b6030 con 0x7f82e80a6800 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82d4038e10 0x7f82d403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_done global_id 114139 payload 824 2024-04-19T00:56:19.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f82dc0e31e0 con 0x7f82e80a6800 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 client.74264.objecter ms_dispatch 0x7f82e8000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82e809fd90 con 0x7f82e8075f30 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 3 client.74264.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 3 client.74264.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 20 client.74264.objecter dump_active .. 0 homeless 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e77fe640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82d4038e10 0x7f82d403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e77fe640 10 monclient: get_auth_request con 0x7f82d4038e10 auth_method 0 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 client.74264.objecter ms_handle_connect 0x7f82e8075f30 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 client.74264.objecter resend_mon_ops 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e6ffd640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 0x7f82e809be60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f82d80119d0 tx=0x7f82d8011e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82d8019070 con 0x7f82e8075f30 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:19.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: dump: 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.232 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 client.74264.objecter ms_handle_connect 0x7f82d4038e10 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82e77fe640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82d4038e10 0x7f82d403b2d0 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7f82e000b7a0 tx=0x7f82e000d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f82d8004300 con 0x7f82e8075f30 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 msgr2=0x7f82e809be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 0x7f82e809be60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f82d80119d0 tx=0x7f82d8011e90 comp rx=0 tx=0).stop 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 msgr2=0x7f82e809be60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.213+0000 7f82cffff640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e8075f30 0x7f82e809be60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82ed86a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82ed86a640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 0x7f82e809c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82ed86a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82e8068240 con 0x7f82e809c3a0 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 0x7f82e809c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient: get_auth_request con 0x7f82e809c3a0 auth_method 0 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 10 monclient(hunting): handle_auth_done global_id 114148 payload 824 2024-04-19T00:56:19.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 1 -- 172.21.15.43:0/3921519464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82e809ccc0 con 0x7f82e809c3a0 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82e6ffd640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 0x7f82e809c780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82d80119d0 tx=0x7f82d80046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 client.74264.objecter ms_handle_connect 0x7f82e809c3a0 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 client.74264.objecter resend_mon_ops 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f82d8019040 con 0x7f82e809c3a0 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 monclient: dump: 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.236 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f82d801e450 con 0x7f82e809c3a0 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 msgr2=0x7f82e809c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 0x7f82e809c780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82d80119d0 tx=0x7f82d80046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 msgr2=0x7f82e809c780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.221+0000 7f82cffff640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82e809c3a0 0x7f82e809c780 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82d80119d0 tx=0x7f82d80046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16387, 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:19.238 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.239 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.240 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:19.241 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:19.242 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.243 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:19.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.251 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.253 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.255 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.259 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.260 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.260 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:19.260 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 10 librados: watch_flush enter 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 10 librados: watch_flush exit 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 20 client.74264.objecter shutdown clearing up homeless session... 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 10 client.74264.objecter successfully canceled tick 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82d4038e10 msgr2=0x7f82d403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f82d4038e10 0x7f82d403b2d0 secure :-1 s=READY pgs=3525 cs=0 l=1 rev1=1 crypto rx=0x7f82e000b7a0 tx=0x7f82e000d040 comp rx=0 tx=0).stop 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 10 monclient: shutdown 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 msgr2=0x7f82e8137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 0x7f82e8137dd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f82dc009870 tx=0x7f82dc019190 comp rx=0 tx=0).stop 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82e77fe640 1 -- 172.21.15.43:0/3921519464 reap_dead start 2024-04-19T00:56:19.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 shutdown_connections 2024-04-19T00:56:19.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 --2- 172.21.15.43:0/3921519464 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f82e80a6800 0x7f82e8137dd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 >> 172.21.15.43:0/3921519464 conn(0x7f82e80ab1b0 msgr2=0x7f82e80a3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 shutdown_connections 2024-04-19T00:56:19.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 -- 172.21.15.43:0/3921519464 wait complete. 2024-04-19T00:56:19.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.225+0000 7f82ed86a640 1 librados: shutdown 2024-04-19T00:56:19.263 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:19.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:56:19.263 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.369+0000 7f0333ef7640 10 monclient: get_monmap_and_config 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.369+0000 7f0333ef7640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.369+0000 7f0333ef7640 10 monclient: monmap: 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:19.373769+0000 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:19.373769+0000 2024-04-19T00:56:19.370 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:19.371 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 Processor -- start 2024-04-19T00:56:19.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 -- start start 2024-04-19T00:56:19.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient: init 2024-04-19T00:56:19.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient: _add_conns ranks=[2,1,3,6,4,8,10,0,7,5,9] 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 0x7f032c136dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): picked mon.noname-b con 0x7f032c0a1560 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f032c0a1940 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c0a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): picked mon.noname-g con 0x7f032c0a1d60 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032c070380 con 0x7f032c0a1940 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f032c137310 con 0x7f032c0a1560 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f032c070660 con 0x7f032c0a1d60 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient(hunting): authenticate will time out at 1649.500890s 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 0x7f032c136dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 0x7f032c136dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35766/0 (socket says 172.21.15.43:35766) 2024-04-19T00:56:19.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 -- 172.21.15.43:0/3299937126 learned_addr learned my addr 172.21.15.43:0/3299937126 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): get_auth_request con 0x7f032c0a1560 auth_method 0 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0332494640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c0a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient(hunting): handle_auth_done global_id 74369 payload 274 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 -- 172.21.15.43:0/3299937126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 msgr2=0x7f032c0a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c0a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 -- 172.21.15.43:0/3299937126 >> v1:172.21.15.161:6789/0 conn(0x7f032c0a1940 legacy=0x7f032c0a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 monclient: found mon.noname-b 2024-04-19T00:56:19.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:19.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 -- 172.21.15.43:0/3299937126 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c0a5210 con 0x7f032c0a1560 2024-04-19T00:56:19.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0332494640 10 monclient: get_auth_request con 0x7f032c0a1d60 auth_method 0 2024-04-19T00:56:19.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0332494640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c0a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:19.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0331c93640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 0x7f032c136dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f03280098a0 tx=0x7f032801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c266cd43ccc437e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 1 -- 172.21.15.43:0/3299937126 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03280b0070 con 0x7f032c0a1560 2024-04-19T00:56:19.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: dump: 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.383 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 1 -- 172.21.15.43:0/3299937126 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03280aa810 con 0x7f032c0a1560 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 5 monclient: authenticate success, global_id 74369 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 1 -- 172.21.15.43:0/3299937126 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03280a9300 con 0x7f032c0a1560 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0333ef7640 4 set_mon_vals no callback set 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.373+0000 7f0330c91640 10 monclient: dump: 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.385 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: shutdown 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3299937126 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 msgr2=0x7f032c136dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 0x7f032c136dd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f03280098a0 tx=0x7f032801c4d0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3299937126 shutdown_connections 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c0a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3299937126 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f032c0a1560 0x7f032c136dd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3299937126 >> 172.21.15.43:0/3299937126 conn(0x7f032c0a7ef0 msgr2=0x7f032c0a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3299937126 shutdown_connections 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3299937126 wait complete. 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: monmap: 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.386 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 librados: starting msgr at 2024-04-19T00:56:19.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 librados: starting objecter 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 Processor -- start 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- start start 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 librados: setting wanted keys 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 librados: calling monclient init 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: init 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: _add_conns ranks=[4,6,8,3,2,5,1,7,0] 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f032c0a1560 0x7f032c130650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): picked mon.noname-c con 0x7f032c0a1560 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f032c0a1940 0x7f032c130b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): picked mon.noname-d con 0x7f032c0a1940 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f032c1d08a0 0x7f032c1d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): picked mon.noname-e con 0x7f032c1d08a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f032c141470 con 0x7f032c0a1560 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f032c0a7660 con 0x7f032c0a1940 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f032c1415f0 con 0x7f032c1d08a0 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient(hunting): authenticate will time out at 1649.506503s 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f032c0a1940 0x7f032c130b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f032c0a1560 0x7f032c130650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f032c0a1940 0x7f032c130b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40454/0 (socket says 172.21.15.43:40454) 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f032c0a1560 0x7f032c130650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50710/0 (socket says 172.21.15.43:50710) 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 -- 172.21.15.43:0/3728262362 learned_addr learned my addr 172.21.15.43:0/3728262362 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): get_auth_request con 0x7f032c0a1940 auth_method 0 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f032c1d08a0 0x7f032c1d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): get_auth_request con 0x7f032c0a1560 auth_method 0 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): get_auth_request con 0x7f032c1d08a0 auth_method 0 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0332494640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient(hunting): handle_auth_done global_id 74364 payload 824 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f032c1d08a0 msgr2=0x7f032c1d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f032c1d08a0 0x7f032c1d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f032c0a1560 msgr2=0x7f032c130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f032c0a1560 0x7f032c130650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 monclient: found mon.noname-d 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c06fae0 con 0x7f032c0a1940 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331c93640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f032c0a1560 0x7f032c130650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:19.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0331492640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f032c0a1940 0x7f032c130b90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f032400c7d0 tx=0x7f032400cc90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 client.?.objecter ms_handle_connect 0x7f032c0a1940 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0324016020 con 0x7f032c0a1940 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: dump: 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.392 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 5 monclient: authenticate success, global_id 74364 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f032400acc0 con 0x7f032c0a1940 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: _renew_subs 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:19.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c13d550 con 0x7f032c0a1940 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0324012890 con 0x7f032c0a1940 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 client.74364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0323fff640 4 set_mon_vals no callback set 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f03227fc640 10 monclient: dump: 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.394 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: _renew_subs 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f032c13ccb0 con 0x7f032c0a1940 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.377+0000 7f0333ef7640 1 librados: init done 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0333ef7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c1d1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f032c06d140 con 0x7f032c0a1d60 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0324019020 con 0x7f032c0a1940 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c1d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0318038e10 0x7f031803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient: get_auth_request con 0x7f032c0a1d60 auth_method 0 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f03240493b0 con 0x7f032c0a1940 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 client.74364.objecter ms_dispatch 0x7f032c000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 3 client.74364.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 3 client.74364.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 20 client.74364.objecter dump_active .. 0 homeless 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0332494640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0318038e10 0x7f031803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0332494640 10 monclient: get_auth_request con 0x7f0318038e10 auth_method 0 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 10 monclient(hunting): handle_auth_done global_id 19416 payload 824 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f032c0008e0 con 0x7f032c0a1d60 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 client.74364.objecter ms_handle_connect 0x7f032c0a1d60 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 client.74364.objecter resend_mon_ops 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0331c93640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c1d1240 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f03280ad000 tx=0x7f0328019270 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 client.74364.objecter ms_handle_connect 0x7f0318038e10 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f0332494640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0318038e10 0x7f031803b2d0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f032c06f970 tx=0x7f031c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03280b0040 con 0x7f032c0a1d60 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 monclient: dump: 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.397 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.398 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f03280aa960 con 0x7f032c0a1d60 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 msgr2=0x7f032c1d1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c1d1240 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f03280ad000 tx=0x7f0328019270 comp rx=0 tx=0).stop 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 msgr2=0x7f032c1d1240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.381+0000 7f03227fc640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c0a1d60 0x7f032c1d1240 secure :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f03280ad000 tx=0x7f0328019270 comp rx=0 tx=0).stop 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 0x7f032c1314e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f032c13dcd0 con 0x7f032c1310d0 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 0x7f032c1314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient: get_auth_request con 0x7f032c1310d0 auth_method 0 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 10 monclient(hunting): handle_auth_done global_id 19425 payload 824 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 1 -- 172.21.15.43:0/3728262362 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f032c1396b0 con 0x7f032c1310d0 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0331c93640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 0x7f032c1314e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f032c077690 tx=0x7f03280a6000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 client.74364.objecter ms_handle_connect 0x7f032c1310d0 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 client.74364.objecter resend_mon_ops 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03280b0040 con 0x7f032c1310d0 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 monclient: dump: 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f03280c4070 con 0x7f032c1310d0 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:19.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 msgr2=0x7f032c1314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 0x7f032c1314e0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f032c077690 tx=0x7f03280a6000 comp rx=0 tx=0).stop 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 msgr2=0x7f032c1314e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f03227fc640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f032c1310d0 0x7f032c1314e0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 722872, 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:19.404 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.405 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.406 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:19.407 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:19.409 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.416 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:19.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.418 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.419 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.423 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:19.424 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 10 librados: watch_flush enter 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 10 librados: watch_flush exit 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 20 client.74364.objecter shutdown clearing up homeless session... 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 10 client.74364.objecter successfully canceled tick 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0318038e10 msgr2=0x7f031803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0318038e10 0x7f031803b2d0 secure :-1 s=READY pgs=3526 cs=0 l=1 rev1=1 crypto rx=0x7f032c06f970 tx=0x7f031c008040 comp rx=0 tx=0).stop 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 10 monclient: shutdown 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f032c0a1940 msgr2=0x7f032c130b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 --2- 172.21.15.43:0/3728262362 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f032c0a1940 0x7f032c130b90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f032400c7d0 tx=0x7f032400cc90 comp rx=0 tx=0).stop 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0332494640 1 -- 172.21.15.43:0/3728262362 reap_dead start 2024-04-19T00:56:19.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0332494640 1 -- 172.21.15.43:0/3728262362 reap_dead start 2024-04-19T00:56:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 shutdown_connections 2024-04-19T00:56:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 >> 172.21.15.43:0/3728262362 conn(0x7f032c0a7ef0 msgr2=0x7f032c0a2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.389+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 shutdown_connections 2024-04-19T00:56:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.393+0000 7f0333ef7640 1 -- 172.21.15.43:0/3728262362 wait complete. 2024-04-19T00:56:19.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.393+0000 7f0333ef7640 1 librados: shutdown 2024-04-19T00:56:19.427 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:19.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:56:19.427 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.533+0000 7fa8902ab640 10 monclient: get_monmap_and_config 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.533+0000 7fa8902ab640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.533+0000 7fa8902ab640 10 monclient: monmap: 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:19.538315+0000 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:19.538315+0000 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:19.535 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:19.536 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 Processor -- start 2024-04-19T00:56:19.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 -- start start 2024-04-19T00:56:19.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient: init 2024-04-19T00:56:19.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient: _add_conns ranks=[8,1,3,7,0,9,6,4,2,5,10] 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8880a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): picked mon.noname-e con 0x7fa8880a51b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa8880a3ef0 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): picked mon.noname-g con 0x7fa8880a42d0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8880a02c0 con 0x7fa8880a3ef0 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa8880a24b0 con 0x7fa8880a42d0 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa88809fee0 con 0x7fa8880a51b0 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa8902ab640 10 monclient(hunting): authenticate will time out at 1649.665823s 2024-04-19T00:56:19.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8880a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8880a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56112/0 (socket says 172.21.15.43:56112) 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 -- 172.21.15.43:0/3476887925 learned_addr learned my addr 172.21.15.43:0/3476887925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): get_auth_request con 0x7fa8880a51b0 auth_method 0 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e848640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient(hunting): handle_auth_done global_id 114182 payload 274 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 -- 172.21.15.43:0/3476887925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 msgr2=0x7fa888135ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 -- 172.21.15.43:0/3476887925 >> v1:172.21.15.161:6789/0 conn(0x7fa8880a3ef0 legacy=0x7fa8881357d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 monclient: found mon.noname-e 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e047640 1 -- 172.21.15.43:0/3476887925 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8880a10e0 con 0x7fa8880a51b0 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e848640 10 monclient: get_auth_request con 0x7fa8880a42d0 auth_method 0 2024-04-19T00:56:19.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.537+0000 7fa88e848640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:19.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e047640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8880a39b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa8840098a0 tx=0x7fa88401c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=d49ecd1138daa625 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 1 -- 172.21.15.43:0/3476887925 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8840b0070 con 0x7fa8880a51b0 2024-04-19T00:56:19.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: dump: 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.548 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 1 -- 172.21.15.43:0/3476887925 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8840aa810 con 0x7fa8880a51b0 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 1 -- 172.21.15.43:0/3476887925 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8840a9300 con 0x7fa8880a51b0 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d045640 10 monclient: dump: 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.549 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 5 monclient: authenticate success, global_id 114182 2024-04-19T00:56:19.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 4 set_mon_vals no callback set 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: shutdown 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/3476887925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 msgr2=0x7fa8880a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8880a39b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa8840098a0 tx=0x7fa88401c440 comp rx=0 tx=0).stop 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/3476887925 shutdown_connections 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 --2- 172.21.15.43:0/3476887925 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8880a39b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/3476887925 >> 172.21.15.43:0/3476887925 conn(0x7fa8880a85a0 msgr2=0x7fa8880a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/3476887925 shutdown_connections 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/3476887925 wait complete. 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: monmap: 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:19.551 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 librados: starting msgr at 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 librados: starting objecter 2024-04-19T00:56:19.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 Processor -- start 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- start start 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 librados: setting wanted keys 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 librados: calling monclient init 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: init 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: _add_conns ranks=[5,3,8,0,4,1,2,7,6] 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8880a3ef0 0x7fa888135800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): picked mon.noname-h con 0x7fa8880a3ef0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): picked mon.noname-g con 0x7fa8880a42d0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8881d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): picked mon.noname-e con 0x7fa8880a51b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa8881406c0 con 0x7fa8880a42d0 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa888140540 con 0x7fa8880a3ef0 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa888140840 con 0x7fa8880a51b0 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient(hunting): authenticate will time out at 1649.670554s 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8881d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8881d0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56128/0 (socket says 172.21.15.43:56128) 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 -- 172.21.15.43:0/1537513183 learned_addr learned my addr 172.21.15.43:0/1537513183 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): get_auth_request con 0x7fa8880a51b0 auth_method 0 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d846640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d846640 10 monclient(hunting): get_auth_request con 0x7fa8880a42d0 auth_method 0 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d846640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d846640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d846640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e047640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8880a3ef0 0x7fa888135800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient(hunting): handle_auth_done global_id 114191 payload 824 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8880a3ef0 msgr2=0x7fa888135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8880a3ef0 0x7fa888135800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 msgr2=0x7fa888135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 monclient: found mon.noname-e 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa888120ed0 con 0x7fa8880a51b0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e047640 10 monclient: get_auth_request con 0x7fa8880a3ef0 auth_method 0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e047640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8880a3ef0 0x7fa888135800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88d846640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa8880a42d0 0x7fa888135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88e848640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8881d0cc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa88000cef0 tx=0x7fa88000a710 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 client.?.objecter ms_handle_connect 0x7fa8880a51b0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa88001a070 con 0x7fa8880a51b0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: dump: 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.556 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa88000dab0 con 0x7fa8880a51b0 2024-04-19T00:56:19.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa88000dc60 con 0x7fa8880a51b0 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa88c844640 4 set_mon_vals no callback set 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa87effd640 10 monclient: dump: 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.558 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 5 monclient: authenticate success, global_id 114191 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: _renew_subs 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8881231c0 con 0x7fa8880a51b0 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 client.114191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: _renew_subs 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8881ce270 con 0x7fa8880a51b0 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.541+0000 7fa8902ab640 1 librados: init done 2024-04-19T00:56:19.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa8902ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa8902ab640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 0x7fa8881d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa88806d140 con 0x7fa8881d1200 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa88000d640 con 0x7fa8880a51b0 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa86c038e10 0x7fa86c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7fa880049cb0 con 0x7fa8880a51b0 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 client.114191.objecter ms_dispatch 0x7fa888000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 3 client.114191.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 3 client.114191.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 0x7fa8881d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 20 client.114191.objecter dump_active .. 0 homeless 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient: get_auth_request con 0x7fa8881d1200 auth_method 0 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88d846640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa86c038e10 0x7fa86c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88d846640 10 monclient: get_auth_request con 0x7fa86c038e10 auth_method 0 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 10 monclient(hunting): handle_auth_done global_id 74273 payload 824 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8881d1b20 con 0x7fa8881d1200 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88e047640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 0x7fa8881d15e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa8840ad000 tx=0x7fa8840191e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 client.114191.objecter ms_handle_connect 0x7fa8881d1200 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 client.114191.objecter resend_mon_ops 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 client.114191.objecter ms_handle_connect 0x7fa86c038e10 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa88d846640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa86c038e10 0x7fa86c03b2d0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fa888120d00 tx=0x7fa878009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8840b9020 con 0x7fa8881d1200 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:19.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 monclient: dump: 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.562 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa8840b1040 con 0x7fa8881d1200 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 msgr2=0x7fa8881d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 0x7fa8881d15e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa8840ad000 tx=0x7fa8840191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 msgr2=0x7fa8881d15e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.545+0000 7fa87effd640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa8881d1200 0x7fa8881d15e0 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa8840ad000 tx=0x7fa8840191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 0x7fa888070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa888136280 con 0x7fa88806fff0 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 0x7fa888070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient: get_auth_request con 0x7fa88806fff0 auth_method 0 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 10 monclient(hunting): handle_auth_done global_id 74282 payload 824 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 1 -- 172.21.15.43:0/1537513183 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8880a3d10 con 0x7fa88806fff0 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e047640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 0x7fa888070400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa884009870 tx=0x7fa8840a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 client.114191.objecter ms_handle_connect 0x7fa88806fff0 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 client.114191.objecter resend_mon_ops 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa8840b9040 con 0x7fa88806fff0 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 monclient: dump: 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.566 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa8840c0020 con 0x7fa88806fff0 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:19.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 msgr2=0x7fa888070400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 0x7fa888070400 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa884009870 tx=0x7fa8840a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 msgr2=0x7fa888070400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa87effd640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa88806fff0 0x7fa888070400 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa884009870 tx=0x7fa8840a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 159170, 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:19.568 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.569 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:19.571 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.572 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:19.573 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.574 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:19.576 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:19.579 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:19.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.586 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:19.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:19.588 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:19.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 10 librados: watch_flush enter 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 10 librados: watch_flush exit 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 20 client.114191.objecter shutdown clearing up homeless session... 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 10 client.114191.objecter successfully canceled tick 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa86c038e10 msgr2=0x7fa86c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa86c038e10 0x7fa86c03b2d0 secure :-1 s=READY pgs=3527 cs=0 l=1 rev1=1 crypto rx=0x7fa888120d00 tx=0x7fa878009040 comp rx=0 tx=0).stop 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 10 monclient: shutdown 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 msgr2=0x7fa8881d0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa88e848640 1 -- 172.21.15.43:0/1537513183 reap_dead start 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8881d0cc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa88000cef0 tx=0x7fa88000a710 comp rx=0 tx=0).stop 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 shutdown_connections 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 --2- 172.21.15.43:0/1537513183 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa8880a51b0 0x7fa8881d0cc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 >> 172.21.15.43:0/1537513183 conn(0x7fa8880a85a0 msgr2=0x7fa8880a5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 shutdown_connections 2024-04-19T00:56:19.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 -- 172.21.15.43:0/1537513183 wait complete. 2024-04-19T00:56:19.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.553+0000 7fa8902ab640 1 librados: shutdown 2024-04-19T00:56:19.591 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:19.591 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:56:19.591 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:56:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient: get_monmap_and_config 2024-04-19T00:56:19.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient: monmap: 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:19.715928+0000 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:19.715928+0000 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.713 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:19.714 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:19.714 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 Processor -- start 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 -- start start 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient: init 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient: _add_conns ranks=[0,2,6,10,3,8,7,9,1,5,4] 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a64138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a640a5b80 0x7f0a6413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): picked mon.noname-b con 0x7f0a640a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a641397c0 0x7f0a641407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): picked mon.noname-d con 0x7f0a641397c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a640a3dc0 con 0x7f0a64138cd0 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0a640a4410 con 0x7f0a640a5b80 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0a640a40a0 con 0x7f0a641397c0 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a6b652640 10 monclient(hunting): authenticate will time out at 1649.843060s 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a693ee640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0a64138cd0 0x7f0a641390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47708/0 (socket says 172.21.15.43:47708) 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a693ee640 1 -- 172.21.15.43:0/4294557826 learned_addr learned my addr 172.21.15.43:0/4294557826 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a641397c0 0x7f0a641407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a640a5b80 0x7f0a6413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): get_auth_request con 0x7f0a641397c0 auth_method 0 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): get_auth_request con 0x7f0a640a5b80 auth_method 0 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient(hunting): handle_auth_done global_id 74373 payload 274 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 1 -- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a640a5b80 msgr2=0x7f0a6413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a5bfff640 1 -- 172.21.15.43:0/4294557826 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1635778957 0 0) 0x7f0a640a3dc0 con 0x7f0a64138cd0 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a640a5b80 0x7f0a6413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 1 -- 172.21.15.43:0/4294557826 >> v1:172.21.15.43:6789/0 conn(0x7f0a64138cd0 legacy=0x7f0a641390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 1 monclient: found mon.noname-d 2024-04-19T00:56:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a69bef640 1 -- 172.21.15.43:0/4294557826 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a640a2ae0 con 0x7f0a641397c0 2024-04-19T00:56:19.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.713+0000 7f0a5bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a69bef640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a641397c0 0x7f0a641407c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0a600097f0 tx=0x7f0a6001c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=c27aefcd2042903c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 1 -- 172.21.15.43:0/4294557826 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a600b0070 con 0x7f0a641397c0 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: dump: 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.726 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 5 monclient: authenticate success, global_id 74373 2024-04-19T00:56:19.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 1 -- 172.21.15.43:0/4294557826 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a600aa810 con 0x7f0a641397c0 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 4 set_mon_vals no callback set 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 1 -- 172.21.15.43:0/4294557826 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a600a9300 con 0x7f0a641397c0 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a5bfff640 10 monclient: dump: 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.728 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 10 monclient: shutdown 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 -- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a641397c0 msgr2=0x7f0a641407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a641397c0 0x7f0a641407c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0a600097f0 tx=0x7f0a6001c3a0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 -- 172.21.15.43:0/4294557826 shutdown_connections 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0a641397c0 0x7f0a641407c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 --2- 172.21.15.43:0/4294557826 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0a640a5b80 0x7f0a6413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 -- 172.21.15.43:0/4294557826 >> 172.21.15.43:0/4294557826 conn(0x7f0a640ab2f0 msgr2=0x7f0a640ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 -- 172.21.15.43:0/4294557826 shutdown_connections 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 -- 172.21.15.43:0/4294557826 wait complete. 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 10 monclient: monmap: 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.730 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 librados: starting msgr at 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 librados: starting objecter 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 Processor -- start 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 -- start start 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 librados: setting wanted keys 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 1 librados: calling monclient init 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.717+0000 7f0a6b652640 10 monclient: init 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient: _add_conns ranks=[4,8,3,0,2,6,1,7,5] 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a640a5b80 0x7f0a640a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): picked mon.noname-c con 0x7f0a640a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a64138cd0 0x7f0a64135a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): picked mon.noname-e con 0x7f0a64138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a641397c0 0x7f0a64135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): picked mon.noname-g con 0x7f0a641397c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0a64143730 con 0x7f0a641397c0 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0a640a20a0 con 0x7f0a640a5b80 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a641438b0 con 0x7f0a64138cd0 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): authenticate will time out at 1649.848298s 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a640a5b80 0x7f0a640a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a64138cd0 0x7f0a64135a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a64138cd0 0x7f0a64135a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56144/0 (socket says 172.21.15.43:56144) 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a640a5b80 0x7f0a640a0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:50726/0 (socket says 172.21.15.43:50726) 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 -- 172.21.15.43:0/1621541008 learned_addr learned my addr 172.21.15.43:0/1621541008 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): get_auth_request con 0x7f0a64138cd0 auth_method 0 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): get_auth_request con 0x7f0a640a5b80 auth_method 0 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a69bef640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a641397c0 0x7f0a64135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient(hunting): handle_auth_done global_id 64606 payload 824 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a64138cd0 msgr2=0x7f0a64135a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a64138cd0 0x7f0a64135a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a641397c0 msgr2=0x7f0a64135f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a641397c0 0x7f0a64135f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 monclient: found mon.noname-c 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a640a3590 con 0x7f0a640a5b80 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a64138cd0 0x7f0a64135a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a69bef640 10 monclient: get_auth_request con 0x7f0a641397c0 auth_method 0 2024-04-19T00:56:19.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a69bef640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0a641397c0 0x7f0a64135f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a693ee640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a640a5b80 0x7f0a640a0e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a640a72e0 tx=0x7f0a5c00cf70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 client.?.objecter ms_handle_connect 0x7f0a640a5b80 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a5c012400 con 0x7f0a640a5b80 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: dump: 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.735 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a5c012600 con 0x7f0a640a5b80 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 5 monclient: authenticate success, global_id 64606 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient: _renew_subs 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:19.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a640a03c0 con 0x7f0a640a5b80 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a5c012ca0 con 0x7f0a640a5b80 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a5b7fe640 4 set_mon_vals no callback set 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 client.64606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: dump: 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.737 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient: _renew_subs 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6409ec60 con 0x7f0a640a5b80 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 librados: init done 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0a5c012830 con 0x7f0a640a5b80 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 0x7f0a6409cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0a6409ef80 con 0x7f0a641364c0 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a48038e10 0x7f0a4803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f0a5c049980 con 0x7f0a640a5b80 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 client.64606.objecter ms_dispatch 0x7f0a64000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 3 client.64606.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 3 client.64606.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 20 client.64606.objecter dump_active .. 0 homeless 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 0x7f0a6409cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient: get_auth_request con 0x7f0a641364c0 auth_method 0 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a69bef640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a48038e10 0x7f0a4803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a69bef640 10 monclient: get_auth_request con 0x7f0a48038e10 auth_method 0 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 10 monclient(hunting): handle_auth_done global_id 74266 payload 824 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a641368a0 con 0x7f0a641364c0 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a69bef640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a48038e10 0x7f0a4803b2d0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f0a600ad000 tx=0x7f0a600a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a68bed640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 0x7f0a6409cea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a54006aa0 tx=0x7f0a5400f610 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 client.64606.objecter ms_handle_connect 0x7f0a48038e10 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 client.64606.objecter ms_handle_connect 0x7f0a641364c0 2024-04-19T00:56:19.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 client.64606.objecter resend_mon_ops 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a540044a0 con 0x7f0a641364c0 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.721+0000 7f0a59ffb640 10 monclient: dump: 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.742 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a54004030 con 0x7f0a641364c0 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 msgr2=0x7f0a6409cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 0x7f0a6409cea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a54006aa0 tx=0x7f0a5400f610 comp rx=0 tx=0).stop 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 msgr2=0x7f0a6409cea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.725+0000 7f0a59ffb640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a641364c0 0x7f0a6409cea0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a54006aa0 tx=0x7f0a5400f610 comp rx=0 tx=0).stop 2024-04-19T00:56:19.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a6b652640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:19.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a6b652640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 0x7f0a6402c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a6b652640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0a6406d140 con 0x7f0a6402c450 2024-04-19T00:56:19.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 0x7f0a6402c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient: get_auth_request con 0x7f0a6402c450 auth_method 0 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 10 monclient(hunting): handle_auth_done global_id 74275 payload 824 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.729+0000 7f0a68bed640 1 -- 172.21.15.43:0/1621541008 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a640693b0 con 0x7f0a6402c450 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a68bed640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 0x7f0a6402c860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a5401b000 tx=0x7f0a540046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 client.64606.objecter ms_handle_connect 0x7f0a6402c450 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 client.64606.objecter resend_mon_ops 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a54003e50 con 0x7f0a6402c450 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 monclient: dump: 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.747 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f0a540273a0 con 0x7f0a6402c450 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 msgr2=0x7f0a6402c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 0x7f0a6402c860 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0a5401b000 tx=0x7f0a540046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 msgr2=0x7f0a6402c860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a59ffb640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0a6402c450 0x7f0a6402c860 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:19.748 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 159350, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:19.749 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.750 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.751 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:19.752 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.753 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:19.754 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.756 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:19.757 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:19.759 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:19.762 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.765 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:19.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.767 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:19.768 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:19.769 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:19.769 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 10 librados: watch_flush enter 2024-04-19T00:56:19.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 10 librados: watch_flush exit 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 20 client.64606.objecter shutdown clearing up homeless session... 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 10 client.64606.objecter successfully canceled tick 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a48038e10 msgr2=0x7f0a4803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a48038e10 0x7f0a4803b2d0 secure :-1 s=READY pgs=3528 cs=0 l=1 rev1=1 crypto rx=0x7f0a600ad000 tx=0x7f0a600a6000 comp rx=0 tx=0).stop 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 10 monclient: shutdown 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a640a5b80 msgr2=0x7f0a640a0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 --2- 172.21.15.43:0/1621541008 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a640a5b80 0x7f0a640a0e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f0a640a72e0 tx=0x7f0a5c00cf70 comp rx=0 tx=0).stop 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a69bef640 1 -- 172.21.15.43:0/1621541008 reap_dead start 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a69bef640 1 -- 172.21.15.43:0/1621541008 reap_dead start 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 shutdown_connections 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 >> 172.21.15.43:0/1621541008 conn(0x7f0a640ab2f0 msgr2=0x7f0a64137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 shutdown_connections 2024-04-19T00:56:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 -- 172.21.15.43:0/1621541008 wait complete. 2024-04-19T00:56:19.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.733+0000 7f0a6b652640 1 librados: shutdown 2024-04-19T00:56:19.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:19.772 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T00:56:19.772 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.889+0000 7f11ba84a640 10 monclient: get_monmap_and_config 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.889+0000 7f11ba84a640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.889+0000 7f11ba84a640 10 monclient: monmap: 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:19.893289+0000 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:19.893289+0000 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.890 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:19.891 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 Processor -- start 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 -- start start 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient: init 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient: _add_conns ranks=[6,10,1,3,4,0,7,9,2,5,8] 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 0x7f11b40a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): picked mon.noname-d con 0x7f11b40a1f20 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f11b40a1270 0x7f11b40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): picked mon.noname-f con 0x7f11b40a1270 addr v2:172.21.15.161:3300/0 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f11b40a4430 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b409fff0 con 0x7f11b40a4430 2024-04-19T00:56:19.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f11b40a0920 con 0x7f11b40a1270 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f11b406bf40 con 0x7f11b40a1f20 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient(hunting): authenticate will time out at 1650.020966s 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 0x7f11b40a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 0x7f11b40a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:40476/0 (socket says 172.21.15.43:40476) 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 -- 172.21.15.43:0/33541814 learned_addr learned my addr 172.21.15.43:0/33541814 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f11b40a1f20 auth_method 0 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b37fe640 1 --2- 172.21.15.43:0/33541814 >> v2:172.21.15.161:3300/0 conn(0x7f11b40a1270 0x7f11b40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient(hunting): handle_auth_done global_id 74382 payload 274 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 -- 172.21.15.43:0/33541814 >> v2:172.21.15.161:3300/0 conn(0x7f11b40a1270 msgr2=0x7f11b40a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 --2- 172.21.15.43:0/33541814 >> v2:172.21.15.161:3300/0 conn(0x7f11b40a1270 0x7f11b40a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 -- 172.21.15.43:0/33541814 >> v1:172.21.15.161:6789/0 conn(0x7f11b40a4430 legacy=0x7f11b413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 monclient: found mon.noname-d 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 -- 172.21.15.43:0/33541814 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b40a02d0 con 0x7f11b40a1f20 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b37fe640 10 monclient: get_auth_request con 0x7f11b40a1270 auth_method 0 2024-04-19T00:56:19.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b37fe640 1 --2- 172.21.15.43:0/33541814 >> v2:172.21.15.161:3300/0 conn(0x7f11b40a1270 0x7f11b40a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:19.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b3fff640 1 --2- 172.21.15.43:0/33541814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 0x7f11b40a39b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f11a4009870 tx=0x7f11a401c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4de9a8291698ed5e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 1 -- 172.21.15.43:0/33541814 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f11a40b0070 con 0x7f11b40a1f20 2024-04-19T00:56:19.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: dump: 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.901 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 5 monclient: authenticate success, global_id 74382 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 1 -- 172.21.15.43:0/33541814 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a40aa810 con 0x7f11b40a1f20 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11ba84a640 4 set_mon_vals no callback set 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 1 -- 172.21.15.43:0/33541814 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f11a40aaa40 con 0x7f11b40a1f20 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.893+0000 7f11b2ffd640 10 monclient: dump: 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.903 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: shutdown 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/33541814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 msgr2=0x7f11b40a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 --2- 172.21.15.43:0/33541814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 0x7f11b40a39b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f11a4009870 tx=0x7f11a401c420 comp rx=0 tx=0).stop 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/33541814 shutdown_connections 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 --2- 172.21.15.43:0/33541814 >> v2:172.21.15.161:3300/0 conn(0x7f11b40a1270 0x7f11b40a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 --2- 172.21.15.43:0/33541814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f11b40a1f20 0x7f11b40a39b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/33541814 >> 172.21.15.43:0/33541814 conn(0x7f11b40a85a0 msgr2=0x7f11b40a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/33541814 shutdown_connections 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/33541814 wait complete. 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: build_initial_monmap 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: monmap: 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:19.904 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 librados: starting msgr at 2024-04-19T00:56:19.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 librados: starting objecter 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 Processor -- start 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- start start 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 librados: setting wanted keys 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 librados: calling monclient init 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: init 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: _add_conns ranks=[0,5,2,1,8,7,4,6,3] 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 0x7f11b41393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): picked mon.noname-a con 0x7f11b40a1270 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f11b40a1f20 0x7f11b4139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): picked mon.noname-h con 0x7f11b40a1f20 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11b40a4430 0x7f11b413ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): picked mon.noname-b con 0x7f11b40a4430 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f11b40a2480 con 0x7f11b40a1270 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f11b40a2300 con 0x7f11b40a4430 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f11b4143ae0 con 0x7f11b40a1f20 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient(hunting): authenticate will time out at 1650.026240s 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 0x7f11b41393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 0x7f11b41393f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:40400/0 (socket says 172.21.15.43:40400) 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 -- 172.21.15.43:0/1328052548 learned_addr learned my addr 172.21.15.43:0/1328052548 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f11b40a1270 auth_method 0 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11b40a4430 0x7f11b413ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): get_auth_request con 0x7f11b40a4430 auth_method 0 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b8de7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient(hunting): handle_auth_done global_id 64791 payload 824 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 -- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f11b40a1f20 msgr2=0x7f11b4139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b37fe640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f11b40a1f20 0x7f11b4139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f11b40a1f20 0x7f11b4139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 -- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11b40a4430 msgr2=0x7f11b413ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:19.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11b40a4430 0x7f11b413ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 monclient: found mon.noname-a 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 -- 172.21.15.43:0/1328052548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b4140890 con 0x7f11b40a1270 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b37fe640 10 monclient: get_auth_request con 0x7f11b40a1f20 auth_method 0 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b37fe640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f11b40a1f20 0x7f11b4139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b3fff640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 0x7f11b41393f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f11a40097c0 tx=0x7f11a40191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 client.?.objecter ms_handle_connect 0x7f11b40a1270 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f11a40b0040 con 0x7f11b40a1270 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: dump: 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.909 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.910 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: _finish_auth 0 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a40b9070 con 0x7f11b40a1270 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 5 monclient: authenticate success, global_id 64791 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: _renew_subs 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b4073100 con 0x7f11b40a1270 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f11a40044b0 con 0x7f11b40a1270 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b27fc640 4 set_mon_vals no callback set 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 client.64791.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11b0ff9640 10 monclient: dump: 2024-04-19T00:56:19.911 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:19.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: _renew_subs 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b4124040 con 0x7f11b40a1270 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.897+0000 7f11ba84a640 1 librados: init done 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11ba84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11ba84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11ba84a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b4076110 con 0x7f11b40a1270 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f11a4004040 con 0x7f11b40a1270 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f11a0038e10 0x7f11a003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f11a40e3260 con 0x7f11b40a1270 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 10 client.64791.objecter ms_dispatch 0x7f11b4000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 3 client.64791.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 3 client.64791.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:19.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 20 client.64791.objecter dump_active .. 0 homeless 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b37fe640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f11a0038e10 0x7f11a003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b37fe640 10 monclient: get_auth_request con 0x7f11a0038e10 auth_method 0 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b37fe640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f11a0038e10 0x7f11a003b2d0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f11b4140320 tx=0x7f11a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.901+0000 7f11b0ff9640 10 client.64791.objecter ms_handle_connect 0x7f11a0038e10 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.905+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f11b4073100 con 0x7f11b40a1270 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.905+0000 7f11b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:19.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:19.905+0000 7f11b0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:20.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.121+0000 7f11ba84a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T00:56:20.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.121+0000 7f11ba84a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T00:56:20.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.121+0000 7f11ba84a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:20.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.121+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f11b409ef70 con 0x7f11b40a1270 2024-04-19T00:56:20.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.125+0000 7f11b0ff9640 1 -- 172.21.15.43:0/1328052548 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f11a4016130 con 0x7f11b40a1270 2024-04-19T00:56:20.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.125+0000 7f11b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T00:56:20.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.125+0000 7f11b0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:20.128 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.125+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.128 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.125+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.128 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.125+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=10,kv=11,logm=60} crc {auth=265809136,config=2586702516,health=4186951596,kv=801871554,logm=3455856297}) 2024-04-19T00:56:20.128 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.125+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=2955163502,config=2586702516,health=1639843326,kv=2349598857,logm=1620985126}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=1,health=12,kv=11,logm=56} crc {auth=3927413247,config=1228700967,health=1639843326,kv=4171600807,logm=1322340349}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=1077833371,config=2586702516,health=3217976746,kv=4196696685,logm=3263430627}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=19,config=2,health=9,kv=12,logm=58} crc {auth=2566421590,config=2586702516,health=2435042222,kv=4196696685,logm=1812642542}) 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.129 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=17,config=2,health=12,kv=11,logm=58} crc {auth=189973409,config=2586702516,health=1639843326,kv=801871553,logm=4146357478}) 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=3805800729,config=2586702516,health=1639843326,kv=2243130063,logm=2591931897}) 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=19,config=2,health=11,kv=10,logm=58} crc {auth=2313604381,config=2586702516,health=80391209,kv=3426330989,logm=3437303672}) 2024-04-19T00:56:20.130 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.129+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=2313604381,config=2586702516,health=1639843326,kv=519798866,logm=3231166490}) 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 10 librados: watch_flush enter 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 10 librados: watch_flush exit 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 20 client.64791.objecter shutdown clearing up homeless session... 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 10 client.64791.objecter successfully canceled tick 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f11a0038e10 msgr2=0x7f11a003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f11a0038e10 0x7f11a003b2d0 secure :-1 s=READY pgs=3529 cs=0 l=1 rev1=1 crypto rx=0x7f11b4140320 tx=0x7f11a8008040 comp rx=0 tx=0).stop 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 10 monclient: shutdown 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 msgr2=0x7f11b41393f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:20.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 0x7f11b41393f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f11a40097c0 tx=0x7f11a40191c0 comp rx=0 tx=0).stop 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 shutdown_connections 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f11a0038e10 0x7f11a003b2d0 unknown :-1 s=CLOSED pgs=3529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f11b40a4430 0x7f11b413ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f11b40a1f20 0x7f11b4139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 --2- 172.21.15.43:0/1328052548 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f11b40a1270 0x7f11b41393f0 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f11a40097c0 tx=0x7f11a40191c0 comp rx=0 tx=0).stop 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 >> 172.21.15.43:0/1328052548 conn(0x7f11b40a85a0 msgr2=0x7f11b40a7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 shutdown_connections 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 -- 172.21.15.43:0/1328052548 wait complete. 2024-04-19T00:56:20.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:20.129+0000 7f11ba84a640 1 librados: shutdown 2024-04-19T00:56:20.134 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3468091067}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1414584129}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555518985}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=691963652}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.135 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1547786728}) 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2765180550}) 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=682099286}) 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=147045819}) 2024-04-19T00:56:20.136 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.133+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4104355302}) 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120620909}) 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491504167}) 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.139 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581255169}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3226401352}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3652537631}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3727456705}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.140 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=79646238}) 2024-04-19T00:56:20.141 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.141 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2052336206}) 2024-04-19T00:56:20.141 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.137+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4291580552}) 2024-04-19T00:56:20.144 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.141+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.144 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.141+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.144 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.141+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188398467}) 2024-04-19T00:56:20.144 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.141+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.144 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3463776664}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2608981054}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295950648}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4284293759}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1866370347}) 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.145 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.146 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1765522001}) 2024-04-19T00:56:20.146 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.146 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3264734676}) 2024-04-19T00:56:20.146 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.145+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2406276150}) 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619790824}) 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1523460306}) 2024-04-19T00:56:20.148 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1029509664}) 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2690320279}) 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3526889321}) 2024-04-19T00:56:20.153 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=933194740}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3490338319}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1897883415}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.149+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1341801986}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1617887706}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.154 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3886528037}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2326850661}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1132549831}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=791278933}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=646243444}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.155 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3425278603}) 2024-04-19T00:56:20.156 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.156 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=805179175}) 2024-04-19T00:56:20.156 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.153+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1547769631}) 2024-04-19T00:56:20.158 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.158 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.158 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213712098}) 2024-04-19T00:56:20.158 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.158 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.158 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447738591}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1728766602}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=805245377}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4019883221}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3013555382}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.159 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=202644938}) 2024-04-19T00:56:20.160 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.160 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1958832521}) 2024-04-19T00:56:20.160 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.157+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1699565427}) 2024-04-19T00:56:20.162 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.162 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.162 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172612503}) 2024-04-19T00:56:20.162 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.162 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.162 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3175024716}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2895930968}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1008163707}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1390691157}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2424851129}) 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.163 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.164 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2353486336}) 2024-04-19T00:56:20.164 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.164 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1069703296}) 2024-04-19T00:56:20.164 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.161+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1929587048}) 2024-04-19T00:56:20.166 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.166 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.166 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3370923347}) 2024-04-19T00:56:20.166 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.166 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527533508}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2936841292}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3615167729}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=353979594}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2818908843}) 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.167 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.168 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1635255612}) 2024-04-19T00:56:20.168 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.168 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3250944519}) 2024-04-19T00:56:20.168 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.165+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3086594708}) 2024-04-19T00:56:20.170 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.170 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.170 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797757556}) 2024-04-19T00:56:20.170 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.170 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3060480256}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915983603}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2984896861}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=137162588}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2223336716}) 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.171 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.172 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2194445609}) 2024-04-19T00:56:20.172 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.172 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=391442043}) 2024-04-19T00:56:20.172 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.169+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=839321005}) 2024-04-19T00:56:20.174 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.174 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.174 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2317574519}) 2024-04-19T00:56:20.174 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921872975}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244058514}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2597554496}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2888501744}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.175 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1866838463}) 2024-04-19T00:56:20.176 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.176 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.176 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1188954115}) 2024-04-19T00:56:20.176 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.176 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388486685}) 2024-04-19T00:56:20.176 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.173+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2552396475}) 2024-04-19T00:56:20.178 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.178 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.178 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169942144}) 2024-04-19T00:56:20.178 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.178 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293005309}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522840920}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=933173781}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2675730222}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3662192796}) 2024-04-19T00:56:20.179 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.180 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.180 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4038060452}) 2024-04-19T00:56:20.180 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.180 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2665613170}) 2024-04-19T00:56:20.180 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.177+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2813236459}) 2024-04-19T00:56:20.182 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.182 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.182 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=819950744}) 2024-04-19T00:56:20.182 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.182 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221891414}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439769361}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2177908830}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3549971212}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3093004142}) 2024-04-19T00:56:20.183 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.184 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.184 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3493227388}) 2024-04-19T00:56:20.184 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.184 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2460117978}) 2024-04-19T00:56:20.184 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.181+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2598260117}) 2024-04-19T00:56:20.186 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.186 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.186 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121246285}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4008647162}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293437839}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3147468983}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=956949554}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.187 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2326489366}) 2024-04-19T00:56:20.188 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.188 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.188 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4128125211}) 2024-04-19T00:56:20.188 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.188 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2498626861}) 2024-04-19T00:56:20.188 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.185+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1490314613}) 2024-04-19T00:56:20.190 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147728563}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553644936}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458396822}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1033827561}) 2024-04-19T00:56:20.191 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1088243162}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1784942998}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=217412687}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=372207168}) 2024-04-19T00:56:20.192 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.189+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1013055784}) 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1940912051}) 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008688878}) 2024-04-19T00:56:20.197 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2925491358}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2784431325}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2737020248}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=734614340}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.198 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2091383259}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=757402432}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.193+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=373858040}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915289998}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1082319756}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1616149399}) 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.199 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2437646114}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2560767518}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1830242125}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=229406398}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3935843530}) 2024-04-19T00:56:20.200 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.197+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2671174823}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340819476}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3188248891}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=572465401}) 2024-04-19T00:56:20.205 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2439454603}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2534723181}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3064145049}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=500550807}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1086564458}) 2024-04-19T00:56:20.206 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1754717538}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.201+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765768248}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945971988}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3133051695}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3813293223}) 2024-04-19T00:56:20.207 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1701845454}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=405432638}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3631579946}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524891655}) 2024-04-19T00:56:20.208 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1075634665}) 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372088054}) 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985717064}) 2024-04-19T00:56:20.215 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1641363458}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.205+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3540113883}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3236274277}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=916344450}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.216 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1365366825}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3916540311}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2867327928}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472583750}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=889619231}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1776602460}) 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.217 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600049910}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1500406875}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2278614858}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1860747680}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937824793}) 2024-04-19T00:56:20.218 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.209+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3325129109}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505788609}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3562913483}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226479415}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=199442347}) 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.219 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3061870156}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1494181307}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2855269131}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2632074588}) 2024-04-19T00:56:20.220 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.213+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2099136501}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019772514}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1017653369}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499709131}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.236 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2148691477}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2218895192}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3861362798}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1464724042}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3069869059}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.217+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1031720866}) 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.237 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693566828}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578921407}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605526845}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1115836381}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2138905448}) 2024-04-19T00:56:20.238 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=611085496}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2489474025}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1677288563}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3313124203}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3990867573}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.239 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1084638378}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977492607}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160440523}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1789607074}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2449392901}) 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.240 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4058090342}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037800437}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.221+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1839725924}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=185062695}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3477622061}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748919757}) 2024-04-19T00:56:20.241 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=830501240}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4269627023}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=975817523}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1320676719}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=631614562}) 2024-04-19T00:56:20.242 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.225+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=69897762}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320113491}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016678247}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034448415}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3444262386}) 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.243 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2871899141}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2853949680}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3088238998}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1353029322}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4004696339}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723268970}) 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.244 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1369671015}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239720244}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.229+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=510271446}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2163409419}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=526328782}) 2024-04-19T00:56:20.245 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3412239495}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3382312435}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2314461115}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2964826783}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=801012403}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.246 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=715348493}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2482203349}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3259387354}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2811251606}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2863231809}) 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.247 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=106509120}) 2024-04-19T00:56:20.248 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.233+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2412384139}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008742104}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170135073}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1538359700}) 2024-04-19T00:56:20.268 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1879373755}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=427035414}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1483632438}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1535015634}) 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.269 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2376553088}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.237+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4218093396}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3533574629}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228136198}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179151915}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3887475611}) 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.270 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2179173058}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1687410792}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4250522193}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781257539}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.241+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1086367475}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901586539}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.271 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335932320}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983249414}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3330238564}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1898657568}) 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.272 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3233052561}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2296724039}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2281139179}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.245+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2206551206}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154000089}) 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.273 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869902902}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185669738}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=587304950}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2239058047}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.274 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1782174997}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2733173774}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1534330036}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.249+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2869860564}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949202134}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.275 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=842265704}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824621519}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3889366431}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3963839712}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=356322540}) 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.276 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=996764562}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=766959841}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2662426925}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552753577}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3276215191}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3350915604}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.277 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=399836271}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3654800114}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1568056427}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3377435114}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1240820528}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.253+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2036925547}) 2024-04-19T00:56:20.278 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1741421335}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882854570}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1781066880}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2575059166}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3930641713}) 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.279 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1230585801}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=36431858}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250031524}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.257+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3823996535}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240622713}) 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.280 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3463332545}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587426743}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2379825693}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1455632205}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2985117661}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.281 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=908793375}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3628185887}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.261+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3170547953}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3826197225}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734848186}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962364742}) 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.282 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1895128557}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1129608979}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2628470187}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=810695734}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1369697582}) 2024-04-19T00:56:20.283 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.265+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=120851428}) 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329302017}) 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1427471380}) 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.306 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202336829}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812249606}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=67644327}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2594121176}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.307 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2637442631}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3738356680}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.269+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2182000695}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653550180}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=367738413}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2513348037}) 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.308 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1468051260}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=641591443}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1494402726}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3899643627}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004362288}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.273+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=715485268}) 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.309 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594731700}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4062431459}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54280699}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1400846326}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3153642524}) 2024-04-19T00:56:20.310 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2785661172}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3190118637}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310034444}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.277+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3677380732}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340906269}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.311 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033531951}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2779374771}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2365640901}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2247741994}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2527479079}) 2024-04-19T00:56:20.312 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2429494032}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2715362647}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1845162947}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783202194}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1145020785}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812971567}) 2024-04-19T00:56:20.313 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=994668789}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2147499543}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3395218137}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1610964575}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621524475}) 2024-04-19T00:56:20.314 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.281+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3591662563}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678479722}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3145762486}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465348951}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=634006073}) 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.315 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1523879854}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2696407015}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1066918817}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062525501}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.285+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=923244155}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255063324}) 2024-04-19T00:56:20.316 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1485603077}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=546028860}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2117232467}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=704494425}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2435288505}) 2024-04-19T00:56:20.317 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3905662406}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3956990772}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3034402411}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=889914819}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2351353640}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.318 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726406818}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=160308683}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3963975049}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1974864947}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.289+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=780097383}) 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.319 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3380269980}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4250062612}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2578630732}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301694959}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1049840827}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=722495489}) 2024-04-19T00:56:20.320 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1121691076}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1748521591}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2048950523}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1614735915}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.293+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3159254722}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492652451}) 2024-04-19T00:56:20.321 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027675342}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2062616230}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3155149546}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=647551402}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.322 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3799650466}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3824497176}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=561811597}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.297+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1862532887}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498387981}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086533513}) 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.323 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867759234}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=523612412}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3632295404}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3152147140}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1227892144}) 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=721231115}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.301+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=926688287}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976658682}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2284246867}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644228137}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2495664082}) 2024-04-19T00:56:20.325 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3135944744}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4181478123}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=655989068}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4004161129}) 2024-04-19T00:56:20.326 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1020067212}) 2024-04-19T00:56:20.356 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.356 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.305+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.356 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499223413}) 2024-04-19T00:56:20.356 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.356 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.356 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=815753579}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226445171}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1656016129}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3027979494}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3324920196}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.357 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1907414996}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1760136041}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3974870501}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713669743}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3359929798}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025361475}) 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.358 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1276695682}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=431180677}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1143507310}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3762654370}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=137766516}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.309+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=708258706}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.359 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015026466}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3108214479}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2521335940}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1422835842}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3307243948}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.360 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2038480100}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3731685873}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3758784550}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.313+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1662616592}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455854386}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3526711730}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.361 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1023844850}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546055276}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1378590615}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=281543629}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3275244751}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3148724360}) 2024-04-19T00:56:20.362 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.317+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1583562066}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763262303}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3054244990}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3359232877}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2921669345}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.363 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=348957805}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3521202233}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1627305849}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3649491917}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.321+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3026072367}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301239014}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3877169398}) 2024-04-19T00:56:20.364 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1676465780}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=755681925}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=42303975}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3681125663}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2390414406}) 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.365 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1131699891}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2049284771}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516399536}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=606398450}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=983111453}) 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.366 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3794048860}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.325+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2130884096}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=829260415}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3668759484}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2139004804}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1869659772}) 2024-04-19T00:56:20.367 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15039785}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2016774265}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3832975452}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2008291757}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3716677379}) 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.368 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1951086}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4018166393}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2121957308}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.329+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=825306511}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000224188}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386503869}) 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.369 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009710600}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1603105943}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3964891314}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3650363840}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3527951648}) 2024-04-19T00:56:20.370 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3830146720}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.333+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4080044335}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070346243}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=448810304}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2090061599}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1327494170}) 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.371 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2620897518}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1054884655}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1533413646}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3685764154}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.337+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3800033832}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607562146}) 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.372 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670911022}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2306787602}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1706924577}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3042434766}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1608935847}) 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.373 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3204041916}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2637971695}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1571334220}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102284636}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915765851}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.374 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=400236618}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3119032510}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2446458711}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2251615327}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=390557976}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227368432}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.341+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3612622155}) 2024-04-19T00:56:20.375 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4817460}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607943597}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511418005}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2512929051}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4107313329}) 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.376 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2709717934}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3013949838}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=257012167}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.345+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4229504304}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875601382}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372472583}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.377 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850500907}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=209390798}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1333342582}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2212831965}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1125865788}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2155254353}) 2024-04-19T00:56:20.378 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.349+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3106991385}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1489427681}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2086493596}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=23987528}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2274652907}) 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.379 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2229329140}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1438271998}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4187182664}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1700976369}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1522109712}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481658174}) 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.353+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.380 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3772827881}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2363075391}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3765323577}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=137735388}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=49931727}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.381 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3480143720}) 2024-04-19T00:56:20.382 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.382 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4185612889}) 2024-04-19T00:56:20.382 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2239315562}) 2024-04-19T00:56:20.436 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283873174}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3093607326}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=953155914}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2717887712}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.437 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1108041}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1029951612}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=254332692}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3908570153}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.357+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=925065698}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1966835920}) 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.438 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223710100}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2515002596}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859557625}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2154236223}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3453853872}) 2024-04-19T00:56:20.439 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4190875785}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=676517460}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.361+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43445251}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899268099}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=944703194}) 2024-04-19T00:56:20.440 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2837470815}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=650252088}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2092404652}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3515499789}) 2024-04-19T00:56:20.441 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=793159491}) 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=682587670}) 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.365+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4145991119}) 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155275016}) 2024-04-19T00:56:20.442 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723532196}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999252559}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3768767215}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2496837523}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2693843885}) 2024-04-19T00:56:20.443 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2567480124}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=626267274}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.369+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2194259475}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3608254412}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86292887}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.444 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2896718931}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1837816743}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1139736972}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4274893225}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3012653774}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4076927465}) 2024-04-19T00:56:20.445 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.373+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=826727047}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595703377}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36848430}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2040737922}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2910621443}) 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.446 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2969672253}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=840209937}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3393613200}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808679043}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3130965509}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.377+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012007482}) 2024-04-19T00:56:20.447 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130899120}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403178577}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1813227855}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2999957151}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3354765876}) 2024-04-19T00:56:20.448 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4081888429}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3906855607}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=262257696}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=143642161}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615627853}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1530620415}) 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.449 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=631251834}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=862386252}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=511146460}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=19948419}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2514887687}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.381+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3998655151}) 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.450 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567554704}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781299125}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20151361}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1644198235}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2365309979}) 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.451 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4037993292}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1863823050}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1724493904}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1954689686}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1206852972}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1565623021}) 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.452 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954010416}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1947920753}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4028494345}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=409622358}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.385+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=626294285}) 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.453 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2522080615}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1869543111}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2544483589}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938621928}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568029280}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4256840315}) 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.454 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2267001028}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3517778229}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=999194204}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353361156}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.389+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2065077751}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490245199}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.455 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379023269}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547980708}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4236657135}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=208941811}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=862970230}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3256100721}) 2024-04-19T00:56:20.456 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4000325956}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.393+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3190641977}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155786557}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2440936125}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461386746}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=427309681}) 2024-04-19T00:56:20.457 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=153592642}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3451637863}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3680408368}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209326821}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4163217341}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058247652}) 2024-04-19T00:56:20.458 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1864426252}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=203580753}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2728464440}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1424947872}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1968904754}) 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.459 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=621828898}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=643849005}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.397+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1015810741}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169038238}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3287696750}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194050730}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.460 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1179902680}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2486690532}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3860782941}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=596176005}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=920933261}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.401+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3087307881}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103023588}) 2024-04-19T00:56:20.461 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918992707}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458314347}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=68494346}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3041038068}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2788907486}) 2024-04-19T00:56:20.462 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2107039849}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3890356317}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=176340366}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477573978}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2048996505}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2828153067}) 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.463 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3419342136}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2706148897}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=90699224}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1401414703}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1777335644}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.405+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2551433167}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.464 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64452824}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682912030}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126738703}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2288495125}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1084250422}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1291711739}) 2024-04-19T00:56:20.465 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3828089953}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2777958875}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.409+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2598957970}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=100416479}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1870808221}) 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.466 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247191786}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1909477691}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=149377282}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1224514624}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.467 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=512497797}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3970224049}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=27444167}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=962197298}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277960413}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1091857630}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.468 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3415704893}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3527262914}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=306849785}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=680555447}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=617195392}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.413+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1415663124}) 2024-04-19T00:56:20.469 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242051593}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1038567615}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494233115}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2603759490}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4218644707}) 2024-04-19T00:56:20.470 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4075732122}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3540033758}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=320712498}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.417+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4240132837}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634394632}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091597590}) 2024-04-19T00:56:20.471 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199550788}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2416855271}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2307785112}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1558803903}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1909004179}) 2024-04-19T00:56:20.472 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=937228786}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2390328508}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161521791}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3325103441}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2774414558}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.473 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1446632059}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4059270561}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=860600111}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2164033959}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=956405191}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.421+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1982222061}) 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.474 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400067074}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889196515}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914342679}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4156118638}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2793489856}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.475 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2176336459}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=726299707}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3185402470}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=7039066}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2717049618}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2838714988}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.425+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.476 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3020346715}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2024840490}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3593701530}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=240374243}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1112602790}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3043749173}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1065227793}) 2024-04-19T00:56:20.477 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385125639}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167038003}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3615357639}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=650006458}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.478 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3699992855}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1088916926}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1983832301}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4041341870}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.429+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=263465348}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27} crc {logm=3777027153,mdsmap=2840952586,mgr=1318271769,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=6045082,mon_config_key=981625139,monmap=4080741109,osd_metadata=1368215097,osd_pg_creating=2844473681,osd_snap=3719795310}) 2024-04-19T00:56:20.479 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=2718044773,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=27,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=25} crc {logm=3715304165,mdsmap=2840952586,mgr=517009004,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=3140514009,osd_metadata=3305883572,osd_pg_creating=2844473681,osd_snap=372443142}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=29,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20} crc {logm=2004285843,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2844473681,osd_snap=4232670351}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=30,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22} crc {logm=1196375443,mdsmap=2840952586,mgr=1596547223,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=3890330159,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2844473681,osd_snap=994606158}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.480 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=28,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=25} crc {logm=1916170010,mdsmap=1401145748,mgr=2803061145,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=1815755404,monmap=2862618235,osd_metadata=3468572908,osd_pg_creating=2844473681,osd_snap=1601630049}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22} crc {logm=1724467374,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3277349063,mon_config_key=3698498464,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2844473681,osd_snap=3172397813}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=31,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=20} crc {logm=3046320650,mdsmap=2840952586,mgr=3047809850,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=2862618235,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=3446439194}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.433+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=28,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=23} crc {logm=716883607,mdsmap=2840952586,mgr=1596547223,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1185093534,mon_config_key=4212284759,monmap=4080741109,osd_metadata=438442333,osd_pg_creating=2844473681,osd_snap=824051975}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=2643321342}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=1682237188}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=399359466,osdmap=3850885046}) 2024-04-19T00:56:20.481 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=2544701289}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1312269949,osdmap=2612209252}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1312269949,osdmap=2516160200}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=550106635}) 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.482 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=9,osdmap=91} crc {osd_snap=2298423087,osdmap=4048552075}) 2024-04-19T00:56:20.483 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.437+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=1459653972,osdmap=1492016709}) 2024-04-19T00:56:20.494 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.494 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2091039817}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1431676731}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1088886745}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3619689895}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2466833287}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.495 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3517126077}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3643720241}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2687114430}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1754142419}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2533519369}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3019529475}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.441+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3389243939}) 2024-04-19T00:56:20.496 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2643629839}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1831056603}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3403827165}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3829436344}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3223024623}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4107017434}) 2024-04-19T00:56:20.497 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2131361129}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3029737682}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1528901553}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2193676245}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4083373487}) 2024-04-19T00:56:20.498 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1859646141}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.445+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=157746766}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.449+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.449+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3800991100}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.449+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2050204301}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=260223970}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2635469988}) 2024-04-19T00:56:20.499 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=257136337}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3577710456}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4261467336}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3369254426}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3597254170}) 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.500 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3996864396}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.453+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3683925240}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=935442140}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2435313640}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1923201119}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3052064737}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.501 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3561274146}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3845711428}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1795451996}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=362970758}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.461+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=609735909}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.502 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=52} crc {osdmap=3646122660}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=3072521010}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=53} crc {osdmap=562081107}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=55} crc {osdmap=1264697165}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=54} crc {osdmap=2539310142}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.503 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=52} crc {osdmap=3149722404}) 2024-04-19T00:56:20.504 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.504 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.504 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=52} crc {osdmap=1794828216}) 2024-04-19T00:56:20.504 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:56:20.504 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=54} crc {osdmap=2607624801}) 2024-04-19T00:56:20.504 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:56:20.481+0000 7efeef988640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=53} crc {osdmap=1883805532}) 2024-04-19T00:56:21.150 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:56:21.150 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient: get_monmap_and_config 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient: build_initial_monmap 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient: monmap: 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:21.280653+0000 2024-04-19T00:56:21.277 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:21.280653+0000 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:21.278 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 Processor -- start 2024-04-19T00:56:21.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 -- start start 2024-04-19T00:56:21.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient: init 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient: _add_conns ranks=[9,10,5,2,3,6,4,7,1,8,0] 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8dfc138cd0 0x7f8dfc1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): picked mon.noname-a con 0x7f8dfc138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8dfc0a5b80 0x7f8dfc1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): picked mon.noname-f con 0x7f8dfc0a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc135ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): picked mon.noname-h con 0x7f8dfc1358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f8dfc0a4410 con 0x7f8dfc138cd0 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8dfc139b30 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8dfc0a3dc0 con 0x7f8dfc1358b0 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.277+0000 7f8e024e3640 10 monclient(hunting): authenticate will time out at 1651.407998s 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8dfc138cd0 0x7f8dfc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8dfc138cd0 0x7f8dfc1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34208/0 (socket says 172.21.15.43:34208) 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 -- 172.21.15.43:0/3565852945 learned_addr learned my addr 172.21.15.43:0/3565852945 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): get_auth_request con 0x7f8dfc138cd0 auth_method 0 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_done global_id 64800 payload 274 2024-04-19T00:56:21.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8e00a80640 1 --2- 172.21.15.43:0/3565852945 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc135ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 -- 172.21.15.43:0/3565852945 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8dfc1358b0 msgr2=0x7f8dfc135ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 --2- 172.21.15.43:0/3565852945 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc135ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 -- 172.21.15.43:0/3565852945 >> v2:172.21.15.161:3300/0 conn(0x7f8dfc0a5b80 msgr2=0x7f8dfc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfb7fe640 1 --2- 172.21.15.43:0/3565852945 >> v2:172.21.15.161:3300/0 conn(0x7f8dfc0a5b80 0x7f8dfc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 --2- 172.21.15.43:0/3565852945 >> v2:172.21.15.161:3300/0 conn(0x7f8dfc0a5b80 0x7f8dfc1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 monclient: found mon.noname-a 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 -- 172.21.15.43:0/3565852945 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc0a40a0 con 0x7f8dfc138cd0 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfb7fe640 10 monclient: get_auth_request con 0x7f8dfc0a5b80 auth_method 0 2024-04-19T00:56:21.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfb7fe640 1 --2- 172.21.15.43:0/3565852945 >> v2:172.21.15.161:3300/0 conn(0x7f8dfc0a5b80 0x7f8dfc1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfbfff640 1 --2- 172.21.15.43:0/3565852945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc138cd0 0x7f8dfc1390b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0a7360 tx=0x7f8dec01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71058f84843aa4d server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 1 -- 172.21.15.43:0/3565852945 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dec0b0070 con 0x7f8dfc138cd0 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: dump: 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.289 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 1 -- 172.21.15.43:0/3565852945 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dec0aa810 con 0x7f8dfc138cd0 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 1 -- 172.21.15.43:0/3565852945 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dec0a9300 con 0x7f8dfc138cd0 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8e024e3640 5 monclient: authenticate success, global_id 64800 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8e024e3640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8e024e3640 4 set_mon_vals no callback set 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.281+0000 7f8dfaffd640 10 monclient: dump: 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.291 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: shutdown 2024-04-19T00:56:21.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/3565852945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc138cd0 msgr2=0x7f8dfc1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- 172.21.15.43:0/3565852945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc138cd0 0x7f8dfc1390b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc0a7360 tx=0x7f8dec01c380 comp rx=0 tx=0).stop 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/3565852945 shutdown_connections 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- 172.21.15.43:0/3565852945 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc135ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- 172.21.15.43:0/3565852945 >> v2:172.21.15.161:3300/0 conn(0x7f8dfc0a5b80 0x7f8dfc1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- 172.21.15.43:0/3565852945 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc138cd0 0x7f8dfc1390b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/3565852945 >> 172.21.15.43:0/3565852945 conn(0x7f8dfc0ab2f0 msgr2=0x7f8dfc0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/3565852945 shutdown_connections 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/3565852945 wait complete. 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: build_initial_monmap 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: monmap: 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:21.293 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 librados: starting msgr at 2024-04-19T00:56:21.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 librados: starting objecter 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 Processor -- start 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- start start 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 librados: setting wanted keys 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 librados: calling monclient init 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: init 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _add_conns ranks=[0,4,6,8,5,7,1,3,2] 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 0x7f8dfc12ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): picked mon.noname-a con 0x7f8dfc0a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc12bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): picked mon.noname-c con 0x7f8dfc1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8dfc138cd0 0x7f8dfc133510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): picked mon.noname-d con 0x7f8dfc138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8dfc0a2010 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f8dfc0a1e90 con 0x7f8dfc1358b0 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8dfc142aa0 con 0x7f8dfc138cd0 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient(hunting): authenticate will time out at 1651.413768s 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 0x7f8dfc12ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 0x7f8dfc12ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34212/0 (socket says 172.21.15.43:34212) 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 -- 172.21.15.43:0/2448168150 learned_addr learned my addr 172.21.15.43:0/2448168150 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): get_auth_request con 0x7f8dfc0a5b80 auth_method 0 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc12bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f8dfc1358b0 auth_method 0 2024-04-19T00:56:21.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8dfc138cd0 0x7f8dfc133510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): get_auth_request con 0x7f8dfc138cd0 auth_method 0 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e00a80640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient(hunting): handle_auth_done global_id 64809 payload 824 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 -- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8dfc138cd0 msgr2=0x7f8dfc133510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8dfc138cd0 0x7f8dfc133510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 -- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8dfc1358b0 msgr2=0x7f8dfc12bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc12bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 monclient: found mon.noname-a 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 -- 172.21.15.43:0/2448168150 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dfc0a3590 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8dfbfff640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 0x7f8dfc12ba70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0097c0 tx=0x7f8dec019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 client.?.objecter ms_handle_connect 0x7f8dfc0a5b80 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dec0b9070 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: dump: 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.299 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 5 monclient: authenticate success, global_id 64809 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dec0044b0 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _renew_subs 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dfc0a3040 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8dec0b0040 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 client.64809.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 10 monclient: dump: 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.301 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _renew_subs 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df9ffb640 4 set_mon_vals no callback set 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dfc09d460 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 librados: init done 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8dec004040 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dfc0693b0 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.285+0000 7f8df8ff9640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8de8038e10 0x7f8de803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f8dec0e2610 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8df8ff9640 10 client.64809.objecter ms_dispatch 0x7f8dfc000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8df8ff9640 3 client.64809.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8df8ff9640 3 client.64809.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8df8ff9640 20 client.64809.objecter dump_active .. 0 homeless 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8dfb7fe640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8de8038e10 0x7f8de803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8dfb7fe640 10 monclient: get_auth_request con 0x7f8de8038e10 auth_method 0 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8dfb7fe640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8de8038e10 0x7f8de803b2d0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f8df000ab60 tx=0x7f8df0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.289+0000 7f8df8ff9640 10 client.64809.objecter ms_handle_connect 0x7f8de8038e10 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.293+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8dec0160f0 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.293+0000 7f8df8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.293+0000 7f8df8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:21.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8e024e3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:56:21.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8e024e3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:56:21.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8e024e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8dfc076190 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8df8ff9640 1 -- 172.21.15.43:0/2448168150 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f8dec0aade0 con 0x7f8dfc0a5b80 2024-04-19T00:56:21.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8df8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:56:21.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.521+0000 7f8df8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:21.521 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":84,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"c","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:56:21.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 10 librados: watch_flush enter 2024-04-19T00:56:21.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 10 librados: watch_flush exit 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 20 client.64809.objecter shutdown clearing up homeless session... 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 10 client.64809.objecter successfully canceled tick 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8de8038e10 msgr2=0x7f8de803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8de8038e10 0x7f8de803b2d0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f8df000ab60 tx=0x7f8df0008040 comp rx=0 tx=0).stop 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 10 monclient: shutdown 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 msgr2=0x7f8dfc12ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 0x7f8dfc12ba70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8dec0097c0 tx=0x7f8dec019180 comp rx=0 tx=0).stop 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 shutdown_connections 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8de8038e10 0x7f8de803b2d0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8dfc138cd0 0x7f8dfc133510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8dfc1358b0 0x7f8dfc12bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 --2- 172.21.15.43:0/2448168150 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8dfc0a5b80 0x7f8dfc12ba70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 >> 172.21.15.43:0/2448168150 conn(0x7f8dfc0ab2f0 msgr2=0x7f8dfc0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:21.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 shutdown_connections 2024-04-19T00:56:21.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 -- 172.21.15.43:0/2448168150 wait complete. 2024-04-19T00:56:21.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.525+0000 7f8e024e3640 1 librados: shutdown 2024-04-19T00:56:21.540 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:56:21.540 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:56:21.540 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:56:21.540 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.657+0000 7f0ed9a75640 10 monclient: get_monmap_and_config 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.657+0000 7f0ed9a75640 10 monclient: build_initial_monmap 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.657+0000 7f0ed9a75640 10 monclient: monmap: 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:21.662260+0000 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:21.662260+0000 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:21.659 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:21.660 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 Processor -- start 2024-04-19T00:56:21.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 -- start start 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient: init 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient: _add_conns ranks=[5,0,7,4,3,2,6,1,8,10,9] 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ed40a1e00 0x7f0ed40a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): picked mon.noname-h con 0x7f0ed40a1e00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ed40a1190 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed413eac0 0x7f0ed4140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): picked mon.noname-i con 0x7f0ed413eac0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ed406c690 con 0x7f0ed40a1190 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0ed40735a0 con 0x7f0ed40a1e00 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0ed409fff0 con 0x7f0ed413eac0 2024-04-19T00:56:21.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 10 monclient(hunting): authenticate will time out at 1651.789486s 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0ed40a1190 0x7f0ed40a3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44960/0 (socket says 172.21.15.43:44960) 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 1 -- 172.21.15.43:0/3626589406 learned_addr learned my addr 172.21.15.43:0/3626589406 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ed40a1e00 0x7f0ed40a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request con 0x7f0ed40a1e00 auth_method 0 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed413eac0 0x7f0ed4140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): get_auth_request con 0x7f0ed413eac0 auth_method 0 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4230688920 0 0) 0x7f0ed406c690 con 0x7f0ed40a1190 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): _negotiate 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): my global_id is 64818 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ec0003530 con 0x7f0ed40a1190 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 624169064 0 0) 0x7f0ec0003530 con 0x7f0ed40a1190 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed413eac0 msgr2=0x7f0ed4140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed413eac0 0x7f0ed4140eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ed40a1e00 msgr2=0x7f0ed40a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ed40a1e00 0x7f0ed40a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed406c330 con 0x7f0ed40a1190 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f0ebc001970 con 0x7f0ed40a1190 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 5 monclient: authenticate success, global_id 64818 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: dump: 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.671 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 >> v1:172.21.15.43:6789/0 conn(0x7f0ed40a1190 legacy=0x7f0ed40a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient: _add_conns ranks=[6,0,2,1,5,3,4,7,8] 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ec0004210 0x7f0ec0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): picked mon.d con 0x7f0ec0004210 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec0004da0 0x7f0ec0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): picked mon.a con 0x7f0ec0004da0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec0009ab0 0x7f0ec000bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): picked mon.b con 0x7f0ec0009ab0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed40735a0 con 0x7f0ec0004da0 2024-04-19T00:56:21.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed409fff0 con 0x7f0ec0009ab0 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0ec000c850 con 0x7f0ec0004210 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed2ffd640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed9a75640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ec0004210 0x7f0ec0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec0004da0 0x7f0ec0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request con 0x7f0ec0004210 auth_method 0 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): get_auth_request con 0x7f0ec0004da0 auth_method 0 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec0009ab0 0x7f0ec000bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): get_auth_request con 0x7f0ec0009ab0 auth_method 0 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ecbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_done global_id 64818 payload 293 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 -- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec0009ab0 msgr2=0x7f0ec000bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0ec0009ab0 0x7f0ec000bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 -- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec0004da0 msgr2=0x7f0ec0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ec0004da0 0x7f0ec0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 monclient: found mon.d 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 1 -- 172.21.15.43:0/3626589406 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec0003530 con 0x7f0ec0004210 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.661+0000 7f0ed3fff640 1 -- 172.21.15.43:0/3626589406 reap_dead start 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ec0004210 0x7f0ec0004810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009ab0 tx=0x7f0ec401c9f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=b30fad31c82b6b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec40096b0 con 0x7f0ec0004210 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: dump: 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.676 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ec40b0400 con 0x7f0ec0004210 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 4 set_mon_vals no callback set 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 1 -- 172.21.15.43:0/3626589406 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec40b0a20 con 0x7f0ec0004210 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:21.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed2ffd640 10 monclient: dump: 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.678 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 10 monclient: shutdown 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 -- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ec0004210 msgr2=0x7f0ec0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ec0004210 0x7f0ec0004810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4009ab0 tx=0x7f0ec401c9f0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 -- 172.21.15.43:0/3626589406 shutdown_connections 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 --2- 172.21.15.43:0/3626589406 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ec0004210 0x7f0ec0004810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 -- 172.21.15.43:0/3626589406 >> 172.21.15.43:0/3626589406 conn(0x7f0ed40a84c0 msgr2=0x7f0ed40a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 -- 172.21.15.43:0/3626589406 shutdown_connections 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 -- 172.21.15.43:0/3626589406 wait complete. 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 10 monclient: build_initial_monmap 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 10 monclient: monmap: 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:21.679 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 librados: starting msgr at 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 librados: starting objecter 2024-04-19T00:56:21.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 Processor -- start 2024-04-19T00:56:21.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 -- start start 2024-04-19T00:56:21.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 librados: setting wanted keys 2024-04-19T00:56:21.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 1 librados: calling monclient init 2024-04-19T00:56:21.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.665+0000 7f0ed9a75640 10 monclient: init 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient: _add_conns ranks=[5,7,8,0,2,1,3,4,6] 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ec0004210 0x7f0ed4122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): picked mon.noname-h con 0x7f0ec0004210 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed4122e60 0x7f0ed41232a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): picked mon.noname-i con 0x7f0ed4122e60 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:21.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ed41d0c80 0x7f0ed41d1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): picked mon.noname-e con 0x7f0ed41d0c80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0ed40a40e0 con 0x7f0ec0004210 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0ed40a23f0 con 0x7f0ed4122e60 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0ed40a4260 con 0x7f0ed41d0c80 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): authenticate will time out at 1651.796405s 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ed41d0c80 0x7f0ed41d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ed41d0c80 0x7f0ed41d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:55378/0 (socket says 172.21.15.43:55378) 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 -- 172.21.15.43:0/1457501408 learned_addr learned my addr 172.21.15.43:0/1457501408 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): get_auth_request con 0x7f0ed41d0c80 auth_method 0 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed4122e60 0x7f0ed41232a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ec0004210 0x7f0ed4122920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 10 monclient(hunting): get_auth_request con 0x7f0ed4122e60 auth_method 0 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request con 0x7f0ec0004210 auth_method 0 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient(hunting): handle_auth_done global_id 114209 payload 824 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed4122e60 msgr2=0x7f0ed41232a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ed4122e60 0x7f0ed41232a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ec0004210 msgr2=0x7f0ed4122920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ec0004210 0x7f0ed4122920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 monclient: found mon.noname-e 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed4073270 con 0x7f0ed41d0c80 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed3fff640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ed41d0c80 0x7f0ed41d1100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ed406ffd0 tx=0x7f0ecc00f580 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ed41d0c80 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ecc019070 con 0x7f0ed41d0c80 2024-04-19T00:56:21.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: dump: 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.691 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ecc0039a0 con 0x7f0ed41d0c80 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 5 monclient: authenticate success, global_id 114209 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient: _renew_subs 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed4125b50 con 0x7f0ed41d0c80 2024-04-19T00:56:21.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ecc004030 con 0x7f0ed41d0c80 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: dump: 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.693 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 client.114209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient: _renew_subs 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed27fc640 4 set_mon_vals no callback set 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed4125870 con 0x7f0ed41d0c80 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 librados: init done 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0ecc01c070 con 0x7f0ed41d0c80 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 0x7f0ed413c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed41246a0 con 0x7f0ed41d1640 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 0x7f0ed413c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient: get_auth_request con 0x7f0ed41d1640 auth_method 0 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0eb4038e70 0x7f0eb403b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f0ecc049f20 con 0x7f0ed41d0c80 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 client.114209.objecter ms_dispatch 0x7f0ed4000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 3 client.114209.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 3 client.114209.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 20 client.114209.objecter dump_active .. 0 homeless 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_done global_id 64827 payload 824 2024-04-19T00:56:21.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ed4123b00 con 0x7f0ed41d1640 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0eb4038e70 0x7f0eb403b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 client.114209.objecter ms_handle_connect 0x7f0ed41d1640 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 0x7f0ed413c1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0004790 tx=0x7f0ec4019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 client.114209.objecter resend_mon_ops 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 10 monclient: get_auth_request con 0x7f0eb4038e70 auth_method 0 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec40be070 con 0x7f0ed41d1640 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: dump: 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.696 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ec40b6070 con 0x7f0ed41d1640 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 msgr2=0x7f0ed413c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 0x7f0ed413c1a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0004790 tx=0x7f0ec4019790 comp rx=0 tx=0).stop 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 msgr2=0x7f0ed413c1a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed41d1640 0x7f0ed413c1a0 secure :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0004790 tx=0x7f0ec4019790 comp rx=0 tx=0).stop 2024-04-19T00:56:21.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ecbfff640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0eb4038e70 0x7f0eb403b330 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f0ed409e0c0 tx=0x7f0ebc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.669+0000 7f0ed0ff9640 10 client.114209.objecter ms_handle_connect 0x7f0eb4038e70 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 0x7f0ed413caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed406d140 con 0x7f0ed413c6e0 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 0x7f0ed413caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient: get_auth_request con 0x7f0ed413c6e0 auth_method 0 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 10 monclient(hunting): handle_auth_done global_id 64836 payload 824 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 1 -- 172.21.15.43:0/1457501408 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ed4073050 con 0x7f0ed413c6e0 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed37fe640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 0x7f0ed413caf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ec401ca30 tx=0x7f0ec0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 client.114209.objecter ms_handle_connect 0x7f0ed413c6e0 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 client.114209.objecter resend_mon_ops 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ec40c4070 con 0x7f0ed413c6e0 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 monclient: dump: 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.699 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f0ec40bf040 con 0x7f0ed413c6e0 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 msgr2=0x7f0ed413caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 0x7f0ed413caf0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ec401ca30 tx=0x7f0ec0005840 comp rx=0 tx=0).stop 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 msgr2=0x7f0ed413caf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:21.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed0ff9640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0ed413c6e0 0x7f0ed413caf0 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0ec401ca30 tx=0x7f0ec0005840 comp rx=0 tx=0).stop 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 307949, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:21.701 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:21.702 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:21.703 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:21.704 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:21.705 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:21.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:21.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:21.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.717 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:21.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:21.719 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.720 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:21.721 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 10 librados: watch_flush enter 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 10 librados: watch_flush exit 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 20 client.114209.objecter shutdown clearing up homeless session... 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 10 client.114209.objecter successfully canceled tick 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0eb4038e70 msgr2=0x7f0eb403b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0eb4038e70 0x7f0eb403b330 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f0ed409e0c0 tx=0x7f0ebc009040 comp rx=0 tx=0).stop 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 10 monclient: shutdown 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:21.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ed41d0c80 msgr2=0x7f0ed41d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 --2- 172.21.15.43:0/1457501408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ed41d0c80 0x7f0ed41d1100 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ed406ffd0 tx=0x7f0ecc00f580 comp rx=0 tx=0).stop 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed3fff640 1 -- 172.21.15.43:0/1457501408 reap_dead start 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed3fff640 1 -- 172.21.15.43:0/1457501408 reap_dead start 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 shutdown_connections 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 >> 172.21.15.43:0/1457501408 conn(0x7f0ed40a84c0 msgr2=0x7f0ed40a6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 shutdown_connections 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 -- 172.21.15.43:0/1457501408 wait complete. 2024-04-19T00:56:21.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.677+0000 7f0ed9a75640 1 librados: shutdown 2024-04-19T00:56:21.724 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:21.724 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:56:21.725 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:56:21.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.841+0000 7f5a77969640 10 monclient: get_monmap_and_config 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.841+0000 7f5a77969640 10 monclient: build_initial_monmap 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.841+0000 7f5a77969640 10 monclient: monmap: 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:21.846259+0000 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:21.846259+0000 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:21.843 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:21.844 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:21.844 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:21.844 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:21.844 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:21.844 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:21.844 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 Processor -- start 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 -- start start 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient: init 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient: _add_conns ranks=[2,5,8,3,10,1,7,0,4,9,6] 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 0x7f5a701390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): picked mon.noname-b con 0x7f5a70138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5a700a5b80 0x7f5a701395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): picked mon.noname-h con 0x7f5a700a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5a701358b0 0x7f5a70135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): picked mon.noname-e con 0x7f5a701358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a700a4410 con 0x7f5a70138cd0 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5a70139b30 con 0x7f5a700a5b80 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f5a700a3dc0 con 0x7f5a701358b0 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a77969640 10 monclient(hunting): authenticate will time out at 1651.973441s 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 0x7f5a701390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5a701358b0 0x7f5a70135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 0x7f5a701390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:58194/0 (socket says 172.21.15.43:58194) 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 1 -- 172.21.15.43:0/1520983770 learned_addr learned my addr 172.21.15.43:0/1520983770 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): get_auth_request con 0x7f5a701358b0 auth_method 0 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): get_auth_request con 0x7f5a70138cd0 auth_method 0 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a74f04640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5a700a5b80 0x7f5a701395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a74f04640 10 monclient(hunting): get_auth_request con 0x7f5a700a5b80 auth_method 0 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a74f04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a74f04640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a74f04640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient(hunting): handle_auth_done global_id 114218 payload 274 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 -- 172.21.15.43:0/1520983770 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5a700a5b80 msgr2=0x7f5a701395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5a700a5b80 0x7f5a701395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 -- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 msgr2=0x7f5a701390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 0x7f5a701390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 monclient: found mon.noname-e 2024-04-19T00:56:21.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:21.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75f06640 1 -- 172.21.15.43:0/1520983770 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700a40a0 con 0x7f5a701358b0 2024-04-19T00:56:21.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.845+0000 7f5a75705640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 0x7f5a701390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75f06640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5a701358b0 0x7f5a70135d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009dc0 tx=0x7f5a6801c7f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=be6b25275efec46e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 1 -- 172.21.15.43:0/1520983770 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a680a6960 con 0x7f5a701358b0 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: dump: 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.857 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 1 -- 172.21.15.43:0/1520983770 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a680a6b20 con 0x7f5a701358b0 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 5 monclient: authenticate success, global_id 114218 2024-04-19T00:56:21.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 4 set_mon_vals no callback set 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 1 -- 172.21.15.43:0/1520983770 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a680a6d10 con 0x7f5a701358b0 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a67fff640 10 monclient: dump: 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.859 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: shutdown 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5a701358b0 msgr2=0x7f5a70135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5a701358b0 0x7f5a70135d30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5a68009dc0 tx=0x7f5a6801c7f0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/1520983770 shutdown_connections 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5a701358b0 0x7f5a70135d30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5a700a5b80 0x7f5a701395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- 172.21.15.43:0/1520983770 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70138cd0 0x7f5a701390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/1520983770 >> 172.21.15.43:0/1520983770 conn(0x7f5a700ab2f0 msgr2=0x7f5a700ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/1520983770 shutdown_connections 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/1520983770 wait complete. 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: build_initial_monmap 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: monmap: 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:21.861 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 librados: starting msgr at 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 librados: starting objecter 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 Processor -- start 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- start start 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 librados: setting wanted keys 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 librados: calling monclient init 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: init 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: _add_conns ranks=[1,0,3,5,2,6,7,4,8] 2024-04-19T00:56:21.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a700a5b80 0x7f5a701d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): picked mon.noname-f con 0x7f5a700a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a701358b0 0x7f5a700a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): picked mon.noname-a con 0x7f5a701358b0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a70138cd0 0x7f5a700a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): picked mon.noname-g con 0x7f5a70138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700a2010 con 0x7f5a701358b0 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a700a1e90 con 0x7f5a700a5b80 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5a70142aa0 con 0x7f5a70138cd0 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient(hunting): authenticate will time out at 1651.978651s 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a701358b0 0x7f5a700a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a701358b0 0x7f5a700a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34250/0 (socket says 172.21.15.43:34250) 2024-04-19T00:56:21.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 -- 172.21.15.43:0/633570088 learned_addr learned my addr 172.21.15.43:0/633570088 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): get_auth_request con 0x7f5a701358b0 auth_method 0 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75f06640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a70138cd0 0x7f5a700a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75705640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a700a5b80 0x7f5a701d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75f06640 10 monclient(hunting): get_auth_request con 0x7f5a70138cd0 auth_method 0 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75f06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75f06640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75f06640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75705640 10 monclient(hunting): get_auth_request con 0x7f5a700a5b80 auth_method 0 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75705640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75705640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a75705640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient(hunting): handle_auth_done global_id 64845 payload 824 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a70138cd0 msgr2=0x7f5a700a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5a70138cd0 0x7f5a700a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a700a5b80 msgr2=0x7f5a701d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a700a5b80 0x7f5a701d0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient: _finish_hunting 0 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 monclient: found mon.noname-a 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70135240 con 0x7f5a701358b0 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a74f04640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a701358b0 0x7f5a700a0ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000d800 tx=0x7f5a6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 client.?.objecter ms_handle_connect 0x7f5a701358b0 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:21.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a600044a0 con 0x7f5a701358b0 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: dump: 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.866 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6000b6c0 con 0x7f5a701358b0 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a6000bce0 con 0x7f5a701358b0 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a65ffb640 10 monclient: dump: 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.867 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a677fe640 4 set_mon_vals no callback set 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 5 monclient: authenticate success, global_id 64845 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: _renew_subs 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a7009ccc0 con 0x7f5a701358b0 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 client.64845.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: _renew_subs 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a7009cfa0 con 0x7f5a701358b0 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.849+0000 7f5a77969640 1 librados: init done 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a77969640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a77969640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 0x7f5a70133650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a7006d140 con 0x7f5a7009bd60 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5a6000b870 con 0x7f5a701358b0 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 0x7f5a70133650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient: get_auth_request con 0x7f5a7009bd60 auth_method 0 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5a54038e10 0x7f5a5403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f5a6000f040 con 0x7f5a701358b0 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 client.64845.objecter ms_dispatch 0x7f5a70000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 3 client.64845.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 3 client.64845.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 10 monclient(hunting): handle_auth_done global_id 74405 payload 824 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a7009d690 con 0x7f5a7009bd60 2024-04-19T00:56:21.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 20 client.64845.objecter dump_active .. 0 homeless 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 client.64845.objecter ms_handle_connect 0x7f5a7009bd60 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 client.64845.objecter resend_mon_ops 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75705640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 0x7f5a70133650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0114c0 tx=0x7f5a6c011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75f06640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5a54038e10 0x7f5a5403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75f06640 10 monclient: get_auth_request con 0x7f5a54038e10 auth_method 0 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a6c015ae0 con 0x7f5a7009bd60 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 monclient: dump: 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.871 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a6c015670 con 0x7f5a7009bd60 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 msgr2=0x7f5a70133650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 0x7f5a70133650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0114c0 tx=0x7f5a6c011980 comp rx=0 tx=0).stop 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 msgr2=0x7f5a70133650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a7009bd60 0x7f5a70133650 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0114c0 tx=0x7f5a6c011980 comp rx=0 tx=0).stop 2024-04-19T00:56:21.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a75f06640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5a54038e10 0x7f5a5403b2d0 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f5a680af000 tx=0x7f5a680a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.853+0000 7f5a65ffb640 10 client.64845.objecter ms_handle_connect 0x7f5a54038e10 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.857+0000 7f5a77969640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.857+0000 7f5a77969640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 0x7f5a70133fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5a701344e0 con 0x7f5a70133b90 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 0x7f5a70133fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient: get_auth_request con 0x7f5a70133b90 auth_method 0 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:21.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 10 monclient(hunting): handle_auth_done global_id 74414 payload 824 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 1 -- 172.21.15.43:0/633570088 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a700693b0 con 0x7f5a70133b90 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 client.64845.objecter ms_handle_connect 0x7f5a70133b90 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 client.64845.objecter resend_mon_ops 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75705640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 0x7f5a70133fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0114c0 tx=0x7f5a6c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5a6c0066c0 con 0x7f5a70133b90 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 monclient: dump: 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:21.875 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f5a6c026070 con 0x7f5a70133b90 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:21.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 msgr2=0x7f5a70133fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 0x7f5a70133fa0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0114c0 tx=0x7f5a6c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 msgr2=0x7f5a70133fa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a65ffb640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5a70133b90 0x7f5a70133fa0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c0114c0 tx=0x7f5a6c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 161482, 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:21.877 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:21.878 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:21.879 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:21.880 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:21.881 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.882 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.883 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:21.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:21.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:21.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:21.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:21.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:21.897 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 10 librados: watch_flush enter 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 10 librados: watch_flush exit 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 20 client.64845.objecter shutdown clearing up homeless session... 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 10 client.64845.objecter successfully canceled tick 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5a54038e10 msgr2=0x7f5a5403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5a54038e10 0x7f5a5403b2d0 secure :-1 s=READY pgs=3532 cs=0 l=1 rev1=1 crypto rx=0x7f5a680af000 tx=0x7f5a680a7000 comp rx=0 tx=0).stop 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 10 monclient: shutdown 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a701358b0 msgr2=0x7f5a700a0ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 --2- 172.21.15.43:0/633570088 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5a701358b0 0x7f5a700a0ab0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a6000d800 tx=0x7f5a6000dcc0 comp rx=0 tx=0).stop 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75f06640 1 -- 172.21.15.43:0/633570088 reap_dead start 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a75f06640 1 -- 172.21.15.43:0/633570088 reap_dead start 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 shutdown_connections 2024-04-19T00:56:21.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 >> 172.21.15.43:0/633570088 conn(0x7f5a700ab2f0 msgr2=0x7f5a700a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:21.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 shutdown_connections 2024-04-19T00:56:21.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 -- 172.21.15.43:0/633570088 wait complete. 2024-04-19T00:56:21.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:21.861+0000 7f5a77969640 1 librados: shutdown 2024-04-19T00:56:21.900 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:21.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:56:21.900 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:56:22.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient: get_monmap_and_config 2024-04-19T00:56:22.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.016 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient: monmap: 2024-04-19T00:56:22.016 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:22.019688+0000 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:22.019688+0000 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:22.017 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:22.018 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 Processor -- start 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 -- start start 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient: init 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient: _add_conns ranks=[1,3,2,8,4,6,7,5,9,10,0] 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7faebc138a50 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faebc139540 0x7faebc13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): picked mon.noname-g con 0x7faebc139540 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 0x7faebc140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): picked mon.noname-b con 0x7faebc0a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faebc0a0d40 con 0x7faebc138a50 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faebc0a1020 con 0x7faebc0a5bb0 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7faebc0a0780 con 0x7faebc139540 2024-04-19T00:56:22.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec35ec640 10 monclient(hunting): authenticate will time out at 1652.146823s 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 0x7faebc140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 0x7faebc140530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:58218/0 (socket says 172.21.15.43:58218) 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 -- 172.21.15.43:0/3030460025 learned_addr learned my addr 172.21.15.43:0/3030460025 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): get_auth_request con 0x7faebc0a5bb0 auth_method 0 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec0b87640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faebc139540 0x7faebc13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient(hunting): handle_auth_done global_id 74423 payload 274 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 -- 172.21.15.43:0/3030460025 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faebc139540 msgr2=0x7faebc13dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faebc139540 0x7faebc13dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 -- 172.21.15.43:0/3030460025 >> v1:172.21.15.161:6789/0 conn(0x7faebc138a50 legacy=0x7faebc138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 monclient: found mon.noname-b 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec1b89640 1 -- 172.21.15.43:0/3030460025 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc0a0a60 con 0x7faebc0a5bb0 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec0b87640 10 monclient: get_auth_request con 0x7faebc139540 auth_method 0 2024-04-19T00:56:22.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.017+0000 7faec0b87640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faebc139540 0x7faebc13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec1b89640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 0x7faebc140530 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009e80 tx=0x7faeb401c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d6c42822768828 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 1 -- 172.21.15.43:0/3030460025 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeb40090d0 con 0x7faebc0a5bb0 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: dump: 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.028 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 5 monclient: authenticate success, global_id 74423 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 1 -- 172.21.15.43:0/3030460025 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb40a5a70 con 0x7faebc0a5bb0 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 4 set_mon_vals no callback set 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 1 -- 172.21.15.43:0/3030460025 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeb40a5c60 con 0x7faebc0a5bb0 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faeb3fff640 10 monclient: dump: 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.030 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: shutdown 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- 172.21.15.43:0/3030460025 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 msgr2=0x7faebc140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 0x7faebc140530 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7faeb4009e80 tx=0x7faeb401c910 comp rx=0 tx=0).stop 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- 172.21.15.43:0/3030460025 shutdown_connections 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faebc0a5bb0 0x7faebc140530 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 --2- 172.21.15.43:0/3030460025 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7faebc139540 0x7faebc13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- 172.21.15.43:0/3030460025 >> 172.21.15.43:0/3030460025 conn(0x7faebc0a7cb0 msgr2=0x7faebc0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- 172.21.15.43:0/3030460025 shutdown_connections 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- 172.21.15.43:0/3030460025 wait complete. 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: monmap: 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:22.032 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 librados: starting msgr at 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 librados: starting objecter 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 Processor -- start 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- start start 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 librados: setting wanted keys 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 librados: calling monclient init 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: init 2024-04-19T00:56:22.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient: _add_conns ranks=[5,4,7,0,1,2,3,8,6] 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7faebc0a5bb0 0x7faebc137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): picked mon.noname-h con 0x7faebc0a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc138a50 0x7faebc13c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): picked mon.noname-c con 0x7faebc138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faebc139540 0x7faebc13cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): picked mon.noname-i con 0x7faebc139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7faebc143620 con 0x7faebc138a50 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7faebc1434a0 con 0x7faebc0a5bb0 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7faebc1437a0 con 0x7faebc139540 2024-04-19T00:56:22.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec35ec640 10 monclient(hunting): authenticate will time out at 1652.151819s 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec0b87640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc138a50 0x7faebc13c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.021+0000 7faec0b87640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc138a50 0x7faebc13c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:53832/0 (socket says 172.21.15.43:53832) 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 -- 172.21.15.43:0/23233498 learned_addr learned my addr 172.21.15.43:0/23233498 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): get_auth_request con 0x7faebc138a50 auth_method 0 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7faebc0a5bb0 0x7faebc137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1b89640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faebc139540 0x7faebc13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): get_auth_request con 0x7faebc0a5bb0 auth_method 0 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.035 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient(hunting): handle_auth_done global_id 64624 payload 824 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faebc139540 msgr2=0x7faebc13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faebc139540 0x7faebc13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7faebc0a5bb0 msgr2=0x7faebc137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7faebc0a5bb0 0x7faebc137860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 monclient: found mon.noname-c 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faebc09fc90 con 0x7faebc138a50 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1b89640 10 monclient: get_auth_request con 0x7faebc139540 auth_method 0 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1b89640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faebc139540 0x7faebc13cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec0b87640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc138a50 0x7faebc13c600 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faeb800c820 tx=0x7faeb800cce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 client.?.objecter ms_handle_connect 0x7faebc138a50 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeb8015070 con 0x7faebc138a50 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: dump: 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.037 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 5 monclient: authenticate success, global_id 64624 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeb800ce10 con 0x7faebc138a50 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 monclient: _renew_subs 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faebc09ee60 con 0x7faebc138a50 2024-04-19T00:56:22.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeb8007ca0 con 0x7faebc138a50 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb37fe640 4 set_mon_vals no callback set 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: dump: 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.039 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 client.64624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 monclient: _renew_subs 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7faebc122210 con 0x7faebc138a50 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 1 librados: init done 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7faeb8018070 con 0x7faebc138a50 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 0x7faebc02c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7faebc06d140 con 0x7faebc121970 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 0x7faebc02c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faea0038d40 0x7faea003b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient: get_auth_request con 0x7faebc121970 auth_method 0 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7faeb8049660 con 0x7faebc138a50 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 client.64624.objecter ms_dispatch 0x7faebc000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 3 client.64624.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 3 client.64624.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 20 client.64624.objecter dump_active .. 0 homeless 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 10 monclient(hunting): handle_auth_done global_id 64633 payload 824 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faebc13e2d0 con 0x7faebc121970 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1b89640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faea0038d40 0x7faea003b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1b89640 10 monclient: get_auth_request con 0x7faea0038d40 auth_method 0 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 client.64624.objecter ms_handle_connect 0x7faebc121970 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 client.64624.objecter resend_mon_ops 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1388640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 0x7faebc02c450 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e450 tx=0x7faeac00e910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeac0076b0 con 0x7faebc121970 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: dump: 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.042 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faeac007240 con 0x7faebc121970 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 msgr2=0x7faebc02c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 0x7faebc02c450 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e450 tx=0x7faeac00e910 comp rx=0 tx=0).stop 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 msgr2=0x7faebc02c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc121970 0x7faebc02c450 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faec1b89640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faea0038d40 0x7faea003b200 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7faeb40ae000 tx=0x7faeb40a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.025+0000 7faeb1ffb640 10 client.64624.objecter ms_handle_connect 0x7faea0038d40 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 0x7faebc02c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7faebc09e190 con 0x7faebc1c8630 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 0x7faebc02c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient: get_auth_request con 0x7faebc1c8630 auth_method 0 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 10 monclient(hunting): handle_auth_done global_id 64642 payload 824 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 1 -- 172.21.15.43:0/23233498 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faebc076190 con 0x7faebc1c8630 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1388640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 0x7faebc02c990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e450 tx=0x7faeac003770 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 client.64624.objecter ms_handle_connect 0x7faebc1c8630 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 client.64624.objecter resend_mon_ops 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7faeac0288a0 con 0x7faebc1c8630 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 monclient: dump: 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.047 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7faeac028430 con 0x7faebc1c8630 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 msgr2=0x7faebc02c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 0x7faebc02c990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e450 tx=0x7faeac003770 comp rx=0 tx=0).stop 2024-04-19T00:56:22.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 msgr2=0x7faebc02c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faeb1ffb640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc1c8630 0x7faebc02c990 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e450 tx=0x7faeac003770 comp rx=0 tx=0).stop 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 308305, 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:22.049 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.050 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.051 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.052 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.053 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.054 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.055 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.056 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:22.057 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.059 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:22.060 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:22.064 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.065 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.067 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:22.068 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 10 librados: watch_flush enter 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 10 librados: watch_flush exit 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 20 client.64624.objecter shutdown clearing up homeless session... 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 10 client.64624.objecter successfully canceled tick 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faea0038d40 msgr2=0x7faea003b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7faea0038d40 0x7faea003b200 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7faeb40ae000 tx=0x7faeb40a7070 comp rx=0 tx=0).stop 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 10 monclient: shutdown 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc138a50 msgr2=0x7faebc13c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 --2- 172.21.15.43:0/23233498 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faebc138a50 0x7faebc13c600 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faeb800c820 tx=0x7faeb800cce0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1b89640 1 -- 172.21.15.43:0/23233498 reap_dead start 2024-04-19T00:56:22.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec1b89640 1 -- 172.21.15.43:0/23233498 reap_dead start 2024-04-19T00:56:22.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 shutdown_connections 2024-04-19T00:56:22.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 >> 172.21.15.43:0/23233498 conn(0x7faebc0a7cb0 msgr2=0x7faebc0a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 shutdown_connections 2024-04-19T00:56:22.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 -- 172.21.15.43:0/23233498 wait complete. 2024-04-19T00:56:22.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.033+0000 7faec35ec640 1 librados: shutdown 2024-04-19T00:56:22.071 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:22.071 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:56:22.071 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.189+0000 7fe9d3f12640 10 monclient: get_monmap_and_config 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.189+0000 7fe9d3f12640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.189+0000 7fe9d3f12640 10 monclient: monmap: 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:22.193557+0000 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:22.193557+0000 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.190 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:22.191 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.189+0000 7fe9d3f12640 1 Processor -- start 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 -- start start 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient: init 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient: _add_conns ranks=[6,7,2,10,8,9,4,1,5,0,3] 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 0x7fe9cc136f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): picked mon.noname-d con 0x7fe9cc0a2ac0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe9cc0a2ea0 0x7fe9cc0a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): picked mon.noname-i con 0x7fe9cc0a2ea0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:22.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc0a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc0a32c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc071890 con 0x7fe9cc0a32c0 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fe9cc071cb0 con 0x7fe9cc0a2ac0 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe9cc072fd0 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d3f12640 10 monclient(hunting): authenticate will time out at 1652.320755s 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 0x7fe9cc136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 0x7fe9cc136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42726/0 (socket says 172.21.15.43:42726) 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 -- 172.21.15.43:0/2962415288 learned_addr learned my addr 172.21.15.43:0/2962415288 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): get_auth_request con 0x7fe9cc0a2ac0 auth_method 0 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc0a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): get_auth_request con 0x7fe9cc0a32c0 auth_method 0 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d24af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d14ad640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe9cc0a2ea0 0x7fe9cc0a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_done global_id 74400 payload 274 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 -- 172.21.15.43:0/2962415288 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe9cc0a2ea0 msgr2=0x7fe9cc0a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe9cc0a2ea0 0x7fe9cc0a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 -- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 msgr2=0x7fe9cc0a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc0a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 monclient: found mon.noname-d 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d1cae640 1 -- 172.21.15.43:0/2962415288 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc0732b0 con 0x7fe9cc0a2ac0 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d14ad640 10 monclient: get_auth_request con 0x7fe9cc0a2ea0 auth_method 0 2024-04-19T00:56:22.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.193+0000 7fe9d14ad640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe9cc0a2ea0 0x7fe9cc0a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 0x7fe9cc136f40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098a0 tx=0x7fe9bc01c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=d0bcc2d9ee49479e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 1 -- 172.21.15.43:0/2962415288 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9bc0b0070 con 0x7fe9cc0a2ac0 2024-04-19T00:56:22.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: dump: 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.204 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 1 -- 172.21.15.43:0/2962415288 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0aa810 con 0x7fe9cc0a2ac0 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 1 -- 172.21.15.43:0/2962415288 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9bc0a9300 con 0x7fe9cc0a2ac0 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 5 monclient: authenticate success, global_id 74400 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 4 set_mon_vals no callback set 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d0cac640 10 monclient: dump: 2024-04-19T00:56:22.205 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.206 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: shutdown 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 msgr2=0x7fe9cc136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 0x7fe9cc136f40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098a0 tx=0x7fe9bc01c440 comp rx=0 tx=0).stop 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- 172.21.15.43:0/2962415288 shutdown_connections 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc0a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe9cc0a2ea0 0x7fe9cc0a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/2962415288 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc0a2ac0 0x7fe9cc136f40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- 172.21.15.43:0/2962415288 >> 172.21.15.43:0/2962415288 conn(0x7fe9cc0a62f0 msgr2=0x7fe9cc06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- 172.21.15.43:0/2962415288 shutdown_connections 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- 172.21.15.43:0/2962415288 wait complete. 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: monmap: 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.207 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 librados: starting msgr at 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 librados: starting objecter 2024-04-19T00:56:22.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 Processor -- start 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- start start 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 librados: setting wanted keys 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 librados: calling monclient init 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: init 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient: _add_conns ranks=[1,4,2,0,7,5,3,8,6] 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9cc0a2ac0 0x7fe9cc1ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): picked mon.noname-f con 0x7fe9cc0a2ac0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9cc0a2ea0 0x7fe9cc1cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): picked mon.noname-c con 0x7fe9cc0a2ea0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc1cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): picked mon.noname-b con 0x7fe9cc0a32c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe9cc140690 con 0x7fe9cc0a2ac0 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe9cc140810 con 0x7fe9cc0a32c0 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe9cc140990 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d3f12640 10 monclient(hunting): authenticate will time out at 1652.327819s 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d14ad640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9cc0a2ea0 0x7fe9cc1cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.197+0000 7fe9d24af640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc1cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9cc0a2ea0 0x7fe9cc1cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:53856/0 (socket says 172.21.15.43:53856) 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 -- 172.21.15.43:0/1783812277 learned_addr learned my addr 172.21.15.43:0/1783812277 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): get_auth_request con 0x7fe9cc0a2ea0 auth_method 0 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): get_auth_request con 0x7fe9cc0a32c0 auth_method 0 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9cc0a2ac0 0x7fe9cc1ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): get_auth_request con 0x7fe9cc0a2ac0 auth_method 0 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient(hunting): handle_auth_done global_id 64651 payload 824 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 msgr2=0x7fe9cc1cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc1cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9cc0a2ac0 msgr2=0x7fe9cc1ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe9cc0a2ac0 0x7fe9cc1ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 monclient: found mon.noname-c 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9cc09de90 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe9cc0a32c0 0x7fe9cc1cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d14ad640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9cc0a2ea0 0x7fe9cc1cee60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800cf80 tx=0x7fe9c800a7d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 client.?.objecter ms_handle_connect 0x7fe9cc0a2ea0 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9c801a070 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: dump: 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.213 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9c800dab0 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9c800dc60 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: dump: 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.214 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c37fe640 4 set_mon_vals no callback set 2024-04-19T00:56:22.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 5 monclient: authenticate success, global_id 64651 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 monclient: _renew_subs 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9cc09e1d0 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 client.64651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 monclient: _renew_subs 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9cc09d150 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 1 librados: init done 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 0x7fe9cc12c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fe9cc137690 con 0x7fe9cc09f010 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 0x7fe9cc12c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient: get_auth_request con 0x7fe9cc09f010 auth_method 0 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe9c800d640 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_done global_id 74409 payload 824 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9cc09e620 con 0x7fe9cc09f010 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9b0038e10 0x7fe9b003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7fe9c8049b60 con 0x7fe9cc0a2ea0 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 client.64651.objecter ms_dispatch 0x7fe9cc000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 3 client.64651.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:22.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 3 client.64651.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 20 client.64651.objecter dump_active .. 0 homeless 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 0x7fe9cc12c7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009870 tx=0x7fe9bc0191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 client.64651.objecter ms_handle_connect 0x7fe9cc09f010 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 client.64651.objecter resend_mon_ops 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9bc0b0050 con 0x7fe9cc09f010 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: dump: 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.218 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9b0038e10 0x7fe9b003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 10 monclient: get_auth_request con 0x7fe9b0038e10 auth_method 0 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe9bc0b1040 con 0x7fe9cc09f010 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:22.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 msgr2=0x7fe9cc12c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 0x7fe9cc12c7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009870 tx=0x7fe9bc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 msgr2=0x7fe9cc12c7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc09f010 0x7fe9cc12c7b0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009870 tx=0x7fe9bc0191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9d24af640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9b0038e10 0x7fe9b003b2d0 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc09ccc0 tx=0x7fe9c4009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.201+0000 7fe9c27fc640 10 client.64651.objecter ms_handle_connect 0x7fe9b0038e10 2024-04-19T00:56:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 0x7fe9cc12d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fe9cc12d640 con 0x7fe9cc12ccf0 2024-04-19T00:56:22.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 0x7fe9cc12d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient: get_auth_request con 0x7fe9cc12ccf0 auth_method 0 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 10 monclient(hunting): handle_auth_done global_id 74418 payload 824 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 1 -- 172.21.15.43:0/1783812277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9cc076440 con 0x7fe9cc12ccf0 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d1cae640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 0x7fe9cc12d100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc01c480 tx=0x7fe9bc0a8000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 client.64651.objecter ms_handle_connect 0x7fe9cc12ccf0 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 client.64651.objecter resend_mon_ops 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe9bc0bddd0 con 0x7fe9cc12ccf0 2024-04-19T00:56:22.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 monclient: dump: 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.223 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe9bc0c1070 con 0x7fe9cc12ccf0 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 msgr2=0x7fe9cc12d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 0x7fe9cc12d100 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc01c480 tx=0x7fe9bc0a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 msgr2=0x7fe9cc12d100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9c27fc640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fe9cc12ccf0 0x7fe9cc12d100 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc01c480 tx=0x7fe9bc0a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.224 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 161822, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.225 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.226 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.227 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:22.228 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.229 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.231 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:22.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.234 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.236 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:22.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.239 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.240 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.241 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.242 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:22.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.244 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.245 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:22.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.246 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:22.246 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:22.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 10 librados: watch_flush enter 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 10 librados: watch_flush exit 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 20 client.64651.objecter shutdown clearing up homeless session... 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 10 client.64651.objecter successfully canceled tick 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9b0038e10 msgr2=0x7fe9b003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe9b0038e10 0x7fe9b003b2d0 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc09ccc0 tx=0x7fe9c4009190 comp rx=0 tx=0).stop 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 10 monclient: shutdown 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9cc0a2ea0 msgr2=0x7fe9cc1cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 --2- 172.21.15.43:0/1783812277 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe9cc0a2ea0 0x7fe9cc1cee60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800cf80 tx=0x7fe9c800a7d0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d24af640 1 -- 172.21.15.43:0/1783812277 reap_dead start 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d24af640 1 -- 172.21.15.43:0/1783812277 reap_dead start 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 shutdown_connections 2024-04-19T00:56:22.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 >> 172.21.15.43:0/1783812277 conn(0x7fe9cc0a62f0 msgr2=0x7fe9cc0a8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 shutdown_connections 2024-04-19T00:56:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 -- 172.21.15.43:0/1783812277 wait complete. 2024-04-19T00:56:22.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.209+0000 7fe9d3f12640 1 librados: shutdown 2024-04-19T00:56:22.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:22.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:56:22.249 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:56:22.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient: get_monmap_and_config 2024-04-19T00:56:22.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient: monmap: 2024-04-19T00:56:22.367 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:22.370646+0000 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:22.370646+0000 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:22.368 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:22.369 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 Processor -- start 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 -- start start 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient: init 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient: _add_conns ranks=[7,6,4,1,5,9,3,0,8,10,2] 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faff00729d0 0x7faff0072db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): picked mon.noname-i con 0x7faff00729d0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 0x7faff00a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): picked mon.noname-d con 0x7faff0139100 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faff00a4fe0 0x7faff00a5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): picked mon.noname-c con 0x7faff00a4fe0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7faff006f5e0 con 0x7faff00a4fe0 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7faff006f830 con 0x7faff0139100 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7faff006f270 con 0x7faff00729d0 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff62c9640 10 monclient(hunting): authenticate will time out at 1652.497770s 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 0x7faff00a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faff00a4fe0 0x7faff00a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 0x7faff00a4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42746/0 (socket says 172.21.15.43:42746) 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faff00a4fe0 0x7faff00a5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:53864/0 (socket says 172.21.15.43:53864) 2024-04-19T00:56:22.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faff00729d0 0x7faff0072db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 -- 172.21.15.43:0/3316629334 learned_addr learned my addr 172.21.15.43:0/3316629334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): get_auth_request con 0x7faff00729d0 auth_method 0 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): get_auth_request con 0x7faff00a4fe0 auth_method 0 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): get_auth_request con 0x7faff0139100 auth_method 0 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7faff4866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient(hunting): handle_auth_done global_id 74427 payload 274 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 -- 172.21.15.43:0/3316629334 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faff00729d0 msgr2=0x7faff0072db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faff00729d0 0x7faff0072db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 -- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faff00a4fe0 msgr2=0x7faff00a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faff00a4fe0 0x7faff00a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 monclient: found mon.noname-d 2024-04-19T00:56:22.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.369+0000 7fafef7fe640 1 -- 172.21.15.43:0/3316629334 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff00732f0 con 0x7faff0139100 2024-04-19T00:56:22.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafef7fe640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 0x7faff00a4aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fafd80098a0 tx=0x7fafd801c4d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=f7d6e80e9ce2500c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 1 -- 172.21.15.43:0/3316629334 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafd80b0070 con 0x7faff0139100 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: dump: 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.381 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 1 -- 172.21.15.43:0/3316629334 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd80aa810 con 0x7faff0139100 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 1 -- 172.21.15.43:0/3316629334 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafd80a9300 con 0x7faff0139100 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7fafeeffd640 10 monclient: dump: 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.382 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 5 monclient: authenticate success, global_id 74427 2024-04-19T00:56:22.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 4 set_mon_vals no callback set 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: shutdown 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 msgr2=0x7faff00a4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 0x7faff00a4aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fafd80098a0 tx=0x7fafd801c4d0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- 172.21.15.43:0/3316629334 shutdown_connections 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff0139100 0x7faff00a4aa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7faff00a4fe0 0x7faff00a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- 172.21.15.43:0/3316629334 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7faff00729d0 0x7faff0072db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- 172.21.15.43:0/3316629334 >> 172.21.15.43:0/3316629334 conn(0x7faff00a6410 msgr2=0x7faff006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- 172.21.15.43:0/3316629334 shutdown_connections 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- 172.21.15.43:0/3316629334 wait complete. 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: monmap: 2024-04-19T00:56:22.384 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.385 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 librados: starting msgr at 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 librados: starting objecter 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 Processor -- start 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- start start 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 librados: setting wanted keys 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 librados: calling monclient init 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: init 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient: _add_conns ranks=[2,6,0,1,5,4,3,7,8] 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00729d0 0x7faff00a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): picked mon.noname-b con 0x7faff00729d0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff00a4fe0 0x7faff00a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): picked mon.noname-d con 0x7faff00a4fe0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff0139100 0x7faff01349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): picked mon.noname-a con 0x7faff0139100 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7faff0142740 con 0x7faff0139100 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7faff00737f0 con 0x7faff00729d0 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7faff01428c0 con 0x7faff00a4fe0 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.373+0000 7faff62c9640 10 monclient(hunting): authenticate will time out at 1652.503973s 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00729d0 0x7faff00a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff00a4fe0 0x7faff00a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff0139100 0x7faff01349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00729d0 0x7faff00a2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:58254/0 (socket says 172.21.15.43:58254) 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 1 -- 172.21.15.43:0/2591819345 learned_addr learned my addr 172.21.15.43:0/2591819345 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): get_auth_request con 0x7faff00729d0 auth_method 0 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): get_auth_request con 0x7faff00a4fe0 auth_method 0 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): get_auth_request con 0x7faff0139100 auth_method 0 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient(hunting): handle_auth_done global_id 74436 payload 824 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00729d0 msgr2=0x7faff00a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7faff00729d0 0x7faff00a2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff0139100 msgr2=0x7faff01349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7faff0139100 0x7faff01349e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 monclient: found mon.noname-d 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff00a1ce0 con 0x7faff00a4fe0 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafef7fe640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff00a4fe0 0x7faff00a1320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faff006f9b0 tx=0x7fafd8019270 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 client.?.objecter ms_handle_connect 0x7faff00a4fe0 2024-04-19T00:56:22.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafd80bb070 con 0x7faff00a4fe0 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: dump: 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.390 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd80044a0 con 0x7faff00a4fe0 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 5 monclient: authenticate success, global_id 74436 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 monclient: _renew_subs 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff00a1fc0 con 0x7faff00a4fe0 2024-04-19T00:56:22.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafd80b0040 con 0x7faff00a4fe0 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafee7fc640 4 set_mon_vals no callback set 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: dump: 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.392 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 client.74436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 monclient: _renew_subs 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7faff006e4f0 con 0x7faff00a4fe0 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 1 librados: init done 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 0x7faff0135300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7faff006d140 con 0x7faff0134f20 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 0x7faff0135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fafd80aac00 con 0x7faff00a4fe0 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient: get_auth_request con 0x7faff0134f20 auth_method 0 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fafd0038e10 0x7fafd003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 10 monclient(hunting): handle_auth_done global_id 114227 payload 824 2024-04-19T00:56:22.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7fafd80e29c0 con 0x7faff00a4fe0 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 client.74436.objecter ms_dispatch 0x7faff0000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 3 client.74436.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 3 client.74436.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faff0135840 con 0x7faff0134f20 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 20 client.74436.objecter dump_active .. 0 homeless 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafeffff640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 0x7faff0135300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009f30 tx=0x7fafe800f640 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 client.74436.objecter ms_handle_connect 0x7faff0134f20 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 client.74436.objecter resend_mon_ops 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fafd0038e10 0x7fafd003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafe8019070 con 0x7faff0134f20 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: dump: 2024-04-19T00:56:22.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 10 monclient: get_auth_request con 0x7fafd0038e10 auth_method 0 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fafe8003bb0 con 0x7faff0134f20 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 msgr2=0x7faff0135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 0x7faff0135300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009f30 tx=0x7fafe800f640 comp rx=0 tx=0).stop 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 msgr2=0x7faff0135300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff0134f20 0x7faff0135300 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009f30 tx=0x7fafe800f640 comp rx=0 tx=0).stop 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7faff4866640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fafd0038e10 0x7fafd003b2d0 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7faff00a0160 tx=0x7fafe0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.377+0000 7fafecff9640 10 client.74436.objecter ms_handle_connect 0x7fafd0038e10 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 0x7faff00a72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7faff00a32b0 con 0x7faff00a6e90 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 0x7faff00a72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient: get_auth_request con 0x7faff00a6e90 auth_method 0 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 10 monclient(hunting): handle_auth_done global_id 114236 payload 824 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 1 -- 172.21.15.43:0/2591819345 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faff0136630 con 0x7faff00a6e90 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafeffff640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 0x7faff00a72a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009f30 tx=0x7fafe8004a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 client.74436.objecter ms_handle_connect 0x7faff00a6e90 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 client.74436.objecter resend_mon_ops 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fafe8020940 con 0x7faff00a6e90 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 monclient: dump: 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.400 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fafe80204d0 con 0x7faff00a6e90 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:22.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 msgr2=0x7faff00a72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 0x7faff00a72a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009f30 tx=0x7fafe8004a50 comp rx=0 tx=0).stop 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 msgr2=0x7faff00a72a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7fafecff9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7faff00a6e90 0x7faff00a72a0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fafe8009f30 tx=0x7fafe8004a50 comp rx=0 tx=0).stop 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:22.402 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19547, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.403 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:22.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:22.405 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:22.406 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.407 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.409 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:22.411 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.413 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.415 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.418 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.422 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:22.423 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 10 librados: watch_flush enter 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 10 librados: watch_flush exit 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 20 client.74436.objecter shutdown clearing up homeless session... 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 10 client.74436.objecter successfully canceled tick 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fafd0038e10 msgr2=0x7fafd003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fafd0038e10 0x7fafd003b2d0 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7faff00a0160 tx=0x7fafe0009190 comp rx=0 tx=0).stop 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 10 monclient: shutdown 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff00a4fe0 msgr2=0x7faff00a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff00a4fe0 0x7faff00a1320 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7faff006f9b0 tx=0x7fafd8019270 comp rx=0 tx=0).stop 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff4866640 1 -- 172.21.15.43:0/2591819345 reap_dead start 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 shutdown_connections 2024-04-19T00:56:22.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 --2- 172.21.15.43:0/2591819345 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7faff00a4fe0 0x7faff00a1320 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 >> 172.21.15.43:0/2591819345 conn(0x7faff00a6410 msgr2=0x7faff0132f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 shutdown_connections 2024-04-19T00:56:22.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 -- 172.21.15.43:0/2591819345 wait complete. 2024-04-19T00:56:22.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.385+0000 7faff62c9640 1 librados: shutdown 2024-04-19T00:56:22.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:22.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:56:22.426 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:56:22.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient: get_monmap_and_config 2024-04-19T00:56:22.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient: monmap: 2024-04-19T00:56:22.543 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:22.546780+0000 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:22.546780+0000 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:22.544 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 Processor -- start 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 -- start start 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient: init 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient: _add_conns ranks=[5,3,4,1,8,7,9,10,0,6,2] 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f06600a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): picked mon.noname-h con 0x7f06600a4f50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600731a0 0x7f06600a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): picked mon.noname-g con 0x7f06600731a0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 0x7f06600a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): picked mon.noname-c con 0x7f06600a3370 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f06600a1c80 con 0x7f06600731a0 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f06600a5330 con 0x7f06600a3370 2024-04-19T00:56:22.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f06601337a0 con 0x7f06600a4f50 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0666f32640 10 monclient(hunting): authenticate will time out at 1652.674608s 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 0x7f06600a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 0x7f06600a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:53870/0 (socket says 172.21.15.43:53870) 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 -- 172.21.15.43:0/4215522073 learned_addr learned my addr 172.21.15.43:0/4215522073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): get_auth_request con 0x7f06600a3370 auth_method 0 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f065ffff640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600731a0 0x7f06600a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0664cce640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f06600a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f065ffff640 10 monclient(hunting): get_auth_request con 0x7f06600731a0 auth_method 0 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f065ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f065ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f065ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0664cce640 10 monclient(hunting): get_auth_request con 0x7f06600a4f50 auth_method 0 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0664cce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0664cce640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f0664cce640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient(hunting): handle_auth_done global_id 64669 payload 274 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 -- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 msgr2=0x7f06600a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f06600a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 -- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600731a0 msgr2=0x7f06600a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600731a0 0x7f06600a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 monclient: found mon.noname-c 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:22.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.545+0000 7f06654cf640 1 -- 172.21.15.43:0/4215522073 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0660133a20 con 0x7f06600a3370 2024-04-19T00:56:22.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f06654cf640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 0x7f06600a37d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0658009e80 tx=0x7f065801c8b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=31ab93e52847a571 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 1 -- 172.21.15.43:0/4215522073 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06580090d0 con 0x7f06600a3370 2024-04-19T00:56:22.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:22.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: dump: 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.557 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 1 -- 172.21.15.43:0/4215522073 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06580a5a50 con 0x7f06600a3370 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 5 monclient: authenticate success, global_id 64669 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 1 -- 172.21.15.43:0/4215522073 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f06580a5c00 con 0x7f06600a3370 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 4 set_mon_vals no callback set 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f065f7fe640 10 monclient: dump: 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.558 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 10 monclient: shutdown 2024-04-19T00:56:22.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 -- 172.21.15.43:0/4215522073 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 msgr2=0x7f06600a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 0x7f06600a37d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0658009e80 tx=0x7f065801c8b0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 -- 172.21.15.43:0/4215522073 shutdown_connections 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f06600a3370 0x7f06600a37d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600731a0 0x7f06600a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 --2- 172.21.15.43:0/4215522073 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f06600a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 -- 172.21.15.43:0/4215522073 >> 172.21.15.43:0/4215522073 conn(0x7f06600a4460 msgr2=0x7f06600a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 -- 172.21.15.43:0/4215522073 shutdown_connections 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 -- 172.21.15.43:0/4215522073 wait complete. 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 10 monclient: monmap: 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:22.560 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 librados: starting msgr at 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 librados: starting objecter 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 Processor -- start 2024-04-19T00:56:22.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 -- start start 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 librados: setting wanted keys 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 1 librados: calling monclient init 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.549+0000 7f0666f32640 10 monclient: init 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient: _add_conns ranks=[0,3,5,1,6,7,2,8,4] 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06600731a0 0x7f06601ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): picked mon.noname-a con 0x7f06600731a0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600a3370 0x7f066013c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): picked mon.noname-g con 0x7f06600a3370 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f066013cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): picked mon.noname-h con 0x7f06600a4f50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f06601404f0 con 0x7f06600731a0 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0660140370 con 0x7f06600a3370 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0660140670 con 0x7f06600a4f50 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): authenticate will time out at 1652.680442s 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06600731a0 0x7f06601ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06600731a0 0x7f06601ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34266/0 (socket says 172.21.15.43:34266) 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 -- 172.21.15.43:0/310519283 learned_addr learned my addr 172.21.15.43:0/310519283 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): get_auth_request con 0x7f06600731a0 auth_method 0 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600a3370 0x7f066013c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f066013cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): get_auth_request con 0x7f06600a3370 auth_method 0 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 10 monclient(hunting): get_auth_request con 0x7f06600a4f50 auth_method 0 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient(hunting): handle_auth_done global_id 64863 payload 824 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 msgr2=0x7f066013cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f06600a4f50 0x7f066013cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600a3370 msgr2=0x7f066013c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f06600a3370 0x7f066013c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 monclient: found mon.noname-a 2024-04-19T00:56:22.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f066006f9c0 con 0x7f06600731a0 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0664cce640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06600731a0 0x7f06601ce9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f065000d860 tx=0x7f065000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 client.?.objecter ms_handle_connect 0x7f06600731a0 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f065000de50 con 0x7f06600731a0 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: dump: 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.565 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0650019070 con 0x7f06600731a0 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f065000bce0 con 0x7f06600731a0 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 5 monclient: authenticate success, global_id 64863 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient: _renew_subs 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f066013ae80 con 0x7f06600731a0 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065effd640 4 set_mon_vals no callback set 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 client.64863.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: dump: 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.567 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient: _renew_subs 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f066013aba0 con 0x7f06600731a0 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 librados: init done 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 0x7f066002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0660139a40 con 0x7f06601379b0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f065000b870 con 0x7f06600731a0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f064c038e10 0x7f064c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f065000f040 con 0x7f06600731a0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 client.64863.objecter ms_dispatch 0x7f0660000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 3 client.64863.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 3 client.64863.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 0x7f066002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 20 client.64863.objecter dump_active .. 0 homeless 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient: get_auth_request con 0x7f06601379b0 auth_method 0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f064c038e10 0x7f064c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 10 monclient: get_auth_request con 0x7f064c038e10 auth_method 0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 10 monclient(hunting): handle_auth_done global_id 114175 payload 824 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f066006d140 con 0x7f06601379b0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 client.64863.objecter ms_handle_connect 0x7f06601379b0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 client.64863.objecter resend_mon_ops 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065ffff640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 0x7f066002c450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f066006c8f0 tx=0x7f065400f8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 client.64863.objecter ms_handle_connect 0x7f064c038e10 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f06654cf640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f064c038e10 0x7f064c03b2d0 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f0658009e80 tx=0x7f06580a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0654018020 con 0x7f06601379b0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.553+0000 7f065d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 10 monclient: dump: 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.570 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0654014670 con 0x7f06601379b0 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:22.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 msgr2=0x7f066002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 0x7f066002c450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f066006c8f0 tx=0x7f065400f8a0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 msgr2=0x7f066002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.557+0000 7f065d7fa640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f06601379b0 0x7f066002c450 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f066006c8f0 tx=0x7f065400f8a0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f0666f32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:22.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f0666f32640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 0x7f066002c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f0666f32640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f06601cf630 con 0x7f0660137d90 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 0x7f066002c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient: get_auth_request con 0x7f0660137d90 auth_method 0 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 10 monclient(hunting): handle_auth_done global_id 114184 payload 824 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.561+0000 7f065ffff640 1 -- 172.21.15.43:0/310519283 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f066009fa70 con 0x7f0660137d90 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065ffff640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 0x7f066002c990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0660069720 tx=0x7f06540046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 client.64863.objecter ms_handle_connect 0x7f0660137d90 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 client.64863.objecter resend_mon_ops 2024-04-19T00:56:22.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0654018040 con 0x7f0660137d90 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 monclient: dump: 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.575 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f065401f020 con 0x7f0660137d90 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 msgr2=0x7f066002c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 0x7f066002c990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0660069720 tx=0x7f06540046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 msgr2=0x7f066002c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f065d7fa640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0660137d90 0x7f066002c990 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0660069720 tx=0x7f06540046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.576 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19731, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:56:22.577 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:22.578 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:22.579 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:22.580 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.581 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:22.584 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.588 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.592 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.595 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.596 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:22.596 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 10 librados: watch_flush enter 2024-04-19T00:56:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 10 librados: watch_flush exit 2024-04-19T00:56:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 20 client.64863.objecter shutdown clearing up homeless session... 2024-04-19T00:56:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 10 client.64863.objecter successfully canceled tick 2024-04-19T00:56:22.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f064c038e10 msgr2=0x7f064c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f064c038e10 0x7f064c03b2d0 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f0658009e80 tx=0x7f06580a7070 comp rx=0 tx=0).stop 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 10 monclient: shutdown 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06600731a0 msgr2=0x7f06601ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 --2- 172.21.15.43:0/310519283 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f06600731a0 0x7f06601ce9c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f065000d860 tx=0x7f065000dd20 comp rx=0 tx=0).stop 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f06654cf640 1 -- 172.21.15.43:0/310519283 reap_dead start 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f06654cf640 1 -- 172.21.15.43:0/310519283 reap_dead start 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 shutdown_connections 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 >> 172.21.15.43:0/310519283 conn(0x7f06600a4460 msgr2=0x7f06600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 shutdown_connections 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 -- 172.21.15.43:0/310519283 wait complete. 2024-04-19T00:56:22.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.565+0000 7f0666f32640 1 librados: shutdown 2024-04-19T00:56:22.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:22.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:56:22.599 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:56:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 10 monclient: get_monmap_and_config 2024-04-19T00:56:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 10 monclient: monmap: 2024-04-19T00:56:22.717 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:22.720899+0000 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:22.720899+0000 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.718 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.719 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:22.719 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:22.719 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 1 Processor -- start 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 1 -- start start 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 10 monclient: init 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 10 monclient: _add_conns ranks=[8,7,3,1,6,0,10,9,2,4,5] 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.717+0000 7f92c4f81640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 0x7f92c01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): picked mon.noname-e con 0x7f92c0138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c00a5b80 0x7f92c01395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): picked mon.noname-i con 0x7f92c00a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c01358b0 0x7f92c0135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): picked mon.noname-g con 0x7f92c01358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f92c00a4410 con 0x7f92c01358b0 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f92c0139b30 con 0x7f92c00a5b80 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f92c00a3dc0 con 0x7f92c0138cd0 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92c4f81640 10 monclient(hunting): authenticate will time out at 1652.848075s 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 0x7f92c01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 0x7f92c01390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:55406/0 (socket says 172.21.15.43:55406) 2024-04-19T00:56:22.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 -- 172.21.15.43:0/3801943750 learned_addr learned my addr 172.21.15.43:0/3801943750 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): get_auth_request con 0x7f92c0138cd0 auth_method 0 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bdd9b640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c00a5b80 0x7f92c01395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bdd9b640 10 monclient(hunting): get_auth_request con 0x7f92c00a5b80 auth_method 0 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient(hunting): handle_auth_done global_id 114245 payload 274 2024-04-19T00:56:22.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 -- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c00a5b80 msgr2=0x7f92c01395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c00a5b80 0x7f92c01395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 -- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c01358b0 msgr2=0x7f92c0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bed9d640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c01358b0 0x7f92c0135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c01358b0 0x7f92c0135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 monclient: found mon.noname-e 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92be59c640 1 -- 172.21.15.43:0/3801943750 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c00a40a0 con 0x7f92c0138cd0 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bed9d640 10 monclient: get_auth_request con 0x7f92c01358b0 auth_method 0 2024-04-19T00:56:22.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.721+0000 7f92bed9d640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c01358b0 0x7f92c0135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92be59c640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 0x7f92c01390b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098a0 tx=0x7f92a801c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=81da1dc0a6486be2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 1 -- 172.21.15.43:0/3801943750 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a80b0070 con 0x7f92c0138cd0 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: dump: 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.731 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 5 monclient: authenticate success, global_id 114245 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 1 -- 172.21.15.43:0/3801943750 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a80aa810 con 0x7f92c0138cd0 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 4 set_mon_vals no callback set 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 1 -- 172.21.15.43:0/3801943750 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a80a9300 con 0x7f92c0138cd0 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bd59a640 10 monclient: dump: 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.733 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: shutdown 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- 172.21.15.43:0/3801943750 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 msgr2=0x7f92c01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 0x7f92c01390b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098a0 tx=0x7f92a801c3e0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- 172.21.15.43:0/3801943750 shutdown_connections 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c01358b0 0x7f92c0135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c00a5b80 0x7f92c01395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- 172.21.15.43:0/3801943750 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92c0138cd0 0x7f92c01390b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- 172.21.15.43:0/3801943750 >> 172.21.15.43:0/3801943750 conn(0x7f92c00ab2f0 msgr2=0x7f92c00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- 172.21.15.43:0/3801943750 shutdown_connections 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- 172.21.15.43:0/3801943750 wait complete. 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: monmap: 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:22.735 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 librados: starting msgr at 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 librados: starting objecter 2024-04-19T00:56:22.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 Processor -- start 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- start start 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 librados: setting wanted keys 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 librados: calling monclient init 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: init 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: _add_conns ranks=[5,7,0,8,4,1,2,6,3] 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92c00a5b80 0x7f92c012ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): picked mon.noname-h con 0x7f92c00a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c01358b0 0x7f92c012bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): picked mon.noname-i con 0x7f92c01358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c0138cd0 0x7f92c01334c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): picked mon.noname-a con 0x7f92c0138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f92c00a2010 con 0x7f92c0138cd0 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f92c00a1e90 con 0x7f92c00a5b80 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f92c0142aa0 con 0x7f92c01358b0 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient(hunting): authenticate will time out at 1652.854897s 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c0138cd0 0x7f92c01334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c0138cd0 0x7f92c01334c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34272/0 (socket says 172.21.15.43:34272) 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 -- 172.21.15.43:0/303440046 learned_addr learned my addr 172.21.15.43:0/303440046 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): get_auth_request con 0x7f92c0138cd0 auth_method 0 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bdd9b640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c01358b0 0x7f92c012bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bdd9b640 10 monclient(hunting): get_auth_request con 0x7f92c01358b0 auth_method 0 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92be59c640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92c00a5b80 0x7f92c012ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92be59c640 10 monclient(hunting): get_auth_request con 0x7f92c00a5b80 auth_method 0 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92be59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient(hunting): handle_auth_done global_id 64872 payload 824 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c01358b0 msgr2=0x7f92c012bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c01358b0 0x7f92c012bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92c00a5b80 msgr2=0x7f92c012ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f92c00a5b80 0x7f92c012ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 monclient: found mon.noname-a 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c013b6c0 con 0x7f92c0138cd0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bdd9b640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f92c01358b0 0x7f92c012bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bed9d640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c0138cd0 0x7f92c01334c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92b800cef0 tx=0x7f92b800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 client.?.objecter ms_handle_connect 0x7f92c0138cd0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92b801a070 con 0x7f92c0138cd0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 monclient: dump: 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b800dab0 con 0x7f92c0138cd0 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92b800dc60 con 0x7f92c0138cd0 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 5 monclient: authenticate success, global_id 64872 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: _renew_subs 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:22.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c00a3590 con 0x7f92c0138cd0 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.725+0000 7f92bcd99640 4 set_mon_vals no callback set 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: dump: 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.743 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 10 client.64872.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 10 monclient: _renew_subs 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c013a0c0 con 0x7f92c0138cd0 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 1 librados: init done 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 0x7f92c0133de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f92c006d140 con 0x7f92c0133a00 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f92b800d640 con 0x7f92c0138cd0 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f92a0038e10 0x7f92a003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 0x7f92c0133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f92b8049b30 con 0x7f92c0138cd0 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 client.64872.objecter ms_dispatch 0x7f92c0000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 3 client.64872.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 3 client.64872.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient: get_auth_request con 0x7f92c0133a00 auth_method 0 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 20 client.64872.objecter dump_active .. 0 homeless 2024-04-19T00:56:22.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92bdd9b640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f92a0038e10 0x7f92a003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92bdd9b640 10 monclient: get_auth_request con 0x7f92a0038e10 auth_method 0 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 10 monclient(hunting): handle_auth_done global_id 19470 payload 824 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92c00693b0 con 0x7f92c0133a00 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 client.64872.objecter ms_handle_connect 0x7f92c0133a00 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 client.64872.objecter resend_mon_ops 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92be59c640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 0x7f92c0133de0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009870 tx=0x7f92a80191e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 client.64872.objecter ms_handle_connect 0x7f92a0038e10 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92bdd9b640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f92a0038e10 0x7f92a003b2d0 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f92c00a7360 tx=0x7f92b0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a80ba030 con 0x7f92c0133a00 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: dump: 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.747 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f92a80036e0 con 0x7f92c0133a00 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 msgr2=0x7f92c0133de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 0x7f92c0133de0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009870 tx=0x7f92a80191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 msgr2=0x7f92c0133de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.729+0000 7f92b6ffd640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0133a00 0x7f92c0133de0 secure :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009870 tx=0x7f92a80191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92c4f81640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92c4f81640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 0x7f92c0137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92c4f81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f92c0068240 con 0x7f92c0134320 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 0x7f92c0137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient: get_auth_request con 0x7f92c0134320 auth_method 0 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 10 monclient(hunting): handle_auth_done global_id 19479 payload 824 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 1 -- 172.21.15.43:0/303440046 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92c0135660 con 0x7f92c0134320 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92be59c640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 0x7f92c0137a90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098a0 tx=0x7f92a80a8000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 client.64872.objecter ms_handle_connect 0x7f92c0134320 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 client.64872.objecter resend_mon_ops 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92a80ba040 con 0x7f92c0134320 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 monclient: dump: 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.751 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f92a80b6680 con 0x7f92c0134320 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 msgr2=0x7f92c0137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 0x7f92c0137a90 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098a0 tx=0x7f92a80a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 msgr2=0x7f92c0137a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.737+0000 7f92b6ffd640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f92c0134320 0x7f92c0137a90 secure :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f92a80098a0 tx=0x7f92a80a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.753 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 726220, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.754 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.755 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.756 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:22.757 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.758 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.763 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.764 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.765 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.766 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.768 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:22.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.771 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.775 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:22.775 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:22.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 10 librados: watch_flush enter 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 10 librados: watch_flush exit 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 20 client.64872.objecter shutdown clearing up homeless session... 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 10 client.64872.objecter successfully canceled tick 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f92a0038e10 msgr2=0x7f92a003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f92a0038e10 0x7f92a003b2d0 secure :-1 s=READY pgs=3537 cs=0 l=1 rev1=1 crypto rx=0x7f92c00a7360 tx=0x7f92b0008040 comp rx=0 tx=0).stop 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 10 monclient: shutdown 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c0138cd0 msgr2=0x7f92c01334c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 --2- 172.21.15.43:0/303440046 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f92c0138cd0 0x7f92c01334c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f92b800cef0 tx=0x7f92b800a710 comp rx=0 tx=0).stop 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92bed9d640 1 -- 172.21.15.43:0/303440046 reap_dead start 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92bed9d640 1 -- 172.21.15.43:0/303440046 reap_dead start 2024-04-19T00:56:22.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 shutdown_connections 2024-04-19T00:56:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 >> 172.21.15.43:0/303440046 conn(0x7f92c00ab2f0 msgr2=0x7f92c00a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 shutdown_connections 2024-04-19T00:56:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 -- 172.21.15.43:0/303440046 wait complete. 2024-04-19T00:56:22.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.741+0000 7f92c4f81640 1 librados: shutdown 2024-04-19T00:56:22.778 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:22.778 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:56:22.778 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:56:22.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient: get_monmap_and_config 2024-04-19T00:56:22.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient: monmap: 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:22.902770+0000 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:22.902770+0000 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:22.900 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:22.901 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 Processor -- start 2024-04-19T00:56:22.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 -- start start 2024-04-19T00:56:22.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient: init 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient: _add_conns ranks=[2,7,0,1,5,4,8,9,3,6,10] 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7184138cd0 0x7f71841390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): picked mon.noname-b con 0x7f7184138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f71840a5b80 0x7f71841395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): picked mon.noname-i con 0x7f71840a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71841358b0 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71840a3dc0 con 0x7f71841358b0 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f71840a4410 con 0x7f7184138cd0 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7184139b30 con 0x7f71840a5b80 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7188fa7640 10 monclient(hunting): authenticate will time out at 1653.029587s 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7184138cd0 0x7f71841390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7182d9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f71841358b0 0x7f7184135d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:44974/0 (socket says 172.21.15.43:44974) 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7182d9d640 1 -- 172.21.15.43:0/1369230211 learned_addr learned my addr 172.21.15.43:0/1369230211 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): get_auth_request con 0x7f7184138cd0 auth_method 0 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7181d9b640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f71840a5b80 0x7f71841395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7181d9b640 10 monclient(hunting): get_auth_request con 0x7f71840a5b80 auth_method 0 2024-04-19T00:56:22.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7181d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718159a640 1 -- 172.21.15.43:0/1369230211 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676834461 0 0) 0x7f71840a3dc0 con 0x7f71841358b0 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718159a640 10 monclient(hunting): _negotiate 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718159a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718159a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718159a640 10 monclient(hunting): my global_id is 64881 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718159a640 1 -- 172.21.15.43:0/1369230211 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7170003530 con 0x7f71841358b0 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient(hunting): handle_auth_done global_id 74459 payload 274 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 1 -- 172.21.15.43:0/1369230211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f71840a5b80 msgr2=0x7f71841395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f71840a5b80 0x7f71841395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 1 -- 172.21.15.43:0/1369230211 >> v1:172.21.15.43:6789/0 conn(0x7f71841358b0 legacy=0x7f7184135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 1 monclient: found mon.noname-b 2024-04-19T00:56:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f718259c640 1 -- 172.21.15.43:0/1369230211 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71840a40a0 con 0x7f7184138cd0 2024-04-19T00:56:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.901+0000 7f7181d9b640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f71840a5b80 0x7f71841395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:56:22.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718259c640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7184138cd0 0x7f71841390b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71740097f0 tx=0x7f717401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23effad9b0bc7788 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 1 -- 172.21.15.43:0/1369230211 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71740b0070 con 0x7f7184138cd0 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: dump: 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.913 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.914 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 1 -- 172.21.15.43:0/1369230211 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71740aa810 con 0x7f7184138cd0 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 5 monclient: authenticate success, global_id 74459 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 4 set_mon_vals no callback set 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 1 -- 172.21.15.43:0/1369230211 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71740a9300 con 0x7f7184138cd0 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:22.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718159a640 10 monclient: dump: 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.916 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: shutdown 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- 172.21.15.43:0/1369230211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7184138cd0 msgr2=0x7f71841390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7184138cd0 0x7f71841390b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71740097f0 tx=0x7f717401c340 comp rx=0 tx=0).stop 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- 172.21.15.43:0/1369230211 shutdown_connections 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f71840a5b80 0x7f71841395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1369230211 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7184138cd0 0x7f71841390b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- 172.21.15.43:0/1369230211 >> 172.21.15.43:0/1369230211 conn(0x7f71840ab2f0 msgr2=0x7f71840ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- 172.21.15.43:0/1369230211 shutdown_connections 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- 172.21.15.43:0/1369230211 wait complete. 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: build_initial_monmap 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: monmap: 2024-04-19T00:56:22.917 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:22.918 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:22.919 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:22.919 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 librados: starting msgr at 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 librados: starting objecter 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 Processor -- start 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- start start 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 librados: setting wanted keys 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 librados: calling monclient init 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: init 2024-04-19T00:56:22.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient: _add_conns ranks=[5,6,1,3,2,4,7,0,8] 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71840a5b80 0x7f718412da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): picked mon.noname-h con 0x7f71840a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71841358b0 0x7f718412dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): picked mon.noname-d con 0x7f71841358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7184138cd0 0x7f718409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): picked mon.noname-f con 0x7f7184138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7184143ab0 con 0x7f7184138cd0 2024-04-19T00:56:22.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f71840a20a0 con 0x7f71840a5b80 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7184143c30 con 0x7f71841358b0 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7188fa7640 10 monclient(hunting): authenticate will time out at 1653.035364s 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71841358b0 0x7f718412dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71841358b0 0x7f718412dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42764/0 (socket says 172.21.15.43:42764) 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 -- 172.21.15.43:0/1973449729 learned_addr learned my addr 172.21.15.43:0/1973449729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): get_auth_request con 0x7f71841358b0 auth_method 0 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718259c640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71840a5b80 0x7f718412da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient(hunting): handle_auth_done global_id 74445 payload 824 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7182d9d640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7184138cd0 0x7f718409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71840a5b80 msgr2=0x7f718412da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71840a5b80 0x7f718412da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7184138cd0 msgr2=0x7f718409ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7184138cd0 0x7f718409ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 monclient: found mon.noname-d 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f7181d9b640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718409f190 con 0x7f71841358b0 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718259c640 10 monclient: get_auth_request con 0x7f71840a5b80 auth_method 0 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.905+0000 7f718259c640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f71840a5b80 0x7f718412da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7181d9b640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71841358b0 0x7f718412dfd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f716c00e790 tx=0x7f716c00ec50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 client.?.objecter ms_handle_connect 0x7f71841358b0 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f716c017070 con 0x7f71841358b0 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: dump: 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.924 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: _finish_auth 0 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:22.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716c00cb40 con 0x7f71841358b0 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 5 monclient: authenticate success, global_id 74445 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 monclient: _renew_subs 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718409dce0 con 0x7f71841358b0 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f716c00f040 con 0x7f71841358b0 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7180d99640 4 set_mon_vals no callback set 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 client.74445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: dump: 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.926 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 monclient: _renew_subs 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f71840a3590 con 0x7f71841358b0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 1 librados: init done 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 0x7f718409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f718409fea0 con 0x7f718409bfb0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f716c01a070 con 0x7f71841358b0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7164038e10 0x7f716403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f716c048f60 con 0x7f71841358b0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 client.74445.objecter ms_dispatch 0x7f7184000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 3 client.74445.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 3 client.74445.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 0x7f718409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient: get_auth_request con 0x7f718409bfb0 auth_method 0 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 20 client.74445.objecter dump_active .. 0 homeless 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7182d9d640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7164038e10 0x7f716403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7182d9d640 10 monclient: get_auth_request con 0x7f7164038e10 auth_method 0 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 10 monclient(hunting): handle_auth_done global_id 74354 payload 824 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71840693b0 con 0x7f718409bfb0 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 client.74445.objecter ms_handle_connect 0x7f7164038e10 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 client.74445.objecter ms_handle_connect 0x7f718409bfb0 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f7182d9d640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7164038e10 0x7f716403b2d0 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7f717c0015b0 tx=0x7f717c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 client.74445.objecter resend_mon_ops 2024-04-19T00:56:22.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f718259c640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 0x7f718409c390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71840a4380 tx=0x7f7174019140 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71740b0040 con 0x7f718409bfb0 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: dump: 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7174003620 con 0x7f718409bfb0 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 msgr2=0x7f718409c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 0x7f718409c390 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71840a4380 tx=0x7f7174019140 comp rx=0 tx=0).stop 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 msgr2=0x7f718409c390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.909+0000 7f717affd640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409bfb0 0x7f718409c390 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f71840a4380 tx=0x7f7174019140 comp rx=0 tx=0).stop 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f7188fa7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 0x7f71841321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f7188fa7640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f718406d140 con 0x7f718409c8d0 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 0x7f71841321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient: get_auth_request con 0x7f718409c8d0 auth_method 0 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:22.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.917+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f718259c640 10 monclient(hunting): handle_auth_done global_id 74363 payload 824 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f718259c640 1 -- 172.21.15.43:0/1973449729 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f718409d5c0 con 0x7f718409c8d0 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f718259c640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 0x7f71841321b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f717401aa50 tx=0x7f71740a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 client.74445.objecter ms_handle_connect 0x7f718409c8d0 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 client.74445.objecter resend_mon_ops 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f71740b0040 con 0x7f718409c8d0 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 monclient: dump: 2024-04-19T00:56:22.934 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f71740b6470 con 0x7f718409c8d0 2024-04-19T00:56:22.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 msgr2=0x7f71841321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 0x7f71841321b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f717401aa50 tx=0x7f71740a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 msgr2=0x7f71841321b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f717affd640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f718409c8d0 0x7f71841321b0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f717401aa50 tx=0x7f71740a8000 comp rx=0 tx=0).stop 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 162538, 2024-04-19T00:56:22.936 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.937 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.938 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:22.939 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:22.940 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:22.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.942 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.945 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.947 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:22.948 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.949 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.950 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.951 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:22.952 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.953 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:22.954 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:22.955 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:22.956 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:22.957 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 10 librados: watch_flush enter 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 10 librados: watch_flush exit 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 20 client.74445.objecter shutdown clearing up homeless session... 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 10 client.74445.objecter successfully canceled tick 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7164038e10 msgr2=0x7f716403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7164038e10 0x7f716403b2d0 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7f717c0015b0 tx=0x7f717c009040 comp rx=0 tx=0).stop 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 10 monclient: shutdown 2024-04-19T00:56:22.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71841358b0 msgr2=0x7f718412dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 --2- 172.21.15.43:0/1973449729 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f71841358b0 0x7f718412dfd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f716c00e790 tx=0x7f716c00ec50 comp rx=0 tx=0).stop 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7182d9d640 1 -- 172.21.15.43:0/1973449729 reap_dead start 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7182d9d640 1 -- 172.21.15.43:0/1973449729 reap_dead start 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 shutdown_connections 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 >> 172.21.15.43:0/1973449729 conn(0x7f71840ab2f0 msgr2=0x7f71840a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 shutdown_connections 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 -- 172.21.15.43:0/1973449729 wait complete. 2024-04-19T00:56:22.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:22.921+0000 7f7188fa7640 1 librados: shutdown 2024-04-19T00:56:22.960 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:22.960 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:56:22.961 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:56:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.077+0000 7f09a74c6640 10 monclient: get_monmap_and_config 2024-04-19T00:56:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.077+0000 7f09a74c6640 10 monclient: build_initial_monmap 2024-04-19T00:56:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient: monmap: 2024-04-19T00:56:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:23.079 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:23.082425+0000 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:23.082425+0000 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:23.080 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:23.081 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 Processor -- start 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 -- start start 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient: init 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient: _add_conns ranks=[6,5,2,1,0,7,4,3,9,8,10] 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 0x7f09a01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): picked mon.noname-d con 0x7f09a0138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a00a5b80 0x7f09a01395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): picked mon.noname-h con 0x7f09a00a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f09a01358b0 0x7f09a0135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): picked mon.noname-b con 0x7f09a01358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f09a00a4410 con 0x7f09a01358b0 2024-04-19T00:56:23.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f09a0139b30 con 0x7f09a00a5b80 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f09a00a3dc0 con 0x7f09a0138cd0 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a74c6640 10 monclient(hunting): authenticate will time out at 1653.209548s 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 0x7f09a01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f09a01358b0 0x7f09a0135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 0x7f09a01390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42774/0 (socket says 172.21.15.43:42774) 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 -- 172.21.15.43:0/101642933 learned_addr learned my addr 172.21.15.43:0/101642933 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): get_auth_request con 0x7f09a01358b0 auth_method 0 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): get_auth_request con 0x7f09a0138cd0 auth_method 0 2024-04-19T00:56:23.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a4a61640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a00a5b80 0x7f09a01395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a4a61640 10 monclient(hunting): get_auth_request con 0x7f09a00a5b80 auth_method 0 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a4a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a4a61640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a4a61640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient(hunting): handle_auth_done global_id 74454 payload 274 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 -- 172.21.15.43:0/101642933 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a00a5b80 msgr2=0x7f09a01395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a00a5b80 0x7f09a01395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 -- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f09a01358b0 msgr2=0x7f09a0135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f09a01358b0 0x7f09a0135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient: _finish_hunting 0 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 monclient: found mon.noname-d 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5262640 1 -- 172.21.15.43:0/101642933 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a00a40a0 con 0x7f09a0138cd0 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.081+0000 7f09a5a63640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f09a01358b0 0x7f09a0135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 0x7f09a01390b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f099c0098a0 tx=0x7f099c01c3e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=64df9ebb96e08b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 1 -- 172.21.15.43:0/101642933 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f099c0b0070 con 0x7f09a0138cd0 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: dump: 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.098 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: _finish_auth 0 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 5 monclient: authenticate success, global_id 74454 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 1 -- 172.21.15.43:0/101642933 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099c0aa810 con 0x7f09a0138cd0 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:23.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 4 set_mon_vals no callback set 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 1 -- 172.21.15.43:0/101642933 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f099c0a9300 con 0x7f09a0138cd0 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0997fff640 10 monclient: dump: 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.100 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: shutdown 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 msgr2=0x7f09a01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 0x7f09a01390b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f099c0098a0 tx=0x7f099c01c3e0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/101642933 shutdown_connections 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f09a01358b0 0x7f09a0135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a00a5b80 0x7f09a01395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- 172.21.15.43:0/101642933 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09a0138cd0 0x7f09a01390b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/101642933 >> 172.21.15.43:0/101642933 conn(0x7f09a00ab2f0 msgr2=0x7f09a00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/101642933 shutdown_connections 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/101642933 wait complete. 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: build_initial_monmap 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: monmap: 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:23.102 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 librados: starting msgr at 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 librados: starting objecter 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 Processor -- start 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- start start 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 librados: setting wanted keys 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 librados: calling monclient init 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: init 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:23.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: _add_conns ranks=[0,5,4,7,6,2,8,1,3] 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a00a5b80 0x7f09a01d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): picked mon.noname-a con 0x7f09a00a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a01358b0 0x7f09a013ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): picked mon.noname-h con 0x7f09a01358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f09a0138cd0 0x7f09a013f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): picked mon.noname-c con 0x7f09a0138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f09a00a2010 con 0x7f09a00a5b80 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f09a00a1e90 con 0x7f09a0138cd0 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f09a0142aa0 con 0x7f09a01358b0 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient(hunting): authenticate will time out at 1653.214818s 2024-04-19T00:56:23.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a00a5b80 0x7f09a01d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f09a0138cd0 0x7f09a013f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a00a5b80 0x7f09a01d0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:34288/0 (socket says 172.21.15.43:34288) 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f09a0138cd0 0x7f09a013f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:53872/0 (socket says 172.21.15.43:53872) 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 1 -- 172.21.15.43:0/1298543093 learned_addr learned my addr 172.21.15.43:0/1298543093 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): get_auth_request con 0x7f09a00a5b80 auth_method 0 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): get_auth_request con 0x7f09a0138cd0 auth_method 0 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a4a61640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a01358b0 0x7f09a013ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a4a61640 10 monclient(hunting): get_auth_request con 0x7f09a01358b0 auth_method 0 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a4a61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a4a61640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a4a61640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient(hunting): handle_auth_done global_id 64678 payload 824 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a01358b0 msgr2=0x7f09a013ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09a01358b0 0x7f09a013ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a00a5b80 msgr2=0x7f09a01d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f09a00a5b80 0x7f09a01d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient: _finish_hunting 0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 monclient: found mon.noname-c 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a01d1150 con 0x7f09a0138cd0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a5a63640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f09a0138cd0 0x7f09a013f1c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0998009e50 tx=0x7f099800c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 client.?.objecter ms_handle_connect 0x7f09a0138cd0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0998018030 con 0x7f09a0138cd0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: dump: 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.107 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:23.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f099800cd90 con 0x7f09a0138cd0 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0998014d60 con 0x7f09a0138cd0 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 5 monclient: authenticate success, global_id 64678 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: _renew_subs 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a013a1c0 con 0x7f09a0138cd0 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09977fe640 4 set_mon_vals no callback set 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f0995ffb640 10 monclient: dump: 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.110 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 client.64678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: _renew_subs 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a013b450 con 0x7f09a0138cd0 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.085+0000 7f09a74c6640 1 librados: init done 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a74c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a74c6640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 0x7f09a01338a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f09a006d140 con 0x7f09a01334c0 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f099801b070 con 0x7f09a0138cd0 2024-04-19T00:56:23.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 0x7f09a01338a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient: get_auth_request con 0x7f09a01334c0 auth_method 0 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0984038e10 0x7f098403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f0998019070 con 0x7f09a0138cd0 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 client.64678.objecter ms_dispatch 0x7f09a0000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 3 client.64678.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 3 client.64678.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 20 client.64678.objecter dump_active .. 0 homeless 2024-04-19T00:56:23.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a4a61640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0984038e10 0x7f098403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a4a61640 10 monclient: get_auth_request con 0x7f0984038e10 auth_method 0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 10 monclient(hunting): handle_auth_done global_id 74338 payload 824 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09a0135030 con 0x7f09a01334c0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a5262640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 0x7f09a01338a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f099c0ae040 tx=0x7f099c0191e0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 client.64678.objecter ms_handle_connect 0x7f09a01334c0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 client.64678.objecter resend_mon_ops 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 client.64678.objecter ms_handle_connect 0x7f0984038e10 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f09a4a61640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0984038e10 0x7f098403b2d0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7f09a012fcc0 tx=0x7f0990008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f099c0ba070 con 0x7f09a01334c0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 monclient: dump: 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.113 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f099c003620 con 0x7f09a01334c0 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:23.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 msgr2=0x7f09a01338a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 0x7f09a01338a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f099c0ae040 tx=0x7f099c0191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 msgr2=0x7f09a01338a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:23.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.089+0000 7f0995ffb640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a01334c0 0x7f09a01338a0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f099c0ae040 tx=0x7f099c0191e0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a74c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a74c6640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 0x7f09a0137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a74c6640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f09a0068240 con 0x7f09a0133de0 2024-04-19T00:56:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 0x7f09a0137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient: get_auth_request con 0x7f09a0133de0 auth_method 0 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 10 monclient(hunting): handle_auth_done global_id 74347 payload 824 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 1 -- 172.21.15.43:0/1298543093 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09a00a3590 con 0x7f09a0133de0 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 client.64678.objecter ms_handle_connect 0x7f09a0133de0 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 client.64678.objecter resend_mon_ops 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f09a5262640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 0x7f09a0137a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f099c01c420 tx=0x7f099c009240 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f099c0b0040 con 0x7f09a0133de0 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:23.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 monclient: dump: 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.118 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f099c0bfc10 con 0x7f09a0133de0 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 msgr2=0x7f09a0137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 0x7f09a0137a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f099c01c420 tx=0x7f099c009240 comp rx=0 tx=0).stop 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 msgr2=0x7f09a0137a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.097+0000 7f0995ffb640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f09a0133de0 0x7f09a0137a90 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f099c01c420 tx=0x7f099c009240 comp rx=0 tx=0).stop 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:23.119 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 84, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 162714, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:23.120 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:23.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:23.122 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:23.123 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:23.124 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:23.125 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.126 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:23.127 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.128 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:23.129 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.130 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.131 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.132 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.133 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.134 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.135 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.136 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:23.137 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:23.138 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:23.139 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 10 librados: watch_flush enter 2024-04-19T00:56:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 10 librados: watch_flush exit 2024-04-19T00:56:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 20 client.64678.objecter shutdown clearing up homeless session... 2024-04-19T00:56:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 10 client.64678.objecter successfully canceled tick 2024-04-19T00:56:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0984038e10 msgr2=0x7f098403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0984038e10 0x7f098403b2d0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7f09a012fcc0 tx=0x7f0990008040 comp rx=0 tx=0).stop 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 10 monclient: shutdown 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f09a0138cd0 msgr2=0x7f09a013f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 --2- 172.21.15.43:0/1298543093 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f09a0138cd0 0x7f09a013f1c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0998009e50 tx=0x7f099800c490 comp rx=0 tx=0).stop 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a5a63640 1 -- 172.21.15.43:0/1298543093 reap_dead start 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a5a63640 1 -- 172.21.15.43:0/1298543093 reap_dead start 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 shutdown_connections 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 >> 172.21.15.43:0/1298543093 conn(0x7f09a00ab2f0 msgr2=0x7f09a00a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 shutdown_connections 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 -- 172.21.15.43:0/1298543093 wait complete. 2024-04-19T00:56:23.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.101+0000 7f09a74c6640 1 librados: shutdown 2024-04-19T00:56:23.143 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:56:23.143 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-19T00:56:23.143 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b', 'e'] 2024-04-19T00:56:23.143 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-19T00:56:23.143 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-04-19T00:56:23.143 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.261+0000 7ff983f82640 10 monclient: get_monmap_and_config 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.261+0000 7ff983f82640 10 monclient: build_initial_monmap 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.261+0000 7ff983f82640 10 monclient: monmap: 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:23.264417+0000 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:23.264417+0000 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:23.261 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:23.262 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.261+0000 7ff983f82640 1 Processor -- start 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.261+0000 7ff983f82640 1 -- start start 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.261+0000 7ff983f82640 10 monclient: init 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient: _add_conns ranks=[2,10,7,1,0,3,4,9,8,6,5] 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 0x7ff97c135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): picked mon.noname-b con 0x7ff97c09d790 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff97c09db70 0x7ff97c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): picked mon.noname-f con 0x7ff97c09db70 addr v2:172.21.15.161:3300/0 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff97c09dfb0 0x7ff97c0a3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): picked mon.noname-i con 0x7ff97c09dfb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:23.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff97c076860 con 0x7ff97c09db70 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff97c09e480 con 0x7ff97c09d790 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff97c0a1110 con 0x7ff97c09dfb0 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient(hunting): authenticate will time out at 1653.392270s 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 0x7ff97c135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 0x7ff97c135fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:58274/0 (socket says 172.21.15.43:58274) 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 -- 172.21.15.43:0/3066848395 learned_addr learned my addr 172.21.15.43:0/3066848395 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): get_auth_request con 0x7ff97c09d790 auth_method 0 2024-04-19T00:56:23.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98151d640 1 --2- 172.21.15.43:0/3066848395 >> v2:172.21.15.161:3300/0 conn(0x7ff97c09db70 0x7ff97c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98251f640 1 --2- 172.21.15.43:0/3066848395 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff97c09dfb0 0x7ff97c0a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98251f640 10 monclient(hunting): get_auth_request con 0x7ff97c09dfb0 auth_method 0 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98251f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98251f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98251f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient(hunting): handle_auth_done global_id 74477 payload 274 2024-04-19T00:56:23.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 -- 172.21.15.43:0/3066848395 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff97c09dfb0 msgr2=0x7ff97c0a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 --2- 172.21.15.43:0/3066848395 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff97c09dfb0 0x7ff97c0a3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 -- 172.21.15.43:0/3066848395 >> v2:172.21.15.161:3300/0 conn(0x7ff97c09db70 msgr2=0x7ff97c0a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 --2- 172.21.15.43:0/3066848395 >> v2:172.21.15.161:3300/0 conn(0x7ff97c09db70 0x7ff97c0a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient: _finish_hunting 0 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 monclient: found mon.noname-b 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 -- 172.21.15.43:0/3066848395 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97c0a13f0 con 0x7ff97c09d790 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98151d640 10 monclient: get_auth_request con 0x7ff97c09db70 auth_method 0 2024-04-19T00:56:23.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff98151d640 1 --2- 172.21.15.43:0/3066848395 >> v2:172.21.15.161:3300/0 conn(0x7ff97c09db70 0x7ff97c0a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff981d1e640 1 --2- 172.21.15.43:0/3066848395 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 0x7ff97c135fb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bb40 tx=0x7ff97801ead0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc900d5c787da4f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 1 -- 172.21.15.43:0/3066848395 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9780b4070 con 0x7ff97c09d790 2024-04-19T00:56:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: dump: 2024-04-19T00:56:23.275 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.276 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: _finish_auth 0 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: _check_auth_tickets 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 1 -- 172.21.15.43:0/3066848395 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9780a7a90 con 0x7ff97c09d790 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 5 monclient: authenticate success, global_id 74477 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 1 -- 172.21.15.43:0/3066848395 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9780a7cc0 con 0x7ff97c09d790 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 4 set_mon_vals no callback set 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff980d1c640 10 monclient: dump: 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.277 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 10 monclient: shutdown 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- 172.21.15.43:0/3066848395 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 msgr2=0x7ff97c135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- 172.21.15.43:0/3066848395 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 0x7ff97c135fb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bb40 tx=0x7ff97801ead0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- 172.21.15.43:0/3066848395 shutdown_connections 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- 172.21.15.43:0/3066848395 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff97c09dfb0 0x7ff97c0a3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- 172.21.15.43:0/3066848395 >> v2:172.21.15.161:3300/0 conn(0x7ff97c09db70 0x7ff97c0a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 --2- 172.21.15.43:0/3066848395 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff97c09d790 0x7ff97c135fb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- 172.21.15.43:0/3066848395 >> 172.21.15.43:0/3066848395 conn(0x7ff97c0a4040 msgr2=0x7ff97c0a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- 172.21.15.43:0/3066848395 shutdown_connections 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.265+0000 7ff983f82640 1 -- 172.21.15.43:0/3066848395 wait complete. 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: build_initial_monmap 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: monmap: 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:23.279 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 librados: starting msgr at 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 librados: starting objecter 2024-04-19T00:56:23.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 Processor -- start 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- start start 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 librados: setting wanted keys 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 librados: calling monclient init 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: init 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: _add_conns ranks=[4,5,6,1,2,7,3,8,0] 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c09d790 0x7ff97c1ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): picked mon.noname-c con 0x7ff97c09d790 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff97c09db70 0x7ff97c1cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): picked mon.noname-h con 0x7ff97c09db70 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff97c09dfb0 0x7ff97c1cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): picked mon.noname-d con 0x7ff97c09dfb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff97c09f5e0 con 0x7ff97c09d790 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ff97c09f460 con 0x7ff97c09db70 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ff97c09f760 con 0x7ff97c09dfb0 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): authenticate will time out at 1653.397205s 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c09d790 0x7ff97c1ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff97c09dfb0 0x7ff97c1cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c09d790 0x7ff97c1ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:53886/0 (socket says 172.21.15.43:53886) 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 -- 172.21.15.43:0/2921607616 learned_addr learned my addr 172.21.15.43:0/2921607616 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): get_auth_request con 0x7ff97c09d790 auth_method 0 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): get_auth_request con 0x7ff97c09dfb0 auth_method 0 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff97c09db70 0x7ff97c1cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): get_auth_request con 0x7ff97c09db70 auth_method 0 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient(hunting): handle_auth_done global_id 64687 payload 824 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff97c09dfb0 msgr2=0x7ff97c1cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff97c09dfb0 0x7ff97c1cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff97c09db70 msgr2=0x7ff97c1cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff97c09db70 0x7ff97c1cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient: _finish_hunting 0 2024-04-19T00:56:23.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 monclient: found mon.noname-c 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97c0a0600 con 0x7ff97c09d790 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff97c09dfb0 0x7ff97c1cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff981d1e640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c09d790 0x7ff97c1ccb80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff97801e900 tx=0x7ff97801b870 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 client.?.objecter ms_handle_connect 0x7ff97c09d790 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9780b4040 con 0x7ff97c09d790 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: dump: 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.285 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: _finish_auth 0 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: _check_auth_tickets 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9780043f0 con 0x7ff97c09d790 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 5 monclient: authenticate success, global_id 64687 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: _renew_subs 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97c0a0040 con 0x7ff97c09d790 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9780af440 con 0x7ff97c09d790 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff973fff640 4 set_mon_vals no callback set 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 client.64687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 monclient: dump: 2024-04-19T00:56:23.287 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.288 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: _renew_subs 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff97c071f80 con 0x7ff97c09d790 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 librados: init done 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 0x7ff97c070480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff97c074f70 con 0x7ff97c0700a0 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff978003620 con 0x7ff97c09d790 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 0x7ff97c070480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient: get_auth_request con 0x7ff97c0700a0 auth_method 0 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff960038e10 0x7ff96003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7ff9780e6190 con 0x7ff97c09d790 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 client.64687.objecter ms_dispatch 0x7ff97c000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 3 client.64687.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 3 client.64687.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 10 monclient(hunting): handle_auth_done global_id 64696 payload 824 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 20 client.64687.objecter dump_active .. 0 homeless 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff97c0693b0 con 0x7ff97c0700a0 2024-04-19T00:56:23.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98151d640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 0x7ff97c070480 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff97c074200 tx=0x7ff96c00f930 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 client.64687.objecter ms_handle_connect 0x7ff97c0700a0 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 10 client.64687.objecter resend_mon_ops 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff960038e10 0x7ff96003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff98251f640 10 monclient: get_auth_request con 0x7ff960038e10 auth_method 0 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.269+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff96c018070 con 0x7ff97c0700a0 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 10 monclient: dump: 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.291 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.292 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff96c014690 con 0x7ff97c0700a0 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 msgr2=0x7ff97c070480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 0x7ff97c070480 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff97c074200 tx=0x7ff96c00f930 comp rx=0 tx=0).stop 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 msgr2=0x7ff97c070480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0700a0 0x7ff97c070480 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff97c074200 tx=0x7ff96c00f930 comp rx=0 tx=0).stop 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff98251f640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff960038e10 0x7ff96003b2d0 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7ff97c09ce80 tx=0x7ff974009820 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.273+0000 7ff9727fc640 10 client.64687.objecter ms_handle_connect 0x7ff960038e10 2024-04-19T00:56:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff983f82640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:56:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff983f82640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 0x7ff97c0a5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff983f82640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff97c06d140 con 0x7ff97c0709c0 2024-04-19T00:56:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 0x7ff97c0a5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:23.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient: get_auth_request con 0x7ff97c0709c0 auth_method 0 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 10 monclient(hunting): handle_auth_done global_id 64705 payload 824 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.277+0000 7ff98151d640 1 -- 172.21.15.43:0/2921607616 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7ff97c06d890 con 0x7ff97c0709c0 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 client.64687.objecter ms_handle_connect 0x7ff97c0709c0 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 client.64687.objecter resend_mon_ops 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff98151d640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 0x7ff97c0a5440 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff96c008010 tx=0x7ff96c0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff96c02d080 con 0x7ff97c0709c0 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:23.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 monclient: dump: 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:23.297 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7ff96c003910 con 0x7ff97c0709c0 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 msgr2=0x7ff97c0a5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 0x7ff97c0a5440 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff96c008010 tx=0x7ff96c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 msgr2=0x7ff97c0a5440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:23.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff9727fc640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c0709c0 0x7ff97c0a5440 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff96c008010 tx=0x7ff96c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:23.299 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:23.299 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:56:23.299 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:56:23.299 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:23.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 10 librados: watch_flush enter 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 10 librados: watch_flush exit 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 20 client.64687.objecter shutdown clearing up homeless session... 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 10 client.64687.objecter successfully canceled tick 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff960038e10 msgr2=0x7ff96003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff960038e10 0x7ff96003b2d0 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7ff97c09ce80 tx=0x7ff974009820 comp rx=0 tx=0).stop 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 10 monclient: shutdown 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c09d790 msgr2=0x7ff97c1ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 --2- 172.21.15.43:0/2921607616 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff97c09d790 0x7ff97c1ccb80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff97801e900 tx=0x7ff97801b870 comp rx=0 tx=0).stop 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff98251f640 1 -- 172.21.15.43:0/2921607616 reap_dead start 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff98251f640 1 -- 172.21.15.43:0/2921607616 reap_dead start 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 shutdown_connections 2024-04-19T00:56:23.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 >> 172.21.15.43:0/2921607616 conn(0x7ff97c0a4040 msgr2=0x7ff97c097890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 shutdown_connections 2024-04-19T00:56:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 -- 172.21.15.43:0/2921607616 wait complete. 2024-04-19T00:56:23.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:23.281+0000 7ff983f82640 1 librados: shutdown 2024-04-19T00:56:23.302 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T00:56:23.302 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T00:56:23.302 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:56:23.317 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T00:56:23.317 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-04-19T00:56:23.317 INFO:tasks.ceph.mon.b:Sent signal 19 2024-04-19T00:56:23.317 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-04-19T00:56:23.318 INFO:tasks.ceph.mon.e:Sent signal 19 2024-04-19T00:56:23.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:56:38.319 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-04-19T00:56:38.320 INFO:tasks.ceph.mon.b:Sent signal 18 2024-04-19T00:56:38.320 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-04-19T00:56:38.320 INFO:tasks.ceph.mon.e:Sent signal 18 2024-04-19T00:56:38.320 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 8 2024-04-19T00:56:38.320 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:56:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient: get_monmap_and_config 2024-04-19T00:56:38.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient: build_initial_monmap 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient: monmap: 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:38.455880+0000 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:38.455880+0000 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:38.453 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:38.454 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:38.455 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:38.455 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:38.455 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:38.455 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:38.455 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:38.455 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:38.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 Processor -- start 2024-04-19T00:56:38.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 -- start start 2024-04-19T00:56:38.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient: init 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient: _add_conns ranks=[3,5,2,4,0,9,10,6,8,7,1] 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): picked mon.noname-g con 0x7f7c8c138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): picked mon.noname-h con 0x7f7c8c0a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): picked mon.noname-b con 0x7f7c8c1358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c0a4410 con 0x7f7c8c1358b0 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c139b30 con 0x7f7c8c138cd0 2024-04-19T00:56:38.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7c8c0a3dc0 con 0x7f7c8c0a5b80 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c933ff640 10 monclient(hunting): authenticate will time out at 1668.583030s 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:39198/0 (socket says 172.21.15.43:39198) 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 learned_addr learned my addr 172.21.15.43:0/3850039974 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 10 monclient(hunting): get_auth_request con 0x7f7c8c1358b0 auth_method 0 2024-04-19T00:56:38.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 10 monclient(hunting): get_auth_request con 0x7f7c8c138cd0 auth_method 0 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:38.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 10 monclient(hunting): get_auth_request con 0x7f7c8c0a5b80 auth_method 0 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:38.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.453+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 10 monclient(hunting): get_auth_request con 0x7f7c8c1358b0 auth_method 0 2024-04-19T00:56:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:38.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 10 monclient(hunting): get_auth_request con 0x7f7c8c138cd0 auth_method 0 2024-04-19T00:56:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:38.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:56:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 10 monclient(hunting): get_auth_request con 0x7f7c8c0a5b80 auth_method 0 2024-04-19T00:56:38.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:56:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:56:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:38.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:56:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:56:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:38.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:38.657+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:56:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 10 monclient(hunting): get_auth_request con 0x7f7c8c1358b0 auth_method 0 2024-04-19T00:56:39.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 10 monclient(hunting): get_auth_request con 0x7f7c8c138cd0 auth_method 0 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 10 monclient(hunting): get_auth_request con 0x7f7c8c0a5b80 auth_method 0 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:39.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:56:39.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.061+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 10 monclient(hunting): get_auth_request con 0x7f7c8c1358b0 auth_method 0 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 10 monclient(hunting): get_auth_request con 0x7f7c8c138cd0 auth_method 0 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 10 monclient(hunting): get_auth_request con 0x7f7c8c0a5b80 auth_method 0 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:56:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:56:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:56:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:56:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:56:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:39.865+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): tick 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 monclient(hunting): continuing hunt 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 msgr2=0x7f7c8c1395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c8c0a5b80 0x7f7c8c1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 msgr2=0x7f7c8c1390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c8c138cd0 0x7f7c8c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 msgr2=0x7f7c8c135d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c8c1358b0 0x7f7c8c135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:41.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient: _add_conns ranks=[1,0,4,10,7,8,6,9] 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7c78003830 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c78007430 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 0x7f7c7800d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): picked mon.noname-c con 0x7f7c7800b160 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 -- 172.21.15.43:0/3850039974 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8c0a40a0 con 0x7f7c78007430 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 -- 172.21.15.43:0/3850039974 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7800ecf0 con 0x7f7c78003830 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 1 -- 172.21.15.43:0/3850039974 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f7c8c0a4410 con 0x7f7c7800b160 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c7effd640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 msgr2=0x7f7c7800d620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:41.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.453+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 0x7f7c7800d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.657+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 msgr2=0x7f7c7800d620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:41.655 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:41.657+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 0x7f7c7800d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:56:42.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:42.061+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 msgr2=0x7f7c7800d620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:42.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:42.061+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 0x7f7c7800d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:56:42.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:42.865+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 msgr2=0x7f7c7800d620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:42.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:42.865+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 0x7f7c7800d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:56:43.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.361+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384467531 0 0) 0x7f7c8c0a40a0 con 0x7f7c78007430 2024-04-19T00:56:43.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.361+0000 7f7c7ffff640 10 monclient(hunting): _negotiate 2024-04-19T00:56:43.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.361+0000 7f7c7ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.361+0000 7f7c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.361+0000 7f7c7ffff640 10 monclient(hunting): my global_id is 64899 2024-04-19T00:56:43.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.361+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c70003530 con 0x7f7c78007430 2024-04-19T00:56:43.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.365+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3627158009 0 0) 0x7f7c7800ecf0 con 0x7f7c78003830 2024-04-19T00:56:43.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.365+0000 7f7c7ffff640 10 monclient(hunting): _negotiate 2024-04-19T00:56:43.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.365+0000 7f7c7ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.365+0000 7f7c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.365+0000 7f7c7ffff640 10 monclient(hunting): my global_id is 114193 2024-04-19T00:56:43.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.365+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c8c0a40a0 con 0x7f7c78003830 2024-04-19T00:56:43.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.389+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 940343316 0 0) 0x7f7c70003530 con 0x7f7c78007430 2024-04-19T00:56:43.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 msgr2=0x7f7c7800d620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:43.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c7800b160 0x7f7c7800d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 >> v1:172.21.15.161:6789/0 conn(0x7f7c78003830 legacy=0x7f7c78005c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 reap_dead start 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7800fff0 con 0x7f7c78007430 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f7c88009bc0 con 0x7f7c78007430 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c933ff640 5 monclient: authenticate success, global_id 64899 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c933ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:43.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: dump: 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.393 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 >> v1:172.21.15.43:6789/0 conn(0x7f7c78007430 legacy=0x7f7c780098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient: _add_conns ranks=[4,3,7,0,6,5,1,2,8] 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c700043a0 0x7f7c70004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): picked mon.c con 0x7f7c700043a0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c70004f20 0x7f7c700099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): picked mon.g con 0x7f7c70004f20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c70009f00 0x7f7c7000c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): picked mon.i con 0x7f7c70009f00 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:43.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c0a4410 con 0x7f7c70004f20 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f7c7000cb10 con 0x7f7c700043a0 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9119b640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c700043a0 msgr2=0x7f7c70004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7c7000ccc0 con 0x7f7c70009f00 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9119b640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c700043a0 0x7f7c70004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c7ffff640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:56:43.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c933ff640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c70009f00 0x7f7c7000c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c70004f20 0x7f7c700099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): get_auth_request con 0x7f7c70009f00 auth_method 0 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): get_auth_request con 0x7f7c70004f20 auth_method 0 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient(hunting): handle_auth_done global_id 64899 payload 293 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c700043a0 msgr2=0x7f7c70004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c700043a0 0x7f7c70004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c70004f20 msgr2=0x7f7c700099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c70004f20 0x7f7c700099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 monclient: found mon.i 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T00:56:43.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 1 -- 172.21.15.43:0/3850039974 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c70003530 con 0x7f7c70009f00 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9199c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.393+0000 7f7c9099a640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c70004f20 0x7f7c700099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c9199c640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c70009f00 0x7f7c7000c2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c139cb0 tx=0x7f7c8401ce70 comp rx=0 tx=0).ready entity=mon.7 client_cookie=66f57847df830d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c840ab070 con 0x7f7c70009f00 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: dump: 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.400 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c840af400 con 0x7f7c70009f00 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:43.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 1 -- 172.21.15.43:0/3850039974 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c8400f040 con 0x7f7c70009f00 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c7ffff640 10 monclient: dump: 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.402 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 4 set_mon_vals no callback set 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 10 monclient: shutdown 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 -- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c70009f00 msgr2=0x7f7c7000c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c70009f00 0x7f7c7000c2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c139cb0 tx=0x7f7c8401ce70 comp rx=0 tx=0).stop 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 -- 172.21.15.43:0/3850039974 shutdown_connections 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7c70004f20 0x7f7c700099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c70009f00 0x7f7c7000c2f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 --2- 172.21.15.43:0/3850039974 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c700043a0 0x7f7c70004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 -- 172.21.15.43:0/3850039974 >> 172.21.15.43:0/3850039974 conn(0x7f7c8c0ab2f0 msgr2=0x7f7c8c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 -- 172.21.15.43:0/3850039974 shutdown_connections 2024-04-19T00:56:43.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.397+0000 7f7c933ff640 1 -- 172.21.15.43:0/3850039974 wait complete. 2024-04-19T00:56:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: build_initial_monmap 2024-04-19T00:56:43.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: monmap: 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:43.409 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 librados: starting msgr at 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 librados: starting objecter 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 Processor -- start 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 -- start start 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 librados: setting wanted keys 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 librados: calling monclient init 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: init 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: _add_conns ranks=[0,2,6,1,4,7,8,3,5] 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c700043a0 0x7f7c8c134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): picked mon.noname-a con 0x7f7c700043a0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 0x7f7c8c134dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): picked mon.noname-b con 0x7f7c70004f20 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:43.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 0x7f7c8c121770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): picked mon.noname-d con 0x7f7c70009f00 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c135c90 con 0x7f7c700043a0 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c0a20a0 con 0x7f7c70004f20 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7c8c135e10 con 0x7f7c70009f00 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient(hunting): authenticate will time out at 1673.529552s 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c700043a0 0x7f7c8c134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 0x7f7c8c121770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 0x7f7c8c134dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 0x7f7c8c121770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:60114/0 (socket says 172.21.15.43:60114) 2024-04-19T00:56:43.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 0x7f7c8c134dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:51918/0 (socket says 172.21.15.43:51918) 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 -- 172.21.15.43:0/1565064813 learned_addr learned my addr 172.21.15.43:0/1565064813 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c700043a0 0x7f7c8c134890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50072/0 (socket says 172.21.15.43:50072) 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): get_auth_request con 0x7f7c70009f00 auth_method 0 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): get_auth_request con 0x7f7c700043a0 auth_method 0 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): get_auth_request con 0x7f7c70004f20 auth_method 0 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9119b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient(hunting): handle_auth_done global_id 74472 payload 824 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 -- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 msgr2=0x7f7c8c134dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 0x7f7c8c134dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 -- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c700043a0 msgr2=0x7f7c8c134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c700043a0 0x7f7c8c134890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 monclient: found mon.noname-d 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 -- 172.21.15.43:0/1565064813 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c0a1280 con 0x7f7c70009f00 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9099a640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 0x7f7c8c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c9199c640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 0x7f7c8c121770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7c84000c10 tx=0x7f7c84019d10 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 client.?.objecter ms_handle_connect 0x7f7c70009f00 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c840ab040 con 0x7f7c70009f00 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: dump: 2024-04-19T00:56:43.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c840be070 con 0x7f7c70009f00 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c840100b0 con 0x7f7c70009f00 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 5 monclient: authenticate success, global_id 74472 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: _renew_subs 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8c0a3590 con 0x7f7c70009f00 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:43.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7dffb640 10 monclient: dump: 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.417 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c7f7fe640 4 set_mon_vals no callback set 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 client.74472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: _renew_subs 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8c123f90 con 0x7f7c70009f00 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.401+0000 7f7c933ff640 1 librados: init done 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c933ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c933ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c933ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c8c0693b0 con 0x7f7c70009f00 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7c840c2410 con 0x7f7c70009f00 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c78039d00 0x7f7c7803c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(440..440 src has 1..440) ==== 6226+0+0 (secure 0 0 0) 0x7f7c840e44a0 con 0x7f7c70009f00 2024-04-19T00:56:43.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 10 client.74472.objecter ms_dispatch 0x7f7c8c000ba0 osd_map(440..440 src has 1..440) 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 3 client.74472.objecter handle_osd_map got epochs [440,440] > 0 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 3 client.74472.objecter handle_osd_map decoding full epoch 440 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.405+0000 7f7c7dffb640 20 client.74472.objecter dump_active .. 0 homeless 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.409+0000 7f7c9119b640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c78039d00 0x7f7c7803c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.409+0000 7f7c9119b640 10 monclient: get_auth_request con 0x7f7c78039d00 auth_method 0 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.409+0000 7f7c7dffb640 10 client.74472.objecter ms_handle_connect 0x7f7c78039d00 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.409+0000 7f7c9119b640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c78039d00 0x7f7c7803c1c0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7f7c80000c10 tx=0x7f7c800112e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.413+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7c84016cc0 con 0x7f7c70009f00 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.413+0000 7f7c7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.413+0000 7f7c7dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.641+0000 7f7c933ff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:56:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.641+0000 7f7c933ff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:56:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.641+0000 7f7c933ff640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:43.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.641+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7c8c122860 con 0x7f7c70009f00 2024-04-19T00:56:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c7dffb640 1 -- 172.21.15.43:0/1565064813 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3186 (secure 0 0 0) 0x7f7c840e1a00 con 0x7f7c70009f00 2024-04-19T00:56:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:56:43.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c7dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:43.643 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":90,"quorum":[0,1,2,3,5,6,7,8],"quorum_names":["a","f","b","g","h","d","i","e"],"quorum_leader_name":"g","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 10 librados: watch_flush enter 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 10 librados: watch_flush exit 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 20 client.74472.objecter shutdown clearing up homeless session... 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 10 client.74472.objecter successfully canceled tick 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c78039d00 msgr2=0x7f7c7803c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c78039d00 0x7f7c7803c1c0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7f7c80000c10 tx=0x7f7c800112e0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 10 monclient: shutdown 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 msgr2=0x7f7c8c121770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 0x7f7c8c121770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7c84000c10 tx=0x7f7c84019d10 comp rx=0 tx=0).stop 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 shutdown_connections 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c78039d00 0x7f7c7803c1c0 unknown :-1 s=CLOSED pgs=3541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f7c70009f00 0x7f7c8c121770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7c70004f20 0x7f7c8c134dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 --2- 172.21.15.43:0/1565064813 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c700043a0 0x7f7c8c134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 >> 172.21.15.43:0/1565064813 conn(0x7f7c8c0ab2f0 msgr2=0x7f7c8c1c1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 shutdown_connections 2024-04-19T00:56:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 -- 172.21.15.43:0/1565064813 wait complete. 2024-04-19T00:56:43.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.645+0000 7f7c933ff640 1 librados: shutdown 2024-04-19T00:56:43.661 INFO:tasks.mon_thrash.ceph_manager:quorum is size 8 2024-04-19T00:56:43.661 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon a 2024-04-19T00:56:43.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:56:43.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient: get_monmap_and_config 2024-04-19T00:56:43.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient: build_initial_monmap 2024-04-19T00:56:43.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient: monmap: 2024-04-19T00:56:43.787 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:43.790843+0000 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:43.790843+0000 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:43.788 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:43.789 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:43.789 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:43.789 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 Processor -- start 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 -- start start 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient: init 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient: _add_conns ranks=[9,1,5,7,8,6,3,0,4,10,2] 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4cd0138cd0 0x7f4cd01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): picked mon.noname-a con 0x7f4cd0138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f4cd00a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4cd01395f0 0x7f4cd0135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): picked mon.noname-h con 0x7f4cd01395f0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cd00a3dc0 con 0x7f4cd00a5b80 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f4cd00a4410 con 0x7f4cd0138cd0 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4cd00a40a0 con 0x7f4cd01395f0 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd68f2640 10 monclient(hunting): authenticate will time out at 1673.918474s 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4cd0138cd0 0x7f4cd01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4cd0138cd0 0x7f4cd01390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50080/0 (socket says 172.21.15.43:50080) 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 -- 172.21.15.43:0/3670555176 learned_addr learned my addr 172.21.15.43:0/3670555176 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:43.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): get_auth_request con 0x7f4cd0138cd0 auth_method 0 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd4e8f640 1 --2- 172.21.15.43:0/3670555176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4cd01395f0 0x7f4cd0135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd4e8f640 10 monclient(hunting): get_auth_request con 0x7f4cd01395f0 auth_method 0 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd4e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd4e8f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4cd4e8f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient(hunting): handle_auth_done global_id 64917 payload 274 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 -- 172.21.15.43:0/3670555176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4cd01395f0 msgr2=0x7f4cd0135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 --2- 172.21.15.43:0/3670555176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4cd01395f0 0x7f4cd0135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 -- 172.21.15.43:0/3670555176 >> v1:172.21.15.161:6789/0 conn(0x7f4cd00a5b80 legacy=0x7f4cd01358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 monclient: found mon.noname-a 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:43.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.789+0000 7f4ccffff640 1 -- 172.21.15.43:0/3670555176 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd00a2ae0 con 0x7f4cd0138cd0 2024-04-19T00:56:43.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4ccffff640 1 --2- 172.21.15.43:0/3670555176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0138cd0 0x7f4cd01390b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009870 tx=0x7f4cc001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becca2535e3a778a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 1 -- 172.21.15.43:0/3670555176 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cc00ab8b0 con 0x7f4cd0138cd0 2024-04-19T00:56:43.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:43.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: dump: 2024-04-19T00:56:43.801 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.802 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.803 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 1 -- 172.21.15.43:0/3670555176 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc00b0070 con 0x7f4cd0138cd0 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 5 monclient: authenticate success, global_id 64917 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 1 -- 172.21.15.43:0/3670555176 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cc00aa3a0 con 0x7f4cd0138cd0 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 4 set_mon_vals no callback set 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cceffd640 10 monclient: dump: 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.804 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: shutdown 2024-04-19T00:56:43.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- 172.21.15.43:0/3670555176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0138cd0 msgr2=0x7f4cd01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/3670555176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0138cd0 0x7f4cd01390b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4cc0009870 tx=0x7f4cc001c420 comp rx=0 tx=0).stop 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- 172.21.15.43:0/3670555176 shutdown_connections 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/3670555176 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4cd01395f0 0x7f4cd0135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/3670555176 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0138cd0 0x7f4cd01390b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- 172.21.15.43:0/3670555176 >> 172.21.15.43:0/3670555176 conn(0x7f4cd00ab2f0 msgr2=0x7f4cd00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- 172.21.15.43:0/3670555176 shutdown_connections 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- 172.21.15.43:0/3670555176 wait complete. 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: build_initial_monmap 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: monmap: 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:43.806 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 librados: starting msgr at 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 librados: starting objecter 2024-04-19T00:56:43.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 Processor -- start 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- start start 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 librados: setting wanted keys 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 librados: calling monclient init 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: init 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient: _add_conns ranks=[0,1,8,2,6,4,5,7,3] 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd00a5b80 0x7f4cd00a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): picked mon.noname-a con 0x7f4cd00a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cd0138cd0 0x7f4cd00a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): picked mon.noname-f con 0x7f4cd0138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cd01395f0 0x7f4cd0134290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): picked mon.noname-e con 0x7f4cd01395f0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd0136500 con 0x7f4cd00a5b80 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd00a20a0 con 0x7f4cd0138cd0 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4cd0136680 con 0x7f4cd01395f0 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.793+0000 7f4cd68f2640 10 monclient(hunting): authenticate will time out at 1673.923919s 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd00a5b80 0x7f4cd00a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd00a5b80 0x7f4cd00a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50086/0 (socket says 172.21.15.43:50086) 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cd01395f0 0x7f4cd0134290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 -- 172.21.15.43:0/2853711623 learned_addr learned my addr 172.21.15.43:0/2853711623 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): get_auth_request con 0x7f4cd00a5b80 auth_method 0 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): get_auth_request con 0x7f4cd01395f0 auth_method 0 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cd0138cd0 0x7f4cd00a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): get_auth_request con 0x7f4cd0138cd0 auth_method 0 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient(hunting): handle_auth_done global_id 64926 payload 824 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cd01395f0 msgr2=0x7f4cd0134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cd01395f0 0x7f4cd0134290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cd0138cd0 msgr2=0x7f4cd00a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4cd0138cd0 0x7f4cd00a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 monclient: found mon.noname-a 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd009be40 con 0x7f4cd00a5b80 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cd01395f0 0x7f4cd0134290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccffff640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd00a5b80 0x7f4cd00a0b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00ad040 tx=0x7f4cc00191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 client.?.objecter ms_handle_connect 0x7f4cd00a5b80 2024-04-19T00:56:43.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cc0003d50 con 0x7f4cd00a5b80 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: dump: 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.812 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 5 monclient: authenticate success, global_id 64926 2024-04-19T00:56:43.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 monclient: _renew_subs 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc00ab440 con 0x7f4cd00a5b80 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd01361e0 con 0x7f4cd00a5b80 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cc00aba60 con 0x7f4cd00a5b80 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 client.64926.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccdffb640 4 set_mon_vals no callback set 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: dump: 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.814 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.815 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 monclient: _renew_subs 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd00a3590 con 0x7f4cd00a5b80 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 1 librados: init done 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 0x7f4cd01347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd006d140 con 0x7f4cd0135dc0 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4cc00abc20 con 0x7f4cd00a5b80 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 0x7f4cd01347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient: get_auth_request con 0x7f4cd0135dc0 auth_method 0 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cbc038e10 0x7f4cbc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(441..441 src has 1..441) ==== 5813+0+0 (secure 0 0 0) 0x7f4cc00e3a70 con 0x7f4cd00a5b80 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 client.64926.objecter ms_dispatch 0x7f4cd0000ba0 osd_map(441..441 src has 1..441) 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 3 client.64926.objecter handle_osd_map got epochs [441,441] > 0 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 3 client.64926.objecter handle_osd_map decoding full epoch 441 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_done global_id 64935 payload 824 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cd01d18f0 con 0x7f4cd0135dc0 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 20 client.64926.objecter dump_active .. 0 homeless 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4ccf7fe640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 0x7f4cd01347d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00a72e0 tx=0x7f4cc400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 client.64926.objecter ms_handle_connect 0x7f4cd0135dc0 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 client.64926.objecter resend_mon_ops 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cc4019070 con 0x7f4cd0135dc0 2024-04-19T00:56:43.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cbc038e10 0x7f4cbc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: dump: 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.818 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 10 monclient: get_auth_request con 0x7f4cbc038e10 auth_method 0 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4cc40052e0 con 0x7f4cd0135dc0 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 msgr2=0x7f4cd01347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 0x7f4cd01347d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00a72e0 tx=0x7f4cc400e960 comp rx=0 tx=0).stop 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 msgr2=0x7f4cd01347d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:43.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd0135dc0 0x7f4cd01347d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cccff9640 10 client.64926.objecter ms_handle_connect 0x7f4cbc038e10 2024-04-19T00:56:43.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.797+0000 7f4cd4e8f640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cbc038e10 0x7f4cbc03b2d0 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0135970 tx=0x7f4cc8009780 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 0x7f4cd0134d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd0068240 con 0x7f4cd01d0c40 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 0x7f4cd0134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient: get_auth_request con 0x7f4cd01d0c40 auth_method 0 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 10 monclient(hunting): handle_auth_done global_id 64944 payload 824 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 1 -- 172.21.15.43:0/2853711623 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cd009cec0 con 0x7f4cd01d0c40 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4ccf7fe640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 0x7f4cd0134d10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00a72e0 tx=0x7f4cc4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 client.64926.objecter ms_handle_connect 0x7f4cd01d0c40 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 client.64926.objecter resend_mon_ops 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4cc4019070 con 0x7f4cd01d0c40 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 monclient: dump: 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.823 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f4cc40067e0 con 0x7f4cd01d0c40 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:43.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 msgr2=0x7f4cd0134d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 0x7f4cd0134d10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00a72e0 tx=0x7f4cc4000ee0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 msgr2=0x7f4cd0134d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cccff9640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd01d0c40 0x7f4cd0134d10 secure :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00a72e0 tx=0x7f4cc4000ee0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 330076, 2024-04-19T00:56:43.825 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:43.826 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:43.827 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.828 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:43.829 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:43.830 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.832 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:43.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:43.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.837 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:43.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:43.842 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.844 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.845 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.846 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:43.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:43.848 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 10 librados: watch_flush enter 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 10 librados: watch_flush exit 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 20 client.64926.objecter shutdown clearing up homeless session... 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 10 client.64926.objecter successfully canceled tick 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cbc038e10 msgr2=0x7f4cbc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cbc038e10 0x7f4cbc03b2d0 secure :-1 s=READY pgs=3542 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0135970 tx=0x7f4cc8009780 comp rx=0 tx=0).stop 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 10 monclient: shutdown 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd00a5b80 msgr2=0x7f4cd00a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 --2- 172.21.15.43:0/2853711623 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4cd00a5b80 0x7f4cd00a0b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4cc00ad040 tx=0x7f4cc00191c0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd4e8f640 1 -- 172.21.15.43:0/2853711623 reap_dead start 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd4e8f640 1 -- 172.21.15.43:0/2853711623 reap_dead start 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 shutdown_connections 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 >> 172.21.15.43:0/2853711623 conn(0x7f4cd00ab2f0 msgr2=0x7f4cd00a8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 shutdown_connections 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 -- 172.21.15.43:0/2853711623 wait complete. 2024-04-19T00:56:43.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.805+0000 7f4cd68f2640 1 librados: shutdown 2024-04-19T00:56:43.851 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:43.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon b 2024-04-19T00:56:43.852 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:56:43.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient: get_monmap_and_config 2024-04-19T00:56:43.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient: build_initial_monmap 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient: monmap: 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:43.971003+0000 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:43.971003+0000 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:43.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:43.969 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:43.969 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:43.969 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:43.969 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:43.969 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 Processor -- start 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 -- start start 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient: init 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient: _add_conns ranks=[0,9,3,4,10,1,7,2,5,8,6] 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ac4138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6ac40a5b80 0x7f6ac413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac40a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T00:56:43.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac41397c0 0x7f6ac41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): picked mon.noname-g con 0x7f6ac41397c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac40a3dc0 con 0x7f6ac4138cd0 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac40a4410 con 0x7f6ac40a5b80 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac40a40a0 con 0x7f6ac41397c0 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6acb7e8640 10 monclient(hunting): authenticate will time out at 1674.098258s 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6ac40a5b80 0x7f6ac413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6ac40a5b80 0x7f6ac413e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50104/0 (socket says 172.21.15.43:50104) 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 -- 172.21.15.43:0/4243589611 learned_addr learned my addr 172.21.15.43:0/4243589611 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): get_auth_request con 0x7f6ac40a5b80 auth_method 0 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac9584640 1 --1- 172.21.15.43:0/4243589611 >> v1:172.21.15.43:6789/0 conn(0x7f6ac4138cd0 0x7f6ac41390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:35242/0 (socket says 172.21.15.43:35242) 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac9d85640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac41397c0 0x7f6ac41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient(hunting): handle_auth_done global_id 64953 payload 274 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 -- 172.21.15.43:0/4243589611 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac41397c0 msgr2=0x7f6ac41407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac41397c0 0x7f6ac41407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 -- 172.21.15.43:0/4243589611 >> v1:172.21.15.43:6789/0 conn(0x7f6ac4138cd0 legacy=0x7f6ac41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 monclient: found mon.noname-a 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac8d83640 1 -- 172.21.15.43:0/4243589611 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac40a2ae0 con 0x7f6ac40a5b80 2024-04-19T00:56:43.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac9d85640 10 monclient: get_auth_request con 0x7f6ac41397c0 auth_method 0 2024-04-19T00:56:43.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.969+0000 7f6ac9d85640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac41397c0 0x7f6ac41407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:43.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac8d83640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ac40a5b80 0x7f6ac413e270 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009870 tx=0x7f6ab801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c7ac54a2e85d20 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 1 -- 172.21.15.43:0/4243589611 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ab80b0070 con 0x7f6ac40a5b80 2024-04-19T00:56:43.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: dump: 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.981 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 1 -- 172.21.15.43:0/4243589611 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab80aa810 con 0x7f6ac40a5b80 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 1 -- 172.21.15.43:0/4243589611 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ab80a9300 con 0x7f6ac40a5b80 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac3fff640 10 monclient: dump: 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.982 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 5 monclient: authenticate success, global_id 64953 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 4 set_mon_vals no callback set 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: shutdown 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- 172.21.15.43:0/4243589611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ac40a5b80 msgr2=0x7f6ac413e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ac40a5b80 0x7f6ac413e270 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f6ab8009870 tx=0x7f6ab801c420 comp rx=0 tx=0).stop 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- 172.21.15.43:0/4243589611 shutdown_connections 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac41397c0 0x7f6ac41407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/4243589611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6ac40a5b80 0x7f6ac413e270 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- 172.21.15.43:0/4243589611 >> 172.21.15.43:0/4243589611 conn(0x7f6ac40ab2f0 msgr2=0x7f6ac40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- 172.21.15.43:0/4243589611 shutdown_connections 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- 172.21.15.43:0/4243589611 wait complete. 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: build_initial_monmap 2024-04-19T00:56:43.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: monmap: 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:43.985 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 librados: starting msgr at 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 librados: starting objecter 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 Processor -- start 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- start start 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 librados: setting wanted keys 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 librados: calling monclient init 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: init 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient: _add_conns ranks=[7,3,2,1,6,4,8,0,5] 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6ac40a5b80 0x7f6ac41d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): picked mon.noname-i con 0x7f6ac40a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:43.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac4138cd0 0x7f6ac4136b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): picked mon.noname-g con 0x7f6ac4138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 0x7f6ac409d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac41397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac4143730 con 0x7f6ac41397c0 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac40a20a0 con 0x7f6ac4138cd0 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6ac41438b0 con 0x7f6ac40a5b80 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6acb7e8640 10 monclient(hunting): authenticate will time out at 1674.103002s 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 0x7f6ac409d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 0x7f6ac409d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:51928/0 (socket says 172.21.15.43:51928) 2024-04-19T00:56:43.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 -- 172.21.15.43:0/2942578691 learned_addr learned my addr 172.21.15.43:0/2942578691 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): get_auth_request con 0x7f6ac41397c0 auth_method 0 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient(hunting): handle_auth_done global_id 74495 payload 824 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6ac40a5b80 msgr2=0x7f6ac41d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6ac40a5b80 0x7f6ac41d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac8d83640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac4138cd0 0x7f6ac4136b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac4138cd0 msgr2=0x7f6ac4136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6ac4138cd0 0x7f6ac4136b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient: _finish_hunting 0 2024-04-19T00:56:43.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 monclient: found mon.noname-b 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac40a3040 con 0x7f6ac41397c0 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac9d85640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 0x7f6ac409d8c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c820 tx=0x7f6ab400cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 client.?.objecter ms_handle_connect 0x7f6ac41397c0 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ab4015020 con 0x7f6ac41397c0 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: dump: 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.989 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: _finish_auth 0 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:43.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ab400ae00 con 0x7f6ac41397c0 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6ab4004240 con 0x7f6ac41397c0 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac1ffb640 10 monclient: dump: 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.991 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.973+0000 7f6ac37fe640 4 set_mon_vals no callback set 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 5 monclient: authenticate success, global_id 74495 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 monclient: _renew_subs 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac409f540 con 0x7f6ac41397c0 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 client.74495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 monclient: _renew_subs 2024-04-19T00:56:43.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac40a0f90 con 0x7f6ac41397c0 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 1 librados: init done 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 0x7f6ac402c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac406d140 con 0x7f6ac409e5c0 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6ab4018020 con 0x7f6ac41397c0 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 0x7f6ac402c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient: get_auth_request con 0x7f6ac409e5c0 auth_method 0 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ab0038e10 0x7f6ab003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 10 monclient(hunting): handle_auth_done global_id 74504 payload 824 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ac40a3590 con 0x7f6ac409e5c0 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(441..441 src has 1..441) ==== 5813+0+0 (secure 0 0 0) 0x7f6ab4049350 con 0x7f6ac41397c0 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 client.74495.objecter ms_dispatch 0x7f6ac4000ba0 osd_map(441..441 src has 1..441) 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 3 client.74495.objecter handle_osd_map got epochs [441,441] > 0 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 3 client.74495.objecter handle_osd_map decoding full epoch 441 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 20 client.74495.objecter dump_active .. 0 homeless 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac9584640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 0x7f6ac402c450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a72e0 tx=0x7f6abc00cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 client.74495.objecter ms_handle_connect 0x7f6ac409e5c0 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 client.74495.objecter resend_mon_ops 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6abc017070 con 0x7f6ac409e5c0 2024-04-19T00:56:43.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 monclient: dump: 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:43.995 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac8d83640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ab0038e10 0x7f6ab003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac8d83640 10 monclient: get_auth_request con 0x7f6ab0038e10 auth_method 0 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6abc005ba0 con 0x7f6ac409e5c0 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 msgr2=0x7f6ac402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 0x7f6ac402c450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a72e0 tx=0x7f6abc00cb10 comp rx=0 tx=0).stop 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 msgr2=0x7f6ac402c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e5c0 0x7f6ac402c450 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a72e0 tx=0x7f6abc00cb10 comp rx=0 tx=0).stop 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac1ffb640 10 client.74495.objecter ms_handle_connect 0x7f6ab0038e10 2024-04-19T00:56:43.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.977+0000 7f6ac8d83640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ab0038e10 0x7f6ab003b2d0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80ad000 tx=0x7f6ab80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6acb7e8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:43.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 0x7f6ac402c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6acb7e8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac4068240 con 0x7f6ac409e9a0 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 0x7f6ac402c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient: get_auth_request con 0x7f6ac409e9a0 auth_method 0 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.981+0000 7f6ac9584640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac9584640 10 monclient(hunting): handle_auth_done global_id 74513 payload 824 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac9584640 1 -- 172.21.15.43:0/2942578691 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ac40693b0 con 0x7f6ac409e9a0 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac9584640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 0x7f6ac402c990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a72e0 tx=0x7f6abc001260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:43.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 client.74495.objecter ms_handle_connect 0x7f6ac409e9a0 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 client.74495.objecter resend_mon_ops 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6abc017040 con 0x7f6ac409e9a0 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 monclient: dump: 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.000 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f6abc00a610 con 0x7f6ac409e9a0 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 msgr2=0x7f6ac402c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 0x7f6ac402c990 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a72e0 tx=0x7f6abc001260 comp rx=0 tx=0).stop 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 msgr2=0x7f6ac402c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac1ffb640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac409e9a0 0x7f6ac402c990 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a72e0 tx=0x7f6abc001260 comp rx=0 tx=0).stop 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 183606, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:44.002 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.003 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.004 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.005 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:44.007 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.008 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.010 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.011 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.013 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.016 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.018 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.021 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.022 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:44.023 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 10 librados: watch_flush enter 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 10 librados: watch_flush exit 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 20 client.74495.objecter shutdown clearing up homeless session... 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 10 client.74495.objecter successfully canceled tick 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ab0038e10 msgr2=0x7f6ab003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6ab0038e10 0x7f6ab003b2d0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7f6ab80ad000 tx=0x7f6ab80a6000 comp rx=0 tx=0).stop 2024-04-19T00:56:44.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 10 monclient: shutdown 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 msgr2=0x7f6ac409d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 0x7f6ac409d8c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6ab400c820 tx=0x7f6ab400cce0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6ac9d85640 1 -- 172.21.15.43:0/2942578691 reap_dead start 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 shutdown_connections 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 --2- 172.21.15.43:0/2942578691 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6ac41397c0 0x7f6ac409d8c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 >> 172.21.15.43:0/2942578691 conn(0x7f6ac40ab2f0 msgr2=0x7f6ac40a9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 shutdown_connections 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 -- 172.21.15.43:0/2942578691 wait complete. 2024-04-19T00:56:44.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:43.985+0000 7f6acb7e8640 1 librados: shutdown 2024-04-19T00:56:44.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:44.027 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon d 2024-04-19T00:56:44.027 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient: get_monmap_and_config 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient: monmap: 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:44.151457+0000 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:44.151457+0000 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.148 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:44.149 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 Processor -- start 2024-04-19T00:56:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 -- start start 2024-04-19T00:56:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient: init 2024-04-19T00:56:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient: _add_conns ranks=[4,6,2,7,10,5,1,8,9,0,3] 2024-04-19T00:56:44.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec880a6420 0x7fec880a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): picked mon.noname-c con 0x7fec880a6420 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec880a6800 0x7fec880a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): picked mon.noname-d con 0x7fec880a6800 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 0x7fec88138fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): picked mon.noname-b con 0x7fec88138b50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fec8809f340 con 0x7fec88138b50 2024-04-19T00:56:44.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fec880a20f0 con 0x7fec880a6420 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fec8809f540 con 0x7fec880a6800 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec90019640 10 monclient(hunting): authenticate will time out at 1674.278619s 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8ddb5640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec880a6420 msgr2=0x7fec880a5bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8ddb5640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec880a6420 0x7fec880a5bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 0x7fec88138fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 0x7fec88138fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:51952/0 (socket says 172.21.15.43:51952) 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 -- 172.21.15.43:0/1272496252 learned_addr learned my addr 172.21.15.43:0/1272496252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): get_auth_request con 0x7fec88138b50 auth_method 0 2024-04-19T00:56:44.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec880a6800 0x7fec880a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): get_auth_request con 0x7fec880a6800 auth_method 0 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_done global_id 74522 payload 274 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 -- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec880a6800 msgr2=0x7fec880a6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec880a6800 0x7fec880a6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 -- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec880a6420 msgr2=0x7fec880a5bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec880a6420 0x7fec880a5bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 monclient: found mon.noname-b 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.149+0000 7fec8e5b6640 1 -- 172.21.15.43:0/1272496252 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec880a07f0 con 0x7fec88138b50 2024-04-19T00:56:44.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8e5b6640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 0x7fec88138fd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec840098a0 tx=0x7fec8401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c61a7d0951a71b4b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 1 -- 172.21.15.43:0/1272496252 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec840b0070 con 0x7fec88138b50 2024-04-19T00:56:44.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: dump: 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.166 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 1 -- 172.21.15.43:0/1272496252 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec840044a0 con 0x7fec88138b50 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 5 monclient: authenticate success, global_id 74522 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:44.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 4 set_mon_vals no callback set 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 1 -- 172.21.15.43:0/1272496252 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec84003620 con 0x7fec88138b50 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec8cdb3640 10 monclient: dump: 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.168 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 10 monclient: shutdown 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 -- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 msgr2=0x7fec88138fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 0x7fec88138fd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fec840098a0 tx=0x7fec8401c430 comp rx=0 tx=0).stop 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 -- 172.21.15.43:0/1272496252 shutdown_connections 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fec88138b50 0x7fec88138fd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec880a6800 0x7fec880a6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 --2- 172.21.15.43:0/1272496252 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fec880a6420 0x7fec880a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 -- 172.21.15.43:0/1272496252 >> 172.21.15.43:0/1272496252 conn(0x7fec880ab1b0 msgr2=0x7fec880ab5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 -- 172.21.15.43:0/1272496252 shutdown_connections 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.153+0000 7fec90019640 1 -- 172.21.15.43:0/1272496252 wait complete. 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: monmap: 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.170 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 librados: starting msgr at 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 librados: starting objecter 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 Processor -- start 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 -- start start 2024-04-19T00:56:44.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 librados: setting wanted keys 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 librados: calling monclient init 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: init 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: _add_conns ranks=[5,7,1,2,4,8,6,0,3] 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec880a6420 0x7fec881373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): picked mon.noname-h con 0x7fec880a6420 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec880a6800 0x7fec881378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): picked mon.noname-i con 0x7fec880a6800 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec88138b50 0x7fec8813d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): picked mon.noname-f con 0x7fec88138b50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fec88142820 con 0x7fec88138b50 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fec881426a0 con 0x7fec880a6420 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fec881429a0 con 0x7fec880a6800 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient(hunting): authenticate will time out at 1674.285828s 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec880a6420 0x7fec881373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec88138b50 0x7fec8813d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec880a6800 0x7fec881378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec880a6420 0x7fec881373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:58408/0 (socket says 172.21.15.43:58408) 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 -- 172.21.15.43:0/2429970790 learned_addr learned my addr 172.21.15.43:0/2429970790 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): get_auth_request con 0x7fec880a6420 auth_method 0 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): get_auth_request con 0x7fec880a6800 auth_method 0 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): get_auth_request con 0x7fec88138b50 auth_method 0 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8e5b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient(hunting): handle_auth_done global_id 74408 payload 824 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec880a6800 msgr2=0x7fec881378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fec880a6800 0x7fec881378f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec88138b50 msgr2=0x7fec8813d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec88138b50 0x7fec8813d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 monclient: found mon.noname-h 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec88123280 con 0x7fec880a6420 2024-04-19T00:56:44.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec8ddb5640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec880a6420 0x7fec881373b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec8000ba10 tx=0x7fec8000bed0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 client.?.objecter ms_handle_connect 0x7fec880a6420 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec8000cde0 con 0x7fec880a6420 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: dump: 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.176 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.177 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 5 monclient: authenticate success, global_id 74408 2024-04-19T00:56:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: _renew_subs 2024-04-19T00:56:44.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:56:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8809fd90 con 0x7fec880a6420 2024-04-19T00:56:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80010070 con 0x7fec880a6420 2024-04-19T00:56:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec80015470 con 0x7fec880a6420 2024-04-19T00:56:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 client.74408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7e7fc640 10 monclient: dump: 2024-04-19T00:56:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.180 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.181 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.182 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.182 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: _renew_subs 2024-04-19T00:56:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec7ffff640 4 set_mon_vals no callback set 2024-04-19T00:56:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:56:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8813f400 con 0x7fec880a6420 2024-04-19T00:56:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.157+0000 7fec90019640 1 librados: init done 2024-04-19T00:56:44.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec90019640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec90019640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 0x7fec8813e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fec8806d140 con 0x7fec8813dcc0 2024-04-19T00:56:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 0x7fec8813e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient: get_auth_request con 0x7fec8813dcc0 auth_method 0 2024-04-19T00:56:44.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_done global_id 74490 payload 824 2024-04-19T00:56:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec880693b0 con 0x7fec8813dcc0 2024-04-19T00:56:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 client.74408.objecter ms_handle_connect 0x7fec8813dcc0 2024-04-19T00:56:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8d5b4640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 0x7fec8813e0a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fec880ac1f0 tx=0x7fec78011d60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 client.74408.objecter resend_mon_ops 2024-04-19T00:56:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fec8000c970 con 0x7fec880a6420 2024-04-19T00:56:44.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec6c038e10 0x7fec6c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(441..441 src has 1..441) ==== 5813+0+0 (secure 0 0 0) 0x7fec800493a0 con 0x7fec880a6420 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 client.74408.objecter ms_dispatch 0x7fec88000ba0 osd_map(441..441 src has 1..441) 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 3 client.74408.objecter handle_osd_map got epochs [441,441] > 0 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 3 client.74408.objecter handle_osd_map decoding full epoch 441 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 20 client.74408.objecter dump_active .. 0 homeless 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec78004030 con 0x7fec8813dcc0 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 monclient: dump: 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.188 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.189 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec7800fb60 con 0x7fec8813dcc0 2024-04-19T00:56:44.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 msgr2=0x7fec8813e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 0x7fec8813e0a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fec880ac1f0 tx=0x7fec78011d60 comp rx=0 tx=0).stop 2024-04-19T00:56:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 msgr2=0x7fec8813e0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813dcc0 0x7fec8813e0a0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fec880ac1f0 tx=0x7fec78011d60 comp rx=0 tx=0).stop 2024-04-19T00:56:44.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8e5b6640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec6c038e10 0x7fec6c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8e5b6640 10 monclient: get_auth_request con 0x7fec6c038e10 auth_method 0 2024-04-19T00:56:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec7e7fc640 10 client.74408.objecter ms_handle_connect 0x7fec6c038e10 2024-04-19T00:56:44.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.161+0000 7fec8e5b6640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec6c038e10 0x7fec6c03b2d0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7fec840ad000 tx=0x7fec840a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.165+0000 7fec90019640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.165+0000 7fec90019640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 0x7fec8809da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fec88068240 con 0x7fec8813e5e0 2024-04-19T00:56:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 0x7fec8809da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient: get_auth_request con 0x7fec8813e5e0 auth_method 0 2024-04-19T00:56:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 10 monclient(hunting): handle_auth_done global_id 74499 payload 824 2024-04-19T00:56:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 1 -- 172.21.15.43:0/2429970790 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec88076190 con 0x7fec8813e5e0 2024-04-19T00:56:44.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8d5b4640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 0x7fec8809da80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fec880ac1f0 tx=0x7fec780046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 client.74408.objecter ms_handle_connect 0x7fec8813e5e0 2024-04-19T00:56:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 client.74408.objecter resend_mon_ops 2024-04-19T00:56:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fec7800fea0 con 0x7fec8813e5e0 2024-04-19T00:56:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:44.199 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 monclient: dump: 2024-04-19T00:56:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.200 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.201 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.202 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7fec78003e30 con 0x7fec8813e5e0 2024-04-19T00:56:44.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 msgr2=0x7fec8809da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 0x7fec8809da80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fec880ac1f0 tx=0x7fec780046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 msgr2=0x7fec8809da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec7e7fc640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fec8813e5e0 0x7fec8809da80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.204 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:44.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:44.204 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:44.204 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:44.205 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:44.205 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 183782, 2024-04-19T00:56:44.205 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:44.205 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:44.205 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:44.205 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:44.206 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:44.206 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:44.206 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:44.206 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:44.206 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.207 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.208 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.208 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.208 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.208 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.208 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.209 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.210 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:44.211 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:44.212 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.213 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.215 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.217 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.219 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.220 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:44.221 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.223 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.224 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:44.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:44.229 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.230 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.231 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.232 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:56:44.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.234 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:44.234 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 10 librados: watch_flush enter 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 10 librados: watch_flush exit 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 20 client.74408.objecter shutdown clearing up homeless session... 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 10 client.74408.objecter successfully canceled tick 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec6c038e10 msgr2=0x7fec6c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fec6c038e10 0x7fec6c03b2d0 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7fec840ad000 tx=0x7fec840a6000 comp rx=0 tx=0).stop 2024-04-19T00:56:44.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 10 monclient: shutdown 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec880a6420 msgr2=0x7fec881373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 --2- 172.21.15.43:0/2429970790 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fec880a6420 0x7fec881373b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fec8000ba10 tx=0x7fec8000bed0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8e5b6640 1 -- 172.21.15.43:0/2429970790 reap_dead start 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec8e5b6640 1 -- 172.21.15.43:0/2429970790 reap_dead start 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 shutdown_connections 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 >> 172.21.15.43:0/2429970790 conn(0x7fec880ab1b0 msgr2=0x7fec880a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 shutdown_connections 2024-04-19T00:56:44.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 -- 172.21.15.43:0/2429970790 wait complete. 2024-04-19T00:56:44.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.169+0000 7fec90019640 1 librados: shutdown 2024-04-19T00:56:44.239 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:44.239 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon e 2024-04-19T00:56:44.240 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:56:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient: get_monmap_and_config 2024-04-19T00:56:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient: monmap: 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:44.371861+0000 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:44.371861+0000 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.369 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:44.370 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:44.370 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 Processor -- start 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 -- start start 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient: init 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient: _add_conns ranks=[1,4,2,6,0,9,5,8,7,10,3] 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f23b8138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b80a5b80 0x7f23b813e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): picked mon.noname-c con 0x7f23b80a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:44.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): picked mon.noname-b con 0x7f23b81397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23b80a3dc0 con 0x7f23b8138cd0 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f23b80a4410 con 0x7f23b81397c0 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f23b80a40a0 con 0x7f23b80a5b80 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23c03a8640 10 monclient(hunting): authenticate will time out at 1674.499491s 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23bd943640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b80a5b80 msgr2=0x7f23b813e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23bd943640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b80a5b80 0x7f23b813e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:51964/0 (socket says 172.21.15.43:51964) 2024-04-19T00:56:44.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 1 -- 172.21.15.43:0/2405238252 learned_addr learned my addr 172.21.15.43:0/2405238252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): get_auth_request con 0x7f23b81397c0 auth_method 0 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.369+0000 7f23be945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 10 monclient(hunting): handle_auth_done global_id 74531 payload 274 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 1 -- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b80a5b80 msgr2=0x7f23b813e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 1 --2- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b80a5b80 0x7f23b813e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 1 -- 172.21.15.43:0/2405238252 >> v1:172.21.15.161:6789/0 conn(0x7f23b8138cd0 legacy=0x7f23b81390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 1 monclient: found mon.noname-b 2024-04-19T00:56:44.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 1 -- 172.21.15.43:0/2405238252 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b80a2ae0 con 0x7f23b81397c0 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23be945640 1 --2- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81407c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009e80 tx=0x7f23b001c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01c69bdd256414b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 1 -- 172.21.15.43:0/2405238252 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23b80a20a0 con 0x7f23b81397c0 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: dump: 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.382 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 1 -- 172.21.15.43:0/2405238252 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23b00090d0 con 0x7f23b81397c0 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 1 -- 172.21.15.43:0/2405238252 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23b00a5c40 con 0x7f23b81397c0 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 5 monclient: authenticate success, global_id 74531 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 4 set_mon_vals no callback set 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23bd142640 10 monclient: dump: 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.384 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 10 monclient: shutdown 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 -- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 msgr2=0x7f23b81407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81407c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f23b0009e80 tx=0x7f23b001c930 comp rx=0 tx=0).stop 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 -- 172.21.15.43:0/2405238252 shutdown_connections 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81407c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2405238252 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b80a5b80 0x7f23b813e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 -- 172.21.15.43:0/2405238252 >> 172.21.15.43:0/2405238252 conn(0x7f23b80ab2f0 msgr2=0x7f23b80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 -- 172.21.15.43:0/2405238252 shutdown_connections 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 -- 172.21.15.43:0/2405238252 wait complete. 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 10 monclient: monmap: 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:44.386 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 librados: starting msgr at 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 librados: starting objecter 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.373+0000 7f23c03a8640 1 Processor -- start 2024-04-19T00:56:44.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- start start 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 librados: setting wanted keys 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 librados: calling monclient init 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: init 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: _add_conns ranks=[1,4,2,0,6,5,3,8,7] 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23b80a5b80 0x7f23b813ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): picked mon.noname-f con 0x7f23b80a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b8138cd0 0x7f23b81d0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): picked mon.noname-c con 0x7f23b8138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): picked mon.noname-b con 0x7f23b81397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f23b81438b0 con 0x7f23b80a5b80 2024-04-19T00:56:44.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f23b8143730 con 0x7f23b81397c0 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f23b8143a30 con 0x7f23b8138cd0 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): authenticate will time out at 1674.504529s 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23bd943640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b8138cd0 msgr2=0x7f23b81d0940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23bd943640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b8138cd0 0x7f23b81d0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81d0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:51976/0 (socket says 172.21.15.43:51976) 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 -- 172.21.15.43:0/2247599298 learned_addr learned my addr 172.21.15.43:0/2247599298 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): get_auth_request con 0x7f23b81397c0 auth_method 0 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23b80a5b80 0x7f23b813ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): get_auth_request con 0x7f23b80a5b80 auth_method 0 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient(hunting): handle_auth_done global_id 74540 payload 824 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b8138cd0 msgr2=0x7f23b81d0940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f23b8138cd0 0x7f23b81d0940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23b80a5b80 msgr2=0x7f23b813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f23b80a5b80 0x7f23b813ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 monclient: found mon.noname-b 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b813e680 con 0x7f23b81397c0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be945640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81d0e80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f23b001afe0 tx=0x7f23b00196d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 client.?.objecter ms_handle_connect 0x7f23b81397c0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23b00bc020 con 0x7f23b81397c0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: dump: 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.391 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23b00a5e50 con 0x7f23b81397c0 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23b00044b0 con 0x7f23b81397c0 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 5 monclient: authenticate success, global_id 74540 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: _renew_subs 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b80a3040 con 0x7f23b81397c0 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23bc941640 4 set_mon_vals no callback set 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: dump: 2024-04-19T00:56:44.393 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.394 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 client.74540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: _renew_subs 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b813cbb0 con 0x7f23b81397c0 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 librados: init done 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 0x7f23b813af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f23b806d140 con 0x7f23b81d13c0 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 0x7f23b813af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient: get_auth_request con 0x7f23b81d13c0 auth_method 0 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f23b00bf020 con 0x7f23b81397c0 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f239c038e10 0x7f239c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 10 monclient(hunting): handle_auth_done global_id 114263 payload 824 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(441..441 src has 1..441) ==== 5813+0+0 (secure 0 0 0) 0x7f23b00b3040 con 0x7f23b81397c0 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 client.74540.objecter ms_dispatch 0x7f23b8000ba0 osd_map(441..441 src has 1..441) 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 3 client.74540.objecter handle_osd_map got epochs [441,441] > 0 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 3 client.74540.objecter handle_osd_map decoding full epoch 441 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23b80a3590 con 0x7f23b81d13c0 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 20 client.74540.objecter dump_active .. 0 homeless 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 client.74540.objecter ms_handle_connect 0x7f23b81d13c0 2024-04-19T00:56:44.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 client.74540.objecter resend_mon_ops 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23be144640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 0x7f23b813af50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f23b80a72e0 tx=0x7f23b400e940 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23bd943640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f239c038e10 0x7f239c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23bd943640 10 monclient: get_auth_request con 0x7f239c038e10 auth_method 0 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23b40052e0 con 0x7f23b81d13c0 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: dump: 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.397 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23b40052e0 con 0x7f23b81d13c0 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:44.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 msgr2=0x7f23b813af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 0x7f23b813af50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f23b80a72e0 tx=0x7f23b400e940 comp rx=0 tx=0).stop 2024-04-19T00:56:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 msgr2=0x7f23b813af50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b81d13c0 0x7f23b813af50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23aeffd640 10 client.74540.objecter ms_handle_connect 0x7f239c038e10 2024-04-19T00:56:44.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.377+0000 7f23bd943640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f239c038e10 0x7f239c03b2d0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f23b8073ff0 tx=0x7f23a8006cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 0x7f23b802c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f23b8068240 con 0x7f23b802c450 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 0x7f23b802c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient: get_auth_request con 0x7f23b802c450 auth_method 0 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 10 monclient(hunting): handle_auth_done global_id 114272 payload 824 2024-04-19T00:56:44.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 1 -- 172.21.15.43:0/2247599298 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23b81d17a0 con 0x7f23b802c450 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be144640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 0x7f23b802c860 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f23b80a72e0 tx=0x7f23b4001130 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 client.74540.objecter ms_handle_connect 0x7f23b802c450 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 client.74540.objecter resend_mon_ops 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23b401c760 con 0x7f23b802c450 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 monclient: dump: 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f23b4006ac0 con 0x7f23b802c450 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 msgr2=0x7f23b802c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 0x7f23b802c860 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f23b80a72e0 tx=0x7f23b4001130 comp rx=0 tx=0).stop 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 msgr2=0x7f23b802c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23aeffd640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23b802c450 0x7f23b802c860 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f23b80a72e0 tx=0x7f23b4001130 comp rx=0 tx=0).stop 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 41547, 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:44.404 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.405 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.406 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:44.407 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:44.408 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.409 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:44.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.414 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.417 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:44.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.423 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.424 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:44.425 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 10 librados: watch_flush enter 2024-04-19T00:56:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 10 librados: watch_flush exit 2024-04-19T00:56:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 20 client.74540.objecter shutdown clearing up homeless session... 2024-04-19T00:56:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 10 client.74540.objecter successfully canceled tick 2024-04-19T00:56:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f239c038e10 msgr2=0x7f239c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f239c038e10 0x7f239c03b2d0 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f23b8073ff0 tx=0x7f23a8006cc0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 10 monclient: shutdown 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 msgr2=0x7f23b81d0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 --2- 172.21.15.43:0/2247599298 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23b81397c0 0x7f23b81d0e80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f23b001afe0 tx=0x7f23b00196d0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be945640 1 -- 172.21.15.43:0/2247599298 reap_dead start 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23be945640 1 -- 172.21.15.43:0/2247599298 reap_dead start 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 shutdown_connections 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 >> 172.21.15.43:0/2247599298 conn(0x7f23b80ab2f0 msgr2=0x7f23b8137360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 shutdown_connections 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 -- 172.21.15.43:0/2247599298 wait complete. 2024-04-19T00:56:44.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.385+0000 7f23c03a8640 1 librados: shutdown 2024-04-19T00:56:44.429 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:44.429 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon f 2024-04-19T00:56:44.429 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient: get_monmap_and_config 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient: monmap: 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:44.546582+0000 2024-04-19T00:56:44.543 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:44.546582+0000 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:44.544 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 Processor -- start 2024-04-19T00:56:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 -- start start 2024-04-19T00:56:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient: init 2024-04-19T00:56:44.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient: _add_conns ranks=[8,9,4,2,10,5,3,6,0,7,1] 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 0x7f21780a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): picked mon.noname-e con 0x7f21780a1ec0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f21780a22a0 0x7f21780a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): picked mon.noname-a con 0x7f21780a22a0 addr v2:172.21.15.43:3300/0 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f21780a12e0 0x7f21780a4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): picked mon.noname-c con 0x7f21780a12e0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f217806c4b0 con 0x7f21780a22a0 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f217806bfc0 con 0x7f21780a12e0 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f21780a0060 con 0x7f21780a1ec0 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217e38b640 10 monclient(hunting): authenticate will time out at 1674.673713s 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217c928640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f21780a12e0 msgr2=0x7f21780a4430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 0x7f21780a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f217c928640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f21780a12e0 0x7f21780a4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 0x7f21780a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:37022/0 (socket says 172.21.15.43:37022) 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f21780a22a0 0x7f21780a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 1 -- 172.21.15.43:0/2007823240 learned_addr learned my addr 172.21.15.43:0/2007823240 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): get_auth_request con 0x7f21780a1ec0 auth_method 0 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): get_auth_request con 0x7f21780a22a0 auth_method 0 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient(hunting): handle_auth_done global_id 64971 payload 274 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 -- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 msgr2=0x7f21780a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 0x7f21780a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 -- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f21780a12e0 msgr2=0x7f21780a4430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f21780a12e0 0x7f21780a4430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 monclient: found mon.noname-a 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f21777fe640 1 -- 172.21.15.43:0/2007823240 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21780a0340 con 0x7f21780a22a0 2024-04-19T00:56:44.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.545+0000 7f2177fff640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 0x7f21780a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:56:44.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21780a22a0 0x7f21780a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f21600097c0 tx=0x7f216001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c535a29d73becd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 1 -- 172.21.15.43:0/2007823240 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21600b0070 con 0x7f21780a22a0 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: dump: 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.559 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 1 -- 172.21.15.43:0/2007823240 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21600aa810 con 0x7f21780a22a0 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 5 monclient: authenticate success, global_id 64971 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 4 set_mon_vals no callback set 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 1 -- 172.21.15.43:0/2007823240 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21600a9300 con 0x7f21780a22a0 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2176ffd640 10 monclient: dump: 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.561 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: shutdown 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21780a22a0 msgr2=0x7f21780a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21780a22a0 0x7f21780a3ef0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f21600097c0 tx=0x7f216001c380 comp rx=0 tx=0).stop 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/2007823240 shutdown_connections 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f21780a12e0 0x7f21780a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f21780a22a0 0x7f21780a3ef0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- 172.21.15.43:0/2007823240 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f21780a1ec0 0x7f21780a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/2007823240 >> 172.21.15.43:0/2007823240 conn(0x7f21780a8600 msgr2=0x7f21780a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/2007823240 shutdown_connections 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/2007823240 wait complete. 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: monmap: 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:44.563 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 librados: starting msgr at 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 librados: starting objecter 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 Processor -- start 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- start start 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 librados: setting wanted keys 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 librados: calling monclient init 2024-04-19T00:56:44.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: init 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: _add_conns ranks=[5,2,6,3,4,1,8,7,0] 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f21780a12e0 0x7f21781d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): picked mon.noname-h con 0x7f21780a12e0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21780a1ec0 0x7f21781d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): picked mon.noname-b con 0x7f21780a1ec0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f217809bb90 0x7f21781d1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): picked mon.noname-d con 0x7f217809bb90 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2178142b60 con 0x7f21780a1ec0 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f21781429e0 con 0x7f21780a12e0 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2178142ce0 con 0x7f217809bb90 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient(hunting): authenticate will time out at 1674.678455s 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21780a1ec0 0x7f21781d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f217809bb90 0x7f21781d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21780a1ec0 0x7f21781d13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:51986/0 (socket says 172.21.15.43:51986) 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 -- 172.21.15.43:0/1941453131 learned_addr learned my addr 172.21.15.43:0/1941453131 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): get_auth_request con 0x7f21780a1ec0 auth_method 0 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f217809bb90 0x7f21781d1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:60146/0 (socket says 172.21.15.43:60146) 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): get_auth_request con 0x7f217809bb90 auth_method 0 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217c928640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient(hunting): handle_auth_done global_id 74549 payload 824 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f217809bb90 msgr2=0x7f21781d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f217809bb90 0x7f21781d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f21780a12e0 msgr2=0x7f21781d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f21780a12e0 0x7f21781d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.567 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 monclient: found mon.noname-b 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217809d4c0 con 0x7f21780a1ec0 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f21777fe640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21780a1ec0 0x7f21781d13c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f216001cc20 tx=0x7f2160019120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 client.?.objecter ms_handle_connect 0x7f21780a1ec0 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21600b0040 con 0x7f21780a1ec0 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: dump: 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.568 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21600b9070 con 0x7f21780a1ec0 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 5 monclient: authenticate success, global_id 74549 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: _renew_subs 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217809d7a0 con 0x7f21780a1ec0 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21600044b0 con 0x7f21780a1ec0 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2174ff9640 10 monclient: dump: 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.570 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f2175ffb640 4 set_mon_vals no callback set 2024-04-19T00:56:44.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 client.74549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: _renew_subs 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f217809e720 con 0x7f21780a1ec0 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.549+0000 7f217e38b640 1 librados: init done 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217e38b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217e38b640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 0x7f2178132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f217806d140 con 0x7f21780a22a0 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f21600aac80 con 0x7f21780a1ec0 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2158038e00 0x7f215803b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(442..442 src has 1..442) ==== 6185+0+0 (secure 0 0 0) 0x7f21600e2f60 con 0x7f21780a1ec0 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 client.74549.objecter ms_dispatch 0x7f2178000ba0 osd_map(442..442 src has 1..442) 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 3 client.74549.objecter handle_osd_map got epochs [442,442] > 0 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 3 client.74549.objecter handle_osd_map decoding full epoch 442 2024-04-19T00:56:44.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 0x7f2178132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient: get_auth_request con 0x7f21780a22a0 auth_method 0 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 20 client.74549.objecter dump_active .. 0 homeless 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217c928640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2158038e00 0x7f215803b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217c928640 10 monclient: get_auth_request con 0x7f2158038e00 auth_method 0 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 10 monclient(hunting): handle_auth_done global_id 114238 payload 824 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f217809ef70 con 0x7f21780a22a0 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2177fff640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 0x7f2178132b00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2170009c00 tx=0x7f217000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 client.74549.objecter ms_handle_connect 0x7f21780a22a0 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 client.74549.objecter resend_mon_ops 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 client.74549.objecter ms_handle_connect 0x7f2158038e00 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f217c928640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2158038e00 0x7f215803b2c0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7f2178069560 tx=0x7f2168009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2170014480 con 0x7f21780a22a0 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 monclient: dump: 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.574 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2170019070 con 0x7f21780a22a0 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 msgr2=0x7f2178132b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 0x7f2178132b00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2170009c00 tx=0x7f217000c490 comp rx=0 tx=0).stop 2024-04-19T00:56:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 msgr2=0x7f2178132b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.553+0000 7f2174ff9640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21780a22a0 0x7f2178132b00 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2170009c00 tx=0x7f217000c490 comp rx=0 tx=0).stop 2024-04-19T00:56:44.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 0x7f2178133450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2178133990 con 0x7f2178133040 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 0x7f2178133450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient: get_auth_request con 0x7f2178133040 auth_method 0 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 10 monclient(hunting): handle_auth_done global_id 114247 payload 824 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 1 -- 172.21.15.43:0/1941453131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f217809d140 con 0x7f2178133040 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 client.74549.objecter ms_handle_connect 0x7f2178133040 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2177fff640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 0x7f2178133450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2170009c90 tx=0x7f2170004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 client.74549.objecter resend_mon_ops 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2170014b00 con 0x7f2178133040 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 monclient: dump: 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.579 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9261 (secure 0 0 0) 0x7f2170014690 con 0x7f2178133040 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:44.580 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 msgr2=0x7f2178133450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 0x7f2178133450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2170009c90 tx=0x7f2170004a50 comp rx=0 tx=0).stop 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 msgr2=0x7f2178133450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f2174ff9640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2178133040 0x7f2178133450 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2170009c90 tx=0x7f2170004a50 comp rx=0 tx=0).stop 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 41727, 2024-04-19T00:56:44.581 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:44.582 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:44.609 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:44.609 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.609 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:44.609 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.609 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.610 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:44.612 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.613 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.615 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.616 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:44.617 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.618 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:44.619 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.620 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.621 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.622 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:44.623 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.624 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:44.625 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.626 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:44.627 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.628 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:44.629 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:44.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 10 librados: watch_flush enter 2024-04-19T00:56:44.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 10 librados: watch_flush exit 2024-04-19T00:56:44.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 20 client.74549.objecter shutdown clearing up homeless session... 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 10 client.74549.objecter successfully canceled tick 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2158038e00 msgr2=0x7f215803b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2158038e00 0x7f215803b2c0 secure :-1 s=READY pgs=3546 cs=0 l=1 rev1=1 crypto rx=0x7f2178069560 tx=0x7f2168009060 comp rx=0 tx=0).stop 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 10 monclient: shutdown 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21780a1ec0 msgr2=0x7f21781d13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 --2- 172.21.15.43:0/1941453131 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f21780a1ec0 0x7f21781d13c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f216001cc20 tx=0x7f2160019120 comp rx=0 tx=0).stop 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217c928640 1 -- 172.21.15.43:0/1941453131 reap_dead start 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217c928640 1 -- 172.21.15.43:0/1941453131 reap_dead start 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 shutdown_connections 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 >> 172.21.15.43:0/1941453131 conn(0x7f21780a8600 msgr2=0x7f2178070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 shutdown_connections 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 -- 172.21.15.43:0/1941453131 wait complete. 2024-04-19T00:56:44.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.561+0000 7f217e38b640 1 librados: shutdown 2024-04-19T00:56:44.633 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:44.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon g 2024-04-19T00:56:44.633 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:56:44.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient: get_monmap_and_config 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient: monmap: 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:44.759544+0000 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:44.759544+0000 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.757 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:44.758 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 Processor -- start 2024-04-19T00:56:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 -- start start 2024-04-19T00:56:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient: init 2024-04-19T00:56:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient: _add_conns ranks=[1,7,5,2,9,3,0,10,8,4,6] 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa040138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa0400a5b80 0x7fa04013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): picked mon.noname-i con 0x7fa0400a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0401397c0 0x7fa0401407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): picked mon.noname-h con 0x7fa0401397c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0400a3dc0 con 0x7fa040138cd0 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa0400a4410 con 0x7fa0401397c0 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa0400a40a0 con 0x7fa0400a5b80 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa044f62640 10 monclient(hunting): authenticate will time out at 1674.886737s 2024-04-19T00:56:44.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa040138cd0 0x7fa0401390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:51256/0 (socket says 172.21.15.43:51256) 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa0400a5b80 0x7fa04013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03e59c640 1 -- 172.21.15.43:0/797780772 learned_addr learned my addr 172.21.15.43:0/797780772 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 1 --2- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0401397c0 0x7fa0401407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): get_auth_request con 0x7fa0400a5b80 auth_method 0 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): get_auth_request con 0x7fa0401397c0 auth_method 0 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_done global_id 74374 payload 274 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 1 -- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0401397c0 msgr2=0x7fa0401407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 1 --2- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0401397c0 0x7fa0401407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 1 -- 172.21.15.43:0/797780772 >> v1:172.21.15.161:6789/0 conn(0x7fa040138cd0 legacy=0x7fa0401390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 1 monclient: found mon.noname-i 2024-04-19T00:56:44.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03d59a640 1 -- 172.21.15.43:0/797780772 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2803239307 0 0) 0x7fa0400a3dc0 con 0x7fa040138cd0 2024-04-19T00:56:44.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:56:44.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03dd9b640 1 -- 172.21.15.43:0/797780772 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0400a2ae0 con 0x7fa0400a5b80 2024-04-19T00:56:44.764 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.757+0000 7fa03d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:56:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03dd9b640 1 --2- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa0400a5b80 0x7fa04013e270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa0340097f0 tx=0x7fa03401c3a0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=193834b10d3629af server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 1 -- 172.21.15.43:0/797780772 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0340b0070 con 0x7fa0400a5b80 2024-04-19T00:56:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: dump: 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.772 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 1 -- 172.21.15.43:0/797780772 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0340aa810 con 0x7fa0400a5b80 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 5 monclient: authenticate success, global_id 74374 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 4 set_mon_vals no callback set 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 1 -- 172.21.15.43:0/797780772 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0340a9300 con 0x7fa0400a5b80 2024-04-19T00:56:44.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa03d59a640 10 monclient: dump: 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.774 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 10 monclient: shutdown 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 -- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa0400a5b80 msgr2=0x7fa04013e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 --2- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa0400a5b80 0x7fa04013e270 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa0340097f0 tx=0x7fa03401c3a0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 -- 172.21.15.43:0/797780772 shutdown_connections 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 --2- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0401397c0 0x7fa0401407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 --2- 172.21.15.43:0/797780772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa0400a5b80 0x7fa04013e270 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 -- 172.21.15.43:0/797780772 >> 172.21.15.43:0/797780772 conn(0x7fa0400ab2f0 msgr2=0x7fa0400ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 -- 172.21.15.43:0/797780772 shutdown_connections 2024-04-19T00:56:44.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.761+0000 7fa044f62640 1 -- 172.21.15.43:0/797780772 wait complete. 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: monmap: 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.776 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 librados: starting msgr at 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 librados: starting objecter 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 Processor -- start 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- start start 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 librados: setting wanted keys 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 librados: calling monclient init 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: init 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: _add_conns ranks=[3,6,4,8,1,7,5,0,2] 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0400a5b80 0x7fa0401d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): picked mon.noname-g con 0x7fa0400a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:44.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa040138cd0 0x7fa0401370c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): picked mon.noname-d con 0x7fa040138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa0401397c0 0x7fa04009def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): picked mon.noname-c con 0x7fa0401397c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa040143730 con 0x7fa0400a5b80 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa0400a20a0 con 0x7fa0401397c0 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa0401438b0 con 0x7fa040138cd0 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): authenticate will time out at 1674.893739s 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03ed9d640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa0401397c0 msgr2=0x7fa04009def0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:44.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03ed9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa0401397c0 0x7fa04009def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa040138cd0 0x7fa0401370c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa040138cd0 0x7fa0401370c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:60154/0 (socket says 172.21.15.43:60154) 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 -- 172.21.15.43:0/1577177297 learned_addr learned my addr 172.21.15.43:0/1577177297 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): get_auth_request con 0x7fa040138cd0 auth_method 0 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03e59c640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0400a5b80 0x7fa0401d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03e59c640 10 monclient(hunting): get_auth_request con 0x7fa0400a5b80 auth_method 0 2024-04-19T00:56:44.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient(hunting): handle_auth_done global_id 74517 payload 824 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa0401397c0 msgr2=0x7fa04009def0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa0401397c0 0x7fa04009def0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0400a5b80 msgr2=0x7fa0401d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa0400a5b80 0x7fa0401d08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 monclient: found mon.noname-d 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0400a3040 con 0x7fa040138cd0 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03dd9b640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa040138cd0 0x7fa0401370c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa0340ae040 tx=0x7fa034019140 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 client.?.objecter ms_handle_connect 0x7fa040138cd0 2024-04-19T00:56:44.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0340b0040 con 0x7fa040138cd0 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: dump: 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.781 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 5 monclient: authenticate success, global_id 74517 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0340044b0 con 0x7fa040138cd0 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: _renew_subs 2024-04-19T00:56:44.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0400a3590 con 0x7fa040138cd0 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0340aacf0 con 0x7fa040138cd0 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03cd99640 4 set_mon_vals no callback set 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 client.74517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 10 monclient: dump: 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.783 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: _renew_subs 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa040121a50 con 0x7fa040138cd0 2024-04-19T00:56:44.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 librados: init done 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 0x7fa04002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa04006d140 con 0x7fa04009ebf0 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa034004040 con 0x7fa040138cd0 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.765+0000 7fa03e59c640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 0x7fa04002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient: get_auth_request con 0x7fa04009ebf0 auth_method 0 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa020038e10 0x7fa02003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(442..442 src has 1..442) ==== 6185+0+0 (secure 0 0 0) 0x7fa0340e2f60 con 0x7fa040138cd0 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 client.74517.objecter ms_dispatch 0x7fa040000ba0 osd_map(442..442 src has 1..442) 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 3 client.74517.objecter handle_osd_map got epochs [442,442] > 0 2024-04-19T00:56:44.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 3 client.74517.objecter handle_osd_map decoding full epoch 442 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 20 client.74517.objecter dump_active .. 0 homeless 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03ed9d640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa020038e10 0x7fa02003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03ed9d640 10 monclient: get_auth_request con 0x7fa020038e10 auth_method 0 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 10 monclient(hunting): handle_auth_done global_id 19497 payload 824 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0400693b0 con 0x7fa04009ebf0 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 client.74517.objecter ms_handle_connect 0x7fa04009ebf0 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 client.74517.objecter resend_mon_ops 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03e59c640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 0x7fa04002c450 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa030000c10 tx=0x7fa03000e960 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 client.74517.objecter ms_handle_connect 0x7fa020038e10 2024-04-19T00:56:44.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa03ed9d640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa020038e10 0x7fa02003b2d0 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7fa0400a1350 tx=0x7fa03800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0300070d0 con 0x7fa04009ebf0 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 monclient: dump: 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.787 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa030018070 con 0x7fa04009ebf0 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 msgr2=0x7fa04002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 0x7fa04002c450 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa030000c10 tx=0x7fa03000e960 comp rx=0 tx=0).stop 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 msgr2=0x7fa04002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.769+0000 7fa02effd640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009ebf0 0x7fa04002c450 secure :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa030000c10 tx=0x7fa03000e960 comp rx=0 tx=0).stop 2024-04-19T00:56:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 0x7fa04002c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa040068240 con 0x7fa04009efd0 2024-04-19T00:56:44.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 0x7fa04002c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient: get_auth_request con 0x7fa04009efd0 auth_method 0 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 10 monclient(hunting): handle_auth_done global_id 19506 payload 824 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 1 -- 172.21.15.43:0/1577177297 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa04013ec00 con 0x7fa04009efd0 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03e59c640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 0x7fa04002c990 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa030017000 tx=0x7fa030000ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 client.74517.objecter ms_handle_connect 0x7fa04009efd0 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 client.74517.objecter resend_mon_ops 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa03001dc30 con 0x7fa04009efd0 2024-04-19T00:56:44.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 monclient: dump: 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.792 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9264 (secure 0 0 0) 0x7fa03001d7c0 con 0x7fa04009efd0 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 msgr2=0x7fa04002c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 0x7fa04002c990 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa030017000 tx=0x7fa030000ee0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 msgr2=0x7fa04002c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa02effd640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa04009efd0 0x7fa04002c990 secure :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fa030017000 tx=0x7fa030000ee0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 748260, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:44.794 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.795 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.796 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:44.797 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:44.798 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.799 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.801 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.802 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.806 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:44.807 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.808 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.809 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.809 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.809 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.810 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:44.810 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:44.813 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:44.816 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.817 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:44.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.821 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.821 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:44.821 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:44.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 10 librados: watch_flush enter 2024-04-19T00:56:44.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 10 librados: watch_flush exit 2024-04-19T00:56:44.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 20 client.74517.objecter shutdown clearing up homeless session... 2024-04-19T00:56:44.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 10 client.74517.objecter successfully canceled tick 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa020038e10 msgr2=0x7fa02003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa020038e10 0x7fa02003b2d0 secure :-1 s=READY pgs=3547 cs=0 l=1 rev1=1 crypto rx=0x7fa0400a1350 tx=0x7fa03800b040 comp rx=0 tx=0).stop 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 10 monclient: shutdown 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa040138cd0 msgr2=0x7fa0401370c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 --2- 172.21.15.43:0/1577177297 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa040138cd0 0x7fa0401370c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa0340ae040 tx=0x7fa034019140 comp rx=0 tx=0).stop 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03ed9d640 1 -- 172.21.15.43:0/1577177297 reap_dead start 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa03ed9d640 1 -- 172.21.15.43:0/1577177297 reap_dead start 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 shutdown_connections 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 >> 172.21.15.43:0/1577177297 conn(0x7fa0400ab2f0 msgr2=0x7fa04013bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 shutdown_connections 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 -- 172.21.15.43:0/1577177297 wait complete. 2024-04-19T00:56:44.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.777+0000 7fa044f62640 1 librados: shutdown 2024-04-19T00:56:44.824 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:44.824 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon h 2024-04-19T00:56:44.824 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.945+0000 7f123d1e1640 10 monclient: get_monmap_and_config 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.945+0000 7f123d1e1640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient: monmap: 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:44.950449+0000 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:44.950449+0000 2024-04-19T00:56:44.947 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:44.948 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 Processor -- start 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 -- start start 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient: init 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient: _add_conns ranks=[3,1,10,9,0,8,7,4,5,2,6] 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1238133b20 0x7f123809d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): picked mon.noname-g con 0x7f1238133b20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1238133f00 addr v1:172.21.15.161:6789/0 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1238134300 0x7f12380a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): picked mon.noname-f con 0x7f1238134300 addr v2:172.21.15.161:3300/0 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12380a0f00 con 0x7f1238133f00 2024-04-19T00:56:44.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f123809cf40 con 0x7f1238134300 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f123809c610 con 0x7f1238133b20 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123d1e1640 10 monclient(hunting): authenticate will time out at 1675.077328s 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1238134300 0x7f12380a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1238134300 0x7f12380a3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:38894/0 (socket says 172.21.15.43:38894) 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1238133b20 0x7f123809d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 -- 172.21.15.43:0/1262917052 learned_addr learned my addr 172.21.15.43:0/1262917052 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): get_auth_request con 0x7f1238134300 auth_method 0 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123659c640 1 --1- 172.21.15.43:0/1262917052 >> v1:172.21.15.161:6789/0 conn(0x7f1238133f00 0x7f12380a2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:51268/0 (socket says 172.21.15.43:51268) 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): get_auth_request con 0x7f1238133b20 auth_method 0 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient(hunting): handle_auth_done global_id 114265 payload 274 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 -- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1238133b20 msgr2=0x7f123809d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 --2- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1238133b20 0x7f123809d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 -- 172.21.15.43:0/1262917052 >> v1:172.21.15.161:6789/0 conn(0x7f1238133f00 legacy=0x7f12380a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 monclient: found mon.noname-f 2024-04-19T00:56:44.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:56:44.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.949+0000 7f123759e640 1 -- 172.21.15.43:0/1262917052 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123809c8f0 con 0x7f1238134300 2024-04-19T00:56:44.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123759e640 1 --2- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1238134300 0x7f12380a3040 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1230009e50 tx=0x7f123001c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f9afcc0d237e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 1 -- 172.21.15.43:0/1262917052 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12300090d0 con 0x7f1238134300 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: dump: 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.963 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 1 -- 172.21.15.43:0/1262917052 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12300a5ba0 con 0x7f1238134300 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 5 monclient: authenticate success, global_id 114265 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 4 set_mon_vals no callback set 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 1 -- 172.21.15.43:0/1262917052 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f12300a5d90 con 0x7f1238134300 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f1235d9b640 10 monclient: dump: 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.965 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 10 monclient: shutdown 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 -- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1238134300 msgr2=0x7f12380a3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 --2- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1238134300 0x7f12380a3040 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1230009e50 tx=0x7f123001c8f0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 -- 172.21.15.43:0/1262917052 shutdown_connections 2024-04-19T00:56:44.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 --2- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1238134300 0x7f12380a3040 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 --2- 172.21.15.43:0/1262917052 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1238133b20 0x7f123809d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 -- 172.21.15.43:0/1262917052 >> 172.21.15.43:0/1262917052 conn(0x7f12380a77d0 msgr2=0x7f12380a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 -- 172.21.15.43:0/1262917052 shutdown_connections 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 -- 172.21.15.43:0/1262917052 wait complete. 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 10 monclient: build_initial_monmap 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 10 monclient: monmap: 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:44.967 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 librados: starting msgr at 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 librados: starting objecter 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 Processor -- start 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 -- start start 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 librados: setting wanted keys 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 1 librados: calling monclient init 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.953+0000 7f123d1e1640 10 monclient: init 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient: _add_conns ranks=[0,6,4,5,1,2,8,3,7] 2024-04-19T00:56:44.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1238133b20 0x7f12381cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): picked mon.noname-a con 0x7f1238133b20 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 0x7f123813a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): picked mon.noname-d con 0x7f1238133f00 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1238134300 0x7f123813ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): picked mon.noname-c con 0x7f1238134300 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f12380a3700 con 0x7f1238133b20 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f12380a3580 con 0x7f1238134300 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f12381413f0 con 0x7f1238133f00 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): authenticate will time out at 1675.084176s 2024-04-19T00:56:44.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123759e640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1238134300 msgr2=0x7f123813ac00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123759e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1238134300 0x7f123813ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 0x7f123813a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 0x7f123813a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:60162/0 (socket says 172.21.15.43:60162) 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 -- 172.21.15.43:0/3110478885 learned_addr learned my addr 172.21.15.43:0/3110478885 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): get_auth_request con 0x7f1238133f00 auth_method 0 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1238133b20 0x7f12381cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): get_auth_request con 0x7f1238133b20 auth_method 0 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient(hunting): handle_auth_done global_id 74526 payload 824 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1238134300 msgr2=0x7f123813ac00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1238134300 0x7f123813ac00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1238133b20 msgr2=0x7f12381cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1238133b20 0x7f12381cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient: _finish_hunting 0 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 monclient: found mon.noname-d 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1238132d10 con 0x7f1238133f00 2024-04-19T00:56:44.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123659c640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 0x7f123813a6c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f122c00c530 tx=0x7f122c00c9f0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 client.?.objecter ms_handle_connect 0x7f1238133f00 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f122c018070 con 0x7f1238133f00 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: dump: 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.972 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 5 monclient: authenticate success, global_id 74526 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f122c0047f0 con 0x7f1238133f00 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:44.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f122c004e50 con 0x7f1238133f00 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123559a640 4 set_mon_vals no callback set 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 monclient: dump: 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.974 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient: _renew_subs 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123813e320 con 0x7f1238133f00 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 client.74526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient: _renew_subs 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:56:44.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f123813e040 con 0x7f1238133f00 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 librados: init done 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 0x7f123813b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f123813ce70 con 0x7f12381cef80 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f122c0049e0 con 0x7f1238133f00 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1224038fb0 0x7f122403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 0x7f123813b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(442..442 src has 1..442) ==== 6185+0+0 (secure 0 0 0) 0x7f122c048520 con 0x7f1238133f00 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 client.74526.objecter ms_dispatch 0x7f1238000ba0 osd_map(442..442 src has 1..442) 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 3 client.74526.objecter handle_osd_map got epochs [442,442] > 0 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 3 client.74526.objecter handle_osd_map decoding full epoch 442 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient: get_auth_request con 0x7f12381cef80 auth_method 0 2024-04-19T00:56:44.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 20 client.74526.objecter dump_active .. 0 homeless 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123759e640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1224038fb0 0x7f122403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123759e640 10 monclient: get_auth_request con 0x7f1224038fb0 auth_method 0 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 10 monclient(hunting): handle_auth_done global_id 74426 payload 824 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12380693b0 con 0x7f12381cef80 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 client.74526.objecter ms_handle_connect 0x7f12381cef80 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f1236d9d640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 0x7f123813b140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12280119d0 tx=0x7f1228011e90 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 client.74526.objecter resend_mon_ops 2024-04-19T00:56:44.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f121f7fe640 10 client.74526.objecter ms_handle_connect 0x7f1224038fb0 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.957+0000 7f123759e640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1224038fb0 0x7f122403b470 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7f12300ae000 tx=0x7f12300a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1228019070 con 0x7f12381cef80 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 10 monclient: dump: 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.978 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1228004300 con 0x7f12381cef80 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 msgr2=0x7f123813b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 0x7f123813b140 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12280119d0 tx=0x7f1228011e90 comp rx=0 tx=0).stop 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 msgr2=0x7f123813b140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.961+0000 7f121f7fe640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f12381cef80 0x7f123813b140 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f12280119d0 tx=0x7f1228011e90 comp rx=0 tx=0).stop 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f123d1e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f123d1e1640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 0x7f12381366f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f123d1e1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f123806d140 con 0x7f123813b680 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 0x7f12381366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient: get_auth_request con 0x7f123813b680 auth_method 0 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:44.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 10 monclient(hunting): handle_auth_done global_id 74435 payload 824 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.965+0000 7f1236d9d640 1 -- 172.21.15.43:0/3110478885 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f123813ba90 con 0x7f123813b680 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f1236d9d640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 0x7f12381366f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12280119d0 tx=0x7f12280046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 client.74526.objecter ms_handle_connect 0x7f123813b680 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 client.74526.objecter resend_mon_ops 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1228019040 con 0x7f123813b680 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 monclient: dump: 2024-04-19T00:56:44.982 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:44.983 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9262 (secure 0 0 0) 0x7f122801e450 con 0x7f123813b680 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 msgr2=0x7f12381366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 0x7f12381366f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12280119d0 tx=0x7f12280046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 msgr2=0x7f12381366f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f121f7fe640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f123813b680 0x7f12381366f0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12280119d0 tx=0x7f12280046d0 comp rx=0 tx=0).stop 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.984 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 184585, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.985 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.986 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.987 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:44.988 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:44.989 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.991 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.993 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.994 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.996 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:44.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:44.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:45.000 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.001 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:45.002 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:45.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.006 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.006 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:45.006 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 10 librados: watch_flush enter 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 10 librados: watch_flush exit 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 20 client.74526.objecter shutdown clearing up homeless session... 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 10 client.74526.objecter successfully canceled tick 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1224038fb0 msgr2=0x7f122403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1224038fb0 0x7f122403b470 secure :-1 s=READY pgs=3548 cs=0 l=1 rev1=1 crypto rx=0x7f12300ae000 tx=0x7f12300a7070 comp rx=0 tx=0).stop 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 10 monclient: shutdown 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 msgr2=0x7f123813a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 0x7f123813a6c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f122c00c530 tx=0x7f122c00c9f0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123759e640 1 -- 172.21.15.43:0/3110478885 reap_dead start 2024-04-19T00:56:45.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 shutdown_connections 2024-04-19T00:56:45.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 --2- 172.21.15.43:0/3110478885 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1238133f00 0x7f123813a6c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 >> 172.21.15.43:0/3110478885 conn(0x7f12380a77d0 msgr2=0x7f12380a27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:45.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 shutdown_connections 2024-04-19T00:56:45.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 -- 172.21.15.43:0/3110478885 wait complete. 2024-04-19T00:56:45.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:44.969+0000 7f123d1e1640 1 librados: shutdown 2024-04-19T00:56:45.009 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:45.009 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 8 for mon i 2024-04-19T00:56:45.009 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:56:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.129+0000 7f179900c640 10 monclient: get_monmap_and_config 2024-04-19T00:56:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.129+0000 7f179900c640 10 monclient: build_initial_monmap 2024-04-19T00:56:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.129+0000 7f179900c640 10 monclient: monmap: 2024-04-19T00:56:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.131 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:56:45.134322+0000 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:56:45.134322+0000 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:56:45.132 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:56:45.133 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 Processor -- start 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 -- start start 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient: init 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient: _add_conns ranks=[9,5,6,8,7,3,10,1,2,0,4] 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17940a1fc0 0x7f17940a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): picked mon.noname-a con 0x7f17940a1fc0 addr v2:172.21.15.43:3300/0 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17940a1280 0x7f17940a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): picked mon.noname-h con 0x7f17940a1280 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17940a43c0 0x7f179413fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): picked mon.noname-d con 0x7f17940a43c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f179406c1a0 con 0x7f17940a1fc0 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f17940a08c0 con 0x7f17940a1280 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f17940a0300 con 0x7f17940a43c0 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179900c640 10 monclient(hunting): authenticate will time out at 1675.261414s 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17940a1fc0 0x7f17940a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f17940a1fc0 0x7f17940a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50128/0 (socket says 172.21.15.43:50128) 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 -- 172.21.15.43:0/3952206220 learned_addr learned my addr 172.21.15.43:0/3952206220 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): get_auth_request con 0x7f17940a1fc0 auth_method 0 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17940a43c0 0x7f179413fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): get_auth_request con 0x7f17940a43c0 auth_method 0 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179259c640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17940a1280 0x7f17940a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.145 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient(hunting): handle_auth_done global_id 64989 payload 274 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 -- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17940a43c0 msgr2=0x7f179413fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17940a43c0 0x7f179413fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 -- 172.21.15.43:0/3952206220 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17940a1280 msgr2=0x7f17940a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17940a1280 0x7f17940a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient: _finish_hunting 0 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 monclient: found mon.noname-a 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f1792d9d640 1 -- 172.21.15.43:0/3952206220 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17940a05e0 con 0x7f17940a1fc0 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179259c640 10 monclient: get_auth_request con 0x7f17940a1280 auth_method 0 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.133+0000 7f179259c640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17940a1280 0x7f17940a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1792d9d640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a1fc0 0x7f17940a3940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f178c009e50 tx=0x7f178c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9220b8c0fc35945 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 1 -- 172.21.15.43:0/3952206220 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f178c0090d0 con 0x7f17940a1fc0 2024-04-19T00:56:45.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: dump: 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:45.147 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: _finish_auth 0 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: _check_auth_tickets 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 1 -- 172.21.15.43:0/3952206220 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178c0a5a80 con 0x7f17940a1fc0 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 5 monclient: authenticate success, global_id 64989 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: get_monmap_and_config success 2024-04-19T00:56:45.148 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 1 -- 172.21.15.43:0/3952206220 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f178c0a5c70 con 0x7f17940a1fc0 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 4 set_mon_vals no callback set 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1791d9b640 10 monclient: dump: 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:45.149 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: shutdown 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:45.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a1fc0 msgr2=0x7f17940a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a1fc0 0x7f17940a3940 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f178c009e50 tx=0x7f178c01c8c0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/3952206220 shutdown_connections 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17940a43c0 0x7f179413fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f17940a1280 0x7f17940a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- 172.21.15.43:0/3952206220 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a1fc0 0x7f17940a3940 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/3952206220 >> 172.21.15.43:0/3952206220 conn(0x7f17940a85c0 msgr2=0x7f17940a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/3952206220 shutdown_connections 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/3952206220 wait complete. 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: build_initial_monmap 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: monmap: 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:56:45.151 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 librados: starting msgr at 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 librados: starting objecter 2024-04-19T00:56:45.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 Processor -- start 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- start start 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 librados: setting wanted keys 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 librados: calling monclient init 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: init 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: _reopen_session rank -1 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: _add_conns ranks=[3,1,0,5,4,8,6,2,7] 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17940a1280 0x7f179413f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): picked mon.noname-g con 0x7f17940a1280 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17940a1fc0 0x7f179413a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): picked mon.noname-f con 0x7f17940a1fc0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a43c0 0x7f1794122990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): picked mon.noname-a con 0x7f17940a43c0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f17940a19f0 con 0x7f17940a43c0 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f17940a47d0 con 0x7f17940a1fc0 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f17940a1b70 con 0x7f17940a1280 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): _renew_subs 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient(hunting): authenticate will time out at 1675.266812s 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a43c0 0x7f1794122990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a43c0 0x7f1794122990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:50144/0 (socket says 172.21.15.43:50144) 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 -- 172.21.15.43:0/2575846073 learned_addr learned my addr 172.21.15.43:0/2575846073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): get_auth_request con 0x7f17940a43c0 auth_method 0 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1792d9d640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17940a1280 0x7f179413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179259c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17940a1fc0 0x7f179413a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179259c640 10 monclient(hunting): get_auth_request con 0x7f17940a1fc0 auth_method 0 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1792d9d640 10 monclient(hunting): get_auth_request con 0x7f17940a1280 auth_method 0 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1792d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1792d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f1792d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient(hunting): handle_auth_done global_id 64998 payload 824 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17940a1280 msgr2=0x7f179413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17940a1280 0x7f179413f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17940a1fc0 msgr2=0x7f179413a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17940a1fc0 0x7f179413a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient: _finish_hunting 0 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 monclient: found mon.noname-a 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1794124a50 con 0x7f17940a43c0 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179359e640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a43c0 0x7f1794122990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f178400e7f0 tx=0x7f178400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 client.?.objecter ms_handle_connect 0x7f17940a43c0 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:56:45.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1784017070 con 0x7f17940a43c0 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: dump: 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:45.157 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f178400cdb0 con 0x7f17940a43c0 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 5 monclient: authenticate success, global_id 64998 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:56:45.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f178400eea0 con 0x7f17940a43c0 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: _renew_subs 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f179409f430 con 0x7f17940a43c0 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179159a640 4 set_mon_vals no callback set 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f178b7fe640 10 monclient: dump: 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:45.159 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 client.64998.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: _renew_subs 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1794125f00 con 0x7f17940a43c0 2024-04-19T00:56:45.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.137+0000 7f179900c640 1 librados: init done 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179900c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179900c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 0x7f17941232b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1794125b80 con 0x7f1794122ed0 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f178401a070 con 0x7f17940a43c0 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 0x7f17941232b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1774038e10 0x7f177403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient: get_auth_request con 0x7f1794122ed0 auth_method 0 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(442..442 src has 1..442) ==== 6185+0+0 (secure 0 0 0) 0x7f178400f040 con 0x7f17940a43c0 2024-04-19T00:56:45.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 client.64998.objecter ms_dispatch 0x7f1794000ba0 osd_map(442..442 src has 1..442) 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 3 client.64998.objecter handle_osd_map got epochs [442,442] > 0 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 3 client.64998.objecter handle_osd_map decoding full epoch 442 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 20 client.64998.objecter dump_active .. 0 homeless 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179259c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1774038e10 0x7f177403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179259c640 10 monclient: get_auth_request con 0x7f1774038e10 auth_method 0 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 10 monclient(hunting): handle_auth_done global_id 74383 payload 824 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17941237f0 con 0x7f1794122ed0 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 client.64998.objecter ms_handle_connect 0x7f1774038e10 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f179259c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1774038e10 0x7f177403b2d0 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7f17940a8fd0 tx=0x7f177c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 client.64998.objecter ms_handle_connect 0x7f1794122ed0 2024-04-19T00:56:45.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 client.64998.objecter resend_mon_ops 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f1792d9d640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 0x7f17941232b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f17940a5900 tx=0x7f178c019660 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f178c0bd070 con 0x7f1794122ed0 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 monclient: dump: 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:45.163 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f178c004040 con 0x7f1794122ed0 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 msgr2=0x7f17941232b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 0x7f17941232b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f17940a5900 tx=0x7f178c019660 comp rx=0 tx=0).stop 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 msgr2=0x7f17941232b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:45.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.141+0000 7f178b7fe640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1794122ed0 0x7f17941232b0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f17940a5900 tx=0x7f178c019660 comp rx=0 tx=0).stop 2024-04-19T00:56:45.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f179900c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f179900c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 0x7f179413c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f179900c640 10 monclient(hunting): start opening mon connection 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f179406d140 con 0x7f179413bd30 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 0x7f179413c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient: get_auth_request con 0x7f179413bd30 auth_method 0 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:56:45.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 10 monclient(hunting): handle_auth_done global_id 74392 payload 824 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 1 -- 172.21.15.43:0/2575846073 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1794124760 con 0x7f179413bd30 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f1792d9d640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 0x7f179413c110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f178c000f90 tx=0x7f178c0a7000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 client.64998.objecter ms_handle_connect 0x7f179413bd30 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 client.64998.objecter resend_mon_ops 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f178c0a5690 con 0x7f179413bd30 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 monclient: dump: 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:56:45.168 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9432 (secure 0 0 0) 0x7f178c0c6070 con 0x7f179413bd30 2024-04-19T00:56:45.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 msgr2=0x7f179413c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 0x7f179413c110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f178c000f90 tx=0x7f178c0a7000 comp rx=0 tx=0).stop 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 msgr2=0x7f179413c110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.149+0000 7f178b7fe640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f179413bd30 0x7f179413c110 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f178c000f90 tx=0x7f178c0a7000 comp rx=0 tx=0).stop 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:45.170 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 90, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 184765, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:56:45.171 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:45.172 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:45.173 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:56:45.174 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:56:45.175 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:56:45.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:56:45.178 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:56:45.180 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.181 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.182 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:56:45.183 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:56:45.184 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.185 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:56:45.187 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:56:45.189 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:56:45.190 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:45.192 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:56:45.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:56:45.194 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:56:45.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:56:45.194 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:56:45.194 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:56:45.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 10 librados: watch_flush enter 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 10 librados: watch_flush exit 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 20 client.64998.objecter shutdown clearing up homeless session... 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 10 client.64998.objecter successfully canceled tick 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1774038e10 msgr2=0x7f177403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1774038e10 0x7f177403b2d0 secure :-1 s=READY pgs=3549 cs=0 l=1 rev1=1 crypto rx=0x7f17940a8fd0 tx=0x7f177c008040 comp rx=0 tx=0).stop 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 10 monclient: shutdown 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a43c0 msgr2=0x7f1794122990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 --2- 172.21.15.43:0/2575846073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f17940a43c0 0x7f1794122990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f178400e7f0 tx=0x7f178400ecb0 comp rx=0 tx=0).stop 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179359e640 1 -- 172.21.15.43:0/2575846073 reap_dead start 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179359e640 1 -- 172.21.15.43:0/2575846073 reap_dead start 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 shutdown_connections 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 >> 172.21.15.43:0/2575846073 conn(0x7f17940a85c0 msgr2=0x7f17940a7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:56:45.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 shutdown_connections 2024-04-19T00:56:45.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 -- 172.21.15.43:0/2575846073 wait complete. 2024-04-19T00:56:45.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:56:45.153+0000 7f179900c640 1 librados: shutdown 2024-04-19T00:56:45.197 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 8 2024-04-19T00:56:45.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T00:56:56.531 INFO:tasks.ceph.osd.1.smithi043.stderr:2024-04-19T00:56:56.533+0000 7f369f72e640 -1 snap_mapper.add_oid found existing snaps mapped on 103:e8467f20:test-rados-api-smithi161-28857-97::foo:21, removing 2024-04-19T00:56:56.532 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-04-19T00:56:56.534+0000 7fab198d3640 -1 snap_mapper.add_oid found existing snaps mapped on 103:e8467f20:test-rados-api-smithi161-28857-97::foo:21, removing 2024-04-19T00:58:15.199 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T00:58:15.199 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-19T00:58:15.199 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T00:58:15.200 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T00:58:15.204 INFO:tasks.ceph.mon.c:Started 2024-04-19T00:58:15.204 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-04-19T00:58:15.204 INFO:tasks.ceph.mon.b:Sent signal 19 2024-04-19T00:58:15.204 INFO:tasks.mon_thrash:Sending STOP to mon e 2024-04-19T00:58:15.204 INFO:tasks.ceph.mon.e:Sent signal 19 2024-04-19T00:58:15.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:58:15.324 INFO:tasks.ceph.mon.c.smithi043.stderr:2024-04-19T00:58:15.324+0000 7efe58910c80 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 472 -> 0 (sync_force? mon rebuild?) 2024-04-19T00:58:30.205 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-04-19T00:58:30.206 INFO:tasks.ceph.mon.b:Sent signal 18 2024-04-19T00:58:30.206 INFO:tasks.mon_thrash:Sending CONT to mon e 2024-04-19T00:58:30.207 INFO:tasks.ceph.mon.e:Sent signal 18 2024-04-19T00:58:30.207 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:58:30.207 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.383+0000 7f9862c42640 10 monclient: get_monmap_and_config 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.383+0000 7f9862c42640 10 monclient: build_initial_monmap 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.383+0000 7f9862c42640 10 monclient: monmap: 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:30.389812+0000 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:30.389812+0000 2024-04-19T00:58:30.389 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:30.390 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 Processor -- start 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 -- start start 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient: init 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient: _add_conns ranks=[5,4,7,9,3,10,6,8,2,1,0] 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): picked mon.noname-h con 0x7f985c09e450 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 0x7f985c0a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): picked mon.noname-c con 0x7f985c09e830 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f985c09ec50 0x7f985c0a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): picked mon.noname-i con 0x7f985c09ec50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:30.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f985c0735a0 con 0x7f985c09e830 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f985c0a1bc0 con 0x7f985c09e450 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f985c0a2000 con 0x7f985c09ec50 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient(hunting): authenticate will time out at 1780.516758s 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 0x7f985c0a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 0x7f985c0a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:42426/0 (socket says 172.21.15.43:42426) 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 -- 172.21.15.43:0/1539288004 learned_addr learned my addr 172.21.15.43:0/1539288004 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): get_auth_request con 0x7f985c09e830 auth_method 0 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98611df640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f985c09ec50 0x7f985c0a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98611df640 10 monclient(hunting): get_auth_request con 0x7f985c09ec50 auth_method 0 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98611df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98611df640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98611df640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98609de640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98609de640 10 monclient(hunting): get_auth_request con 0x7f985c09e450 auth_method 0 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98609de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98609de640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f98609de640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient(hunting): handle_auth_done global_id 134104 payload 274 2024-04-19T00:58:30.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 -- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f985c09ec50 msgr2=0x7f985c0a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f985c09ec50 0x7f985c0a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 -- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 msgr2=0x7f985c136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 monclient: found mon.noname-c 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:30.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 -- 172.21.15.43:0/1539288004 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c06c6f0 con 0x7f985c09e830 2024-04-19T00:58:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985bfff640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 0x7f985c0a38d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f98480098a0 tx=0x7f984801c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=615296ece51c51ef server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 1 -- 172.21.15.43:0/1539288004 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98480b0070 con 0x7f985c09e830 2024-04-19T00:58:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:30.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: dump: 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.402 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 1 -- 172.21.15.43:0/1539288004 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98480aa810 con 0x7f985c09e830 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 1 -- 172.21.15.43:0/1539288004 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f98480a9300 con 0x7f985c09e830 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 5 monclient: authenticate success, global_id 134104 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f9862c42640 4 set_mon_vals no callback set 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:30.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.387+0000 7f985b7fe640 10 monclient: dump: 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.404 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: shutdown 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1539288004 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 msgr2=0x7f985c0a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 0x7f985c0a38d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f98480098a0 tx=0x7f984801c440 comp rx=0 tx=0).stop 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1539288004 shutdown_connections 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f985c09ec50 0x7f985c0a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f985c09e830 0x7f985c0a38d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- 172.21.15.43:0/1539288004 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1539288004 >> 172.21.15.43:0/1539288004 conn(0x7f985c0a84c0 msgr2=0x7f985c0a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1539288004 shutdown_connections 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1539288004 wait complete. 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: build_initial_monmap 2024-04-19T00:58:30.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: monmap: 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:30.406 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 librados: starting msgr at 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 librados: starting objecter 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 Processor -- start 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- start start 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 librados: setting wanted keys 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 librados: calling monclient init 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: init 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _add_conns ranks=[5,3,8,6,0,2,7,1,4] 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c134830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): picked mon.noname-h con 0x7f985c09e450 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:30.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f985c09e830 0x7f985c13b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): picked mon.noname-g con 0x7f985c09e830 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 0x7f985c13bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): picked mon.noname-e con 0x7f985c09ec50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f985c1406e0 con 0x7f985c09e830 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f985c140860 con 0x7f985c09e450 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f985c1409e0 con 0x7f985c09ec50 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient(hunting): authenticate will time out at 1780.521743s 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 0x7f985c13bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 0x7f985c13bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:48892/0 (socket says 172.21.15.43:48892) 2024-04-19T00:58:30.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 -- 172.21.15.43:0/1363426818 learned_addr learned my addr 172.21.15.43:0/1363426818 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): get_auth_request con 0x7f985c09ec50 auth_method 0 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f985c09e830 0x7f985c13b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): get_auth_request con 0x7f985c09e450 auth_method 0 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): get_auth_request con 0x7f985c09e830 auth_method 0 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98609de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient(hunting): handle_auth_done global_id 114308 payload 824 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 -- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 msgr2=0x7f985c134830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c134830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 -- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f985c09e830 msgr2=0x7f985c13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f985c09e830 0x7f985c13b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient: _finish_hunting 0 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 monclient: found mon.noname-e 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 -- 172.21.15.43:0/1363426818 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c11ed10 con 0x7f985c09ec50 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98611df640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 0x7f985c13bed0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9854007e60 tx=0x7f985400e4b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 client.?.objecter ms_handle_connect 0x7f985c09ec50 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f985400edd0 con 0x7f985c09ec50 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: dump: 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.411 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.412 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: _finish_auth 0 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: _check_auth_tickets 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9854019070 con 0x7f985c09ec50 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9854014d00 con 0x7f985c09ec50 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f985affd640 4 set_mon_vals no callback set 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 monclient: dump: 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.413 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 5 monclient: authenticate success, global_id 114308 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _renew_subs 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c06b9e0 con 0x7f985c09ec50 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 client.114308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _renew_subs 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c11f930 con 0x7f985c09ec50 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 librados: init done 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9854014420 con 0x7f985c09ec50 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985c0008e0 con 0x7f985c09ec50 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9840038e10 0x7f984003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(529..529 src has 1..529) ==== 6625+0+0 (secure 0 0 0) 0x7f9854049ce0 con 0x7f985c09ec50 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 10 client.114308.objecter ms_dispatch 0x7f985c000ba0 osd_map(529..529 src has 1..529) 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 3 client.114308.objecter handle_osd_map got epochs [529,529] > 0 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.391+0000 7f98597fa640 3 client.114308.objecter handle_osd_map decoding full epoch 529 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.395+0000 7f98597fa640 20 client.114308.objecter dump_active .. 0 homeless 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.395+0000 7f98609de640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9840038e10 0x7f984003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.395+0000 7f98609de640 10 monclient: get_auth_request con 0x7f9840038e10 auth_method 0 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.395+0000 7f98609de640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9840038e10 0x7f984003b2d0 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7f9850007a60 tx=0x7f9850008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.395+0000 7f98597fa640 10 client.114308.objecter ms_handle_connect 0x7f9840038e10 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.399+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f985401e980 con 0x7f985c09ec50 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.399+0000 7f98597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.399+0000 7f98597fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:30.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:58:30.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:58:30.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f985c11e9d0 con 0x7f985c09ec50 2024-04-19T00:58:30.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f98597fa640 1 -- 172.21.15.43:0/1363426818 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f985401e460 con 0x7f985c09ec50 2024-04-19T00:58:30.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f98597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:58:30.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f98597fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:30.625 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":94,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 10 librados: watch_flush enter 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 10 librados: watch_flush exit 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 20 client.114308.objecter shutdown clearing up homeless session... 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.623+0000 7f9862c42640 10 client.114308.objecter successfully canceled tick 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9840038e10 msgr2=0x7f984003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9840038e10 0x7f984003b2d0 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7f9850007a60 tx=0x7f9850008040 comp rx=0 tx=0).stop 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 10 monclient: shutdown 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 msgr2=0x7f985c13bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 0x7f985c13bed0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9854007e60 tx=0x7f985400e4b0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 shutdown_connections 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9840038e10 0x7f984003b2d0 unknown :-1 s=CLOSED pgs=3556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f985c09ec50 0x7f985c13bed0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f985c09e830 0x7f985c13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 --2- 172.21.15.43:0/1363426818 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f985c09e450 0x7f985c134830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 >> 172.21.15.43:0/1363426818 conn(0x7f985c0a84c0 msgr2=0x7f985c0a7110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 shutdown_connections 2024-04-19T00:58:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 -- 172.21.15.43:0/1363426818 wait complete. 2024-04-19T00:58:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.627+0000 7f9862c42640 1 librados: shutdown 2024-04-19T00:58:30.642 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:58:30.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:58:30.642 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:58:30.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient: get_monmap_and_config 2024-04-19T00:58:30.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient: build_initial_monmap 2024-04-19T00:58:30.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient: monmap: 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:30.766893+0000 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:30.766893+0000 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:30.764 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:30.765 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:30.765 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 Processor -- start 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 -- start start 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient: init 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient: _add_conns ranks=[9,5,7,0,4,2,8,1,6,3,10] 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe488138cd0 0x7fe4881390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): picked mon.noname-a con 0x7fe488138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe4880a5b80 0x7fe4881395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): picked mon.noname-h con 0x7fe4880a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4881358b0 0x7fe488135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): picked mon.noname-i con 0x7fe4881358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fe4880a4410 con 0x7fe488138cd0 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe488139b30 con 0x7fe4880a5b80 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe4880a3dc0 con 0x7fe4881358b0 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe490569640 10 monclient(hunting): authenticate will time out at 1780.894332s 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe488138cd0 0x7fe4881390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe488138cd0 0x7fe4881390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54244/0 (socket says 172.21.15.43:54244) 2024-04-19T00:58:30.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 -- 172.21.15.43:0/2844886836 learned_addr learned my addr 172.21.15.43:0/2844886836 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): get_auth_request con 0x7fe488138cd0 auth_method 0 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48eb06640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4881358b0 0x7fe488135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48db04640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe4880a5b80 0x7fe4881395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48eb06640 10 monclient(hunting): get_auth_request con 0x7fe4881358b0 auth_method 0 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48eb06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48eb06640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48eb06640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48db04640 10 monclient(hunting): get_auth_request con 0x7fe4880a5b80 auth_method 0 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48db04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48db04640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48db04640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient(hunting): handle_auth_done global_id 65043 payload 274 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 -- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4881358b0 msgr2=0x7fe488135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4881358b0 0x7fe488135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 -- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe4880a5b80 msgr2=0x7fe4881395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe4880a5b80 0x7fe4881395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient: _finish_hunting 0 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 monclient: found mon.noname-a 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:30.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.763+0000 7fe48e305640 1 -- 172.21.15.43:0/2844886836 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4880a40a0 con 0x7fe488138cd0 2024-04-19T00:58:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48e305640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe488138cd0 0x7fe4881390b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe484009e50 tx=0x7fe48401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6285b43ee8800b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 1 -- 172.21.15.43:0/2844886836 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4840090d0 con 0x7fe488138cd0 2024-04-19T00:58:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:30.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: dump: 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.779 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: _finish_auth 0 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: _check_auth_tickets 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 1 -- 172.21.15.43:0/2844886836 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4840a5a50 con 0x7fe488138cd0 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 5 monclient: authenticate success, global_id 65043 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 1 -- 172.21.15.43:0/2844886836 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4840a5c40 con 0x7fe488138cd0 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 4 set_mon_vals no callback set 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe48d303640 10 monclient: dump: 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.780 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.781 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 10 monclient: shutdown 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 -- 172.21.15.43:0/2844886836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe488138cd0 msgr2=0x7fe4881390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe488138cd0 0x7fe4881390b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe484009e50 tx=0x7fe48401c890 comp rx=0 tx=0).stop 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 -- 172.21.15.43:0/2844886836 shutdown_connections 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe4881358b0 0x7fe488135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe4880a5b80 0x7fe4881395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 --2- 172.21.15.43:0/2844886836 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe488138cd0 0x7fe4881390b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 -- 172.21.15.43:0/2844886836 >> 172.21.15.43:0/2844886836 conn(0x7fe4880ab2f0 msgr2=0x7fe4880ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 -- 172.21.15.43:0/2844886836 shutdown_connections 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 -- 172.21.15.43:0/2844886836 wait complete. 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 10 monclient: build_initial_monmap 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 10 monclient: monmap: 2024-04-19T00:58:30.782 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:30.783 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 librados: starting msgr at 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 librados: starting objecter 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.767+0000 7fe490569640 1 Processor -- start 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- start start 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 librados: setting wanted keys 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 librados: calling monclient init 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: init 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: _add_conns ranks=[0,4,5,1,3,7,8,2,6] 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4880a5b80 0x7fe4881d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): picked mon.noname-a con 0x7fe4880a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4881358b0 0x7fe4881d1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): picked mon.noname-c con 0x7fe4881358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe488138cd0 0x7fe4881d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): picked mon.noname-h con 0x7fe488138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe4880a2010 con 0x7fe4880a5b80 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe4880a1e90 con 0x7fe4881358b0 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe488142aa0 con 0x7fe488138cd0 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): authenticate will time out at 1780.900727s 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4880a5b80 0x7fe4881d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4881358b0 0x7fe4881d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4880a5b80 0x7fe4881d0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:54248/0 (socket says 172.21.15.43:54248) 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 -- 172.21.15.43:0/3797041927 learned_addr learned my addr 172.21.15.43:0/3797041927 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): get_auth_request con 0x7fe4880a5b80 auth_method 0 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): get_auth_request con 0x7fe4881358b0 auth_method 0 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48eb06640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe488138cd0 0x7fe4881d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient(hunting): handle_auth_done global_id 65052 payload 824 2024-04-19T00:58:30.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe488138cd0 msgr2=0x7fe4881d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe488138cd0 0x7fe4881d15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4881358b0 msgr2=0x7fe4881d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4881358b0 0x7fe4881d1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient: _finish_hunting 0 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 monclient: found mon.noname-a 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4880a0770 con 0x7fe4880a5b80 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48eb06640 10 monclient: get_auth_request con 0x7fe488138cd0 auth_method 0 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48eb06640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe488138cd0 0x7fe4881d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe4881358b0 0x7fe4881d1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48e305640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4880a5b80 0x7fe4881d0b30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe48401c6c0 tx=0x7fe484019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 client.?.objecter ms_handle_connect 0x7fe4880a5b80 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4840bc020 con 0x7fe4880a5b80 2024-04-19T00:58:30.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: dump: 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.788 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: _finish_auth 0 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: _check_auth_tickets 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe484003e10 con 0x7fe4880a5b80 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe4840a57f0 con 0x7fe4880a5b80 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47ffff640 4 set_mon_vals no callback set 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: dump: 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.789 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.790 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 5 monclient: authenticate success, global_id 65052 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: _renew_subs 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4880a3590 con 0x7fe4880a5b80 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 client.65052.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: _renew_subs 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4880a0490 con 0x7fe4880a5b80 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 librados: init done 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 0x7fe488134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe48806d140 con 0x7fe4881348a0 2024-04-19T00:58:30.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe4840bf020 con 0x7fe4880a5b80 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 0x7fe488134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient: get_auth_request con 0x7fe4881348a0 auth_method 0 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe46c038e10 0x7fe46c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 6625+0+0 (secure 0 0 0) 0x7fe4840e4340 con 0x7fe4880a5b80 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 10 monclient(hunting): handle_auth_done global_id 65061 payload 824 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 client.65052.objecter ms_dispatch 0x7fe488000ba0 osd_map(529..529 src has 1..529) 2024-04-19T00:58:30.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 3 client.65052.objecter handle_osd_map got epochs [529,529] > 0 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 3 client.65052.objecter handle_osd_map decoding full epoch 529 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4880a0290 con 0x7fe4881348a0 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 20 client.65052.objecter dump_active .. 0 homeless 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 client.65052.objecter ms_handle_connect 0x7fe4881348a0 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 client.65052.objecter resend_mon_ops 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48db04640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 0x7fe488134c80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe4880a7360 tx=0x7fe47800e1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe478018070 con 0x7fe4881348a0 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: dump: 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.793 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48eb06640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe46c038e10 0x7fe46c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe48eb06640 10 monclient: get_auth_request con 0x7fe46c038e10 auth_method 0 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe47800ed10 con 0x7fe4881348a0 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 msgr2=0x7fe488134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 0x7fe488134c80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe4880a7360 tx=0x7fe47800e1d0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 msgr2=0x7fe488134c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.771+0000 7fe47effd640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881348a0 0x7fe488134c80 secure :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe4880a7360 tx=0x7fe47800e1d0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.775+0000 7fe47effd640 10 client.65052.objecter ms_handle_connect 0x7fe46c038e10 2024-04-19T00:58:30.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.775+0000 7fe48eb06640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe46c038e10 0x7fe46c03b2d0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7fe4881d1bf0 tx=0x7fe480008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 0x7fe48813bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe488068240 con 0x7fe4881351c0 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 0x7fe48813bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient: get_auth_request con 0x7fe4881351c0 auth_method 0 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 10 monclient(hunting): handle_auth_done global_id 65070 payload 824 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 1 -- 172.21.15.43:0/3797041927 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe4880693b0 con 0x7fe4881351c0 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 client.65052.objecter ms_handle_connect 0x7fe4881351c0 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48db04640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 0x7fe48813bcc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe47801a000 tx=0x7fe4780046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 client.65052.objecter resend_mon_ops 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe478014ab0 con 0x7fe4881351c0 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 monclient: dump: 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.798 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.799 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fe478014640 con 0x7fe4881351c0 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 msgr2=0x7fe48813bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 0x7fe48813bcc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe47801a000 tx=0x7fe4780046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 msgr2=0x7fe48813bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe47effd640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4881351c0 0x7fe48813bcc0 secure :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe47801a000 tx=0x7fe4780046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:30.800 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 437051, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:58:30.801 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:30.802 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:30.804 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:30.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:30.809 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.810 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:30.812 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.813 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:30.814 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.815 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.816 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.817 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.818 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.819 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.820 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.821 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.822 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T00:58:30.823 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.823 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.823 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:30.823 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:30.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 10 librados: watch_flush enter 2024-04-19T00:58:30.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 10 librados: watch_flush exit 2024-04-19T00:58:30.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 20 client.65052.objecter shutdown clearing up homeless session... 2024-04-19T00:58:30.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 10 client.65052.objecter successfully canceled tick 2024-04-19T00:58:30.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe46c038e10 msgr2=0x7fe46c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe46c038e10 0x7fe46c03b2d0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7fe4881d1bf0 tx=0x7fe480008040 comp rx=0 tx=0).stop 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 10 monclient: shutdown 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4880a5b80 msgr2=0x7fe4881d0b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 --2- 172.21.15.43:0/3797041927 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe4880a5b80 0x7fe4881d0b30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe48401c6c0 tx=0x7fe484019690 comp rx=0 tx=0).stop 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48eb06640 1 -- 172.21.15.43:0/3797041927 reap_dead start 2024-04-19T00:58:30.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe48eb06640 1 -- 172.21.15.43:0/3797041927 reap_dead start 2024-04-19T00:58:30.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 shutdown_connections 2024-04-19T00:58:30.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 >> 172.21.15.43:0/3797041927 conn(0x7fe4880ab2f0 msgr2=0x7fe4881368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:30.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 shutdown_connections 2024-04-19T00:58:30.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 -- 172.21.15.43:0/3797041927 wait complete. 2024-04-19T00:58:30.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.779+0000 7fe490569640 1 librados: shutdown 2024-04-19T00:58:30.826 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:30.827 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:58:30.827 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient: get_monmap_and_config 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient: build_initial_monmap 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient: monmap: 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:30.939565+0000 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:30.939565+0000 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.936 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:30.937 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 Processor -- start 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 -- start start 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient: init 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient: _add_conns ranks=[0,6,3,10,7,8,2,1,5,4,9] 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00e00a2ac0 addr v1:172.21.15.43:6789/0 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 0x7f00e013bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): picked mon.noname-d con 0x7f00e00a2ea0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:30.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 0x7f00e013e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): picked mon.noname-g con 0x7f00e00a32c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e0137650 con 0x7f00e00a2ac0 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f00e0071890 con 0x7f00e00a32c0 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f00e0071cb0 con 0x7f00e00a2ea0 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00e6249640 10 monclient(hunting): authenticate will time out at 1781.066950s 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 0x7f00e013bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 0x7f00e013bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42664/0 (socket says 172.21.15.43:42664) 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 -- 172.21.15.43:0/2641595005 learned_addr learned my addr 172.21.15.43:0/2641595005 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): get_auth_request con 0x7f00e00a2ea0 auth_method 0 2024-04-19T00:58:30.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient(hunting): handle_auth_done global_id 74562 payload 274 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 -- 172.21.15.43:0/2641595005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 msgr2=0x7f00e013e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 --2- 172.21.15.43:0/2641595005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 0x7f00e013e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 -- 172.21.15.43:0/2641595005 >> v1:172.21.15.43:6789/0 conn(0x7f00e00a2ac0 legacy=0x7f00e0136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient: _finish_hunting 0 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 monclient: found mon.noname-d 2024-04-19T00:58:30.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:30.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:30.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.935+0000 7f00deffd640 1 -- 172.21.15.43:0/2641595005 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e0072fd0 con 0x7f00e00a2ea0 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00deffd640 1 --2- 172.21.15.43:0/2641595005 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 0x7f00e013bc50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f00c80098d0 tx=0x7f00c801c3e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=6354267b4605beaf server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 1 -- 172.21.15.43:0/2641595005 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00c80b0070 con 0x7f00e00a2ea0 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: dump: 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.949 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:30.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 1 -- 172.21.15.43:0/2641595005 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c80aa810 con 0x7f00e00a2ea0 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 5 monclient: authenticate success, global_id 74562 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 1 -- 172.21.15.43:0/2641595005 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00c80a9300 con 0x7f00e00a2ea0 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 4 set_mon_vals no callback set 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00de7fc640 10 monclient: dump: 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.951 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: shutdown 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- 172.21.15.43:0/2641595005 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 msgr2=0x7f00e013bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 --2- 172.21.15.43:0/2641595005 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 0x7f00e013bc50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f00c80098d0 tx=0x7f00c801c3e0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- 172.21.15.43:0/2641595005 shutdown_connections 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 --2- 172.21.15.43:0/2641595005 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 0x7f00e013e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 --2- 172.21.15.43:0/2641595005 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f00e00a2ea0 0x7f00e013bc50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- 172.21.15.43:0/2641595005 >> 172.21.15.43:0/2641595005 conn(0x7f00e00a62f0 msgr2=0x7f00e006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- 172.21.15.43:0/2641595005 shutdown_connections 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- 172.21.15.43:0/2641595005 wait complete. 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: build_initial_monmap 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: monmap: 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:30.953 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 librados: starting msgr at 2024-04-19T00:58:30.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 librados: starting objecter 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 Processor -- start 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- start start 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 librados: setting wanted keys 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 librados: calling monclient init 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: init 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient: _add_conns ranks=[8,7,3,6,4,1,2,0,5] 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 0x7f00e01d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): picked mon.noname-e con 0x7f00e00a2ac0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00e00a2ea0 0x7f00e01d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): picked mon.noname-i con 0x7f00e00a2ea0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 0x7f00e01d1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): picked mon.noname-g con 0x7f00e00a32c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f00e0141290 con 0x7f00e00a32c0 2024-04-19T00:58:30.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f00e0141110 con 0x7f00e00a2ea0 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f00e0141410 con 0x7f00e00a2ac0 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00e6249640 10 monclient(hunting): authenticate will time out at 1781.071729s 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 0x7f00e01d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 0x7f00e01d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:32804/0 (socket says 172.21.15.43:32804) 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 1 -- 172.21.15.43:0/3425235234 learned_addr learned my addr 172.21.15.43:0/3425235234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): get_auth_request con 0x7f00e00a2ac0 auth_method 0 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.939+0000 7f00df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 10 monclient(hunting): handle_auth_done global_id 114317 payload 824 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00e00a2ea0 msgr2=0x7f00e01d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f00e00a2ea0 0x7f00e01d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 msgr2=0x7f00e01d1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f00e00a32c0 0x7f00e01d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 monclient: found mon.noname-e 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e013b140 con 0x7f00e00a2ac0 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00df7fe640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 0x7f00e01d07c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f00d000c530 tx=0x7f00d000c9f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 client.?.objecter ms_handle_connect 0x7f00e00a2ac0 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00d0017070 con 0x7f00e00a2ac0 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: dump: 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.958 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: _finish_auth 0 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d0003600 con 0x7f00e00a2ac0 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00d0003c60 con 0x7f00e00a2ac0 2024-04-19T00:58:30.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 5 monclient: authenticate success, global_id 114317 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 monclient: _renew_subs 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e013c8d0 con 0x7f00e00a2ac0 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00ddffb640 4 set_mon_vals no callback set 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: dump: 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.960 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 client.114317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 monclient: _renew_subs 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e009ff90 con 0x7f00e00a2ac0 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 1 librados: init done 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f00d00037f0 con 0x7f00e00a2ac0 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 0x7f00e0135c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00e006d140 con 0x7f00e013c270 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00c0038e10 0x7f00c003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 0x7f00e0135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient: get_auth_request con 0x7f00e013c270 auth_method 0 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(529..529 src has 1..529) ==== 6625+0+0 (secure 0 0 0) 0x7f00d0048850 con 0x7f00e00a2ac0 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 client.114317.objecter ms_dispatch 0x7f00e0000ba0 osd_map(529..529 src has 1..529) 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 3 client.114317.objecter handle_osd_map got epochs [529,529] > 0 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 3 client.114317.objecter handle_osd_map decoding full epoch 529 2024-04-19T00:58:30.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 20 client.114317.objecter dump_active .. 0 homeless 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 10 monclient(hunting): handle_auth_done global_id 74567 payload 824 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00e013c650 con 0x7f00e013c270 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00dffff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00c0038e10 0x7f00c003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00dffff640 10 monclient: get_auth_request con 0x7f00c0038e10 auth_method 0 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 client.114317.objecter ms_handle_connect 0x7f00e013c270 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 client.114317.objecter resend_mon_ops 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00deffd640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 0x7f00e0135c10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00e0073ab0 tx=0x7f00c8004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00c80b0040 con 0x7f00e013c270 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: dump: 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.964 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00c8003620 con 0x7f00e013c270 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 msgr2=0x7f00e0135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 0x7f00e0135c10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00e0073ab0 tx=0x7f00c8004610 comp rx=0 tx=0).stop 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 msgr2=0x7f00e0135c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e013c270 0x7f00e0135c10 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f00e0073ab0 tx=0x7f00c8004610 comp rx=0 tx=0).stop 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00d7fff640 10 client.114317.objecter ms_handle_connect 0x7f00c0038e10 2024-04-19T00:58:30.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.943+0000 7f00dffff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00c0038e10 0x7f00c003b2d0 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7f00e00a3750 tx=0x7f00d8009250 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 0x7f00e0136530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f00e0136a70 con 0x7f00e0136150 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 0x7f00e0136530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient: get_auth_request con 0x7f00e0136150 auth_method 0 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:30.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 10 monclient(hunting): handle_auth_done global_id 74576 payload 824 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 1 -- 172.21.15.43:0/3425235234 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00e0076440 con 0x7f00e0136150 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00deffd640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 0x7f00e0136530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f00c80ae040 tx=0x7f00c80a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 client.114317.objecter ms_handle_connect 0x7f00e0136150 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 client.114317.objecter resend_mon_ops 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f00c80b0040 con 0x7f00e0136150 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 monclient: dump: 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:30.968 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7f00c80c7420 con 0x7f00e0136150 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 msgr2=0x7f00e0136530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 0x7f00e0136530 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f00c80ae040 tx=0x7f00c80a8000 comp rx=0 tx=0).stop 2024-04-19T00:58:30.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 msgr2=0x7f00e0136530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00d7fff640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f00e0136150 0x7f00e0136530 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f00c80ae040 tx=0x7f00c80a8000 comp rx=0 tx=0).stop 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 290572, 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:30.970 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:30.971 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:30.972 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:30.973 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:30.974 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.975 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.976 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.977 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:30.978 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.979 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:30.980 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.981 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.982 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.983 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:30.985 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:30.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:30.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:30.990 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.991 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:30.992 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:30.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:30.994 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:30.994 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 10 librados: watch_flush enter 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 10 librados: watch_flush exit 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 20 client.114317.objecter shutdown clearing up homeless session... 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 10 client.114317.objecter successfully canceled tick 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00c0038e10 msgr2=0x7f00c003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f00c0038e10 0x7f00c003b2d0 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7f00e00a3750 tx=0x7f00d8009250 comp rx=0 tx=0).stop 2024-04-19T00:58:30.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 10 monclient: shutdown 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 msgr2=0x7f00e01d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 0x7f00e01d07c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f00d000c530 tx=0x7f00d000c9f0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00dffff640 1 -- 172.21.15.43:0/3425235234 reap_dead start 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 shutdown_connections 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 --2- 172.21.15.43:0/3425235234 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f00e00a2ac0 0x7f00e01d07c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 >> 172.21.15.43:0/3425235234 conn(0x7f00e00a62f0 msgr2=0x7f00e00a4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 shutdown_connections 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 -- 172.21.15.43:0/3425235234 wait complete. 2024-04-19T00:58:30.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:30.951+0000 7f00e6249640 1 librados: shutdown 2024-04-19T00:58:30.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:30.998 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:58:30.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:58:31.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient: get_monmap_and_config 2024-04-19T00:58:31.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient: monmap: 2024-04-19T00:58:31.107 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:31.110892+0000 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:31.110892+0000 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.108 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:31.109 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:31.109 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 Processor -- start 2024-04-19T00:58:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 -- start start 2024-04-19T00:58:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient: init 2024-04-19T00:58:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient: _add_conns ranks=[2,3,5,7,9,10,8,4,1,6,0] 2024-04-19T00:58:31.111 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 0x7fdf581390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): picked mon.noname-b con 0x7fdf58138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdf580a5b80 0x7fdf581395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): picked mon.noname-g con 0x7fdf580a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): picked mon.noname-h con 0x7fdf581358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdf580a4410 con 0x7fdf58138cd0 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdf58139b30 con 0x7fdf580a5b80 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdf580a3dc0 con 0x7fdf581358b0 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf5e111640 10 monclient(hunting): authenticate will time out at 1781.237948s 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 0x7fdf581390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.112 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 0x7fdf581390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35762/0 (socket says 172.21.15.43:35762) 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 -- 172.21.15.43:0/204788860 learned_addr learned my addr 172.21.15.43:0/204788860 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): get_auth_request con 0x7fdf58138cd0 auth_method 0 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf57fff640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf56ffd640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdf580a5b80 0x7fdf581395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf57fff640 10 monclient(hunting): get_auth_request con 0x7fdf581358b0 auth_method 0 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf57fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.113 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf56ffd640 10 monclient(hunting): get_auth_request con 0x7fdf580a5b80 auth_method 0 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient(hunting): handle_auth_done global_id 74585 payload 274 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 -- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 msgr2=0x7fdf58135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 -- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdf580a5b80 msgr2=0x7fdf581395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdf580a5b80 0x7fdf581395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 monclient: found mon.noname-b 2024-04-19T00:58:31.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:58:31.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.107+0000 7fdf577fe640 1 -- 172.21.15.43:0/204788860 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf580a40a0 con 0x7fdf58138cd0 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf577fe640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 0x7fdf581390b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009e80 tx=0x7fdf4801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e41de1cb3b31e1ce server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 1 -- 172.21.15.43:0/204788860 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdf480090d0 con 0x7fdf58138cd0 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: dump: 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.122 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 5 monclient: authenticate success, global_id 74585 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 1 -- 172.21.15.43:0/204788860 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf480a5a10 con 0x7fdf58138cd0 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 4 set_mon_vals no callback set 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 1 -- 172.21.15.43:0/204788860 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdf480a5c00 con 0x7fdf58138cd0 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf567fc640 10 monclient: dump: 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.124 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 10 monclient: shutdown 2024-04-19T00:58:31.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 -- 172.21.15.43:0/204788860 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 msgr2=0x7fdf581390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 0x7fdf581390b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009e80 tx=0x7fdf4801c8b0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 -- 172.21.15.43:0/204788860 shutdown_connections 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdf580a5b80 0x7fdf581395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 --2- 172.21.15.43:0/204788860 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdf58138cd0 0x7fdf581390b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 -- 172.21.15.43:0/204788860 >> 172.21.15.43:0/204788860 conn(0x7fdf580ab2f0 msgr2=0x7fdf580ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 -- 172.21.15.43:0/204788860 shutdown_connections 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 -- 172.21.15.43:0/204788860 wait complete. 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 10 monclient: monmap: 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:31.126 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 librados: starting msgr at 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.111+0000 7fdf5e111640 1 librados: starting objecter 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 Processor -- start 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- start start 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 librados: setting wanted keys 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 librados: calling monclient init 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: init 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: _add_conns ranks=[0,5,6,1,3,8,4,2,7] 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf580a5b80 0x7fdf58133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): picked mon.noname-a con 0x7fdf580a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): picked mon.noname-h con 0x7fdf581358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdf58138cd0 0x7fdf58133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): picked mon.noname-d con 0x7fdf58138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fdf580a2010 con 0x7fdf580a5b80 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdf580a1e90 con 0x7fdf581358b0 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fdf58142aa0 con 0x7fdf58138cd0 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): authenticate will time out at 1781.244663s 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf580a5b80 0x7fdf58133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdf58138cd0 0x7fdf58133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf580a5b80 0x7fdf58133430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58208/0 (socket says 172.21.15.43:58208) 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 -- 172.21.15.43:0/3569590815 learned_addr learned my addr 172.21.15.43:0/3569590815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): get_auth_request con 0x7fdf580a5b80 auth_method 0 2024-04-19T00:58:31.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): get_auth_request con 0x7fdf58138cd0 auth_method 0 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): get_auth_request con 0x7fdf581358b0 auth_method 0 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient(hunting): handle_auth_done global_id 65088 payload 824 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdf58138cd0 msgr2=0x7fdf58133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdf58138cd0 0x7fdf58133eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 msgr2=0x7fdf58133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdf581358b0 0x7fdf58133970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 monclient: found mon.noname-a 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf5809cb80 con 0x7fdf580a5b80 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf577fe640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf580a5b80 0x7fdf58133430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009040 tx=0x7fdf480196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 client.?.objecter ms_handle_connect 0x7fdf580a5b80 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdf480bc020 con 0x7fdf580a5b80 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: dump: 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.132 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf48003e10 con 0x7fdf580a5b80 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 5 monclient: authenticate success, global_id 65088 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: _renew_subs 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdf480a57b0 con 0x7fdf580a5b80 2024-04-19T00:58:31.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf5809bc70 con 0x7fdf580a5b80 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: dump: 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.134 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf55ffb640 4 set_mon_vals no callback set 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 client.65088.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: _renew_subs 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf580a14b0 con 0x7fdf580a5b80 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 librados: init done 2024-04-19T00:58:31.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 0x7fdf581d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdf5806d140 con 0x7fdf581d0aa0 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdf480bf020 con 0x7fdf580a5b80 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 0x7fdf581d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient: get_auth_request con 0x7fdf581d0aa0 auth_method 0 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdf38038e10 0x7fdf3803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_done global_id 134122 payload 824 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(529..529 src has 1..529) ==== 6625+0+0 (secure 0 0 0) 0x7fdf480b3040 con 0x7fdf580a5b80 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf580a3040 con 0x7fdf581d0aa0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 client.65088.objecter ms_dispatch 0x7fdf58000ba0 osd_map(529..529 src has 1..529) 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 3 client.65088.objecter handle_osd_map got epochs [529,529] > 0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 3 client.65088.objecter handle_osd_map decoding full epoch 529 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 20 client.65088.objecter dump_active .. 0 homeless 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 client.65088.objecter ms_handle_connect 0x7fdf581d0aa0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 client.65088.objecter resend_mon_ops 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf56ffd640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 0x7fdf581d0e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf58134a40 tx=0x7fdf4000f910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdf40019020 con 0x7fdf581d0aa0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdf38038e10 0x7fdf3803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: dump: 2024-04-19T00:58:31.137 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.138 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 10 monclient: get_auth_request con 0x7fdf38038e10 auth_method 0 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdf40014670 con 0x7fdf581d0aa0 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 msgr2=0x7fdf581d0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 0x7fdf581d0e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf58134a40 tx=0x7fdf4000f910 comp rx=0 tx=0).stop 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 msgr2=0x7fdf581d0e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d0aa0 0x7fdf581d0e80 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf58134a40 tx=0x7fdf4000f910 comp rx=0 tx=0).stop 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf4ffff640 10 client.65088.objecter ms_handle_connect 0x7fdf38038e10 2024-04-19T00:58:31.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.115+0000 7fdf57fff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdf38038e10 0x7fdf3803b2d0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fdf50006aa0 tx=0x7fdf50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 0x7fdf58137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdf58068240 con 0x7fdf581d13c0 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 0x7fdf58137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient: get_auth_request con 0x7fdf581d13c0 auth_method 0 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 10 monclient(hunting): handle_auth_done global_id 134131 payload 824 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 1 -- 172.21.15.43:0/3569590815 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf5809d2d0 con 0x7fdf581d13c0 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 client.65088.objecter ms_handle_connect 0x7fdf581d13c0 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 client.65088.objecter resend_mon_ops 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf56ffd640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 0x7fdf58137990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf40000c10 tx=0x7fdf400046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdf40019040 con 0x7fdf581d13c0 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:31.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 monclient: dump: 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.143 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7fdf4001e420 con 0x7fdf581d13c0 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 msgr2=0x7fdf58137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 0x7fdf58137990 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdf40000c10 tx=0x7fdf400046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 msgr2=0x7fdf58137990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf4ffff640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdf581d13c0 0x7fdf58137990 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:31.144 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15827, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:58:31.145 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.146 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.147 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:31.148 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:31.149 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:31.150 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.151 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.152 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.153 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.154 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.155 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.156 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.157 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:31.158 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.159 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.160 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:31.161 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:31.162 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.163 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:31.164 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:31.165 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:31.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 10 librados: watch_flush enter 2024-04-19T00:58:31.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 10 librados: watch_flush exit 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 20 client.65088.objecter shutdown clearing up homeless session... 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 10 client.65088.objecter successfully canceled tick 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdf38038e10 msgr2=0x7fdf3803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdf38038e10 0x7fdf3803b2d0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fdf50006aa0 tx=0x7fdf50008040 comp rx=0 tx=0).stop 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 10 monclient: shutdown 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf580a5b80 msgr2=0x7fdf58133430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 --2- 172.21.15.43:0/3569590815 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdf580a5b80 0x7fdf58133430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009040 tx=0x7fdf480196b0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf57fff640 1 -- 172.21.15.43:0/3569590815 reap_dead start 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf57fff640 1 -- 172.21.15.43:0/3569590815 reap_dead start 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 shutdown_connections 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 >> 172.21.15.43:0/3569590815 conn(0x7fdf580ab2f0 msgr2=0x7fdf580a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 shutdown_connections 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 -- 172.21.15.43:0/3569590815 wait complete. 2024-04-19T00:58:31.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.123+0000 7fdf5e111640 1 librados: shutdown 2024-04-19T00:58:31.168 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:31.168 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:58:31.168 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:58:31.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.287+0000 7fa32a401640 10 monclient: get_monmap_and_config 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.287+0000 7fa32a401640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.287+0000 7fa32a401640 10 monclient: monmap: 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:31.294313+0000 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:31.294313+0000 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.291 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:31.292 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 Processor -- start 2024-04-19T00:58:31.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 -- start start 2024-04-19T00:58:31.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient: init 2024-04-19T00:58:31.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient: _add_conns ranks=[3,7,9,8,5,6,1,10,2,0,4] 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa324138cd0 0x7fa3241390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): picked mon.noname-g con 0x7fa324138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3240a5b80 0x7fa3241395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): picked mon.noname-i con 0x7fa3240a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3241358b0 0x7fa324135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): picked mon.noname-a con 0x7fa3241358b0 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fa3240a4410 con 0x7fa3241358b0 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa324139b30 con 0x7fa324138cd0 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa3240a3dc0 con 0x7fa3240a5b80 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32a401640 10 monclient(hunting): authenticate will time out at 1781.421473s 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3241358b0 0x7fa324135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa3241358b0 0x7fa324135d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58216/0 (socket says 172.21.15.43:58216) 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 -- 172.21.15.43:0/1078446972 learned_addr learned my addr 172.21.15.43:0/1078446972 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): get_auth_request con 0x7fa3241358b0 auth_method 0 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa323fff640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa324138cd0 0x7fa3241390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa323fff640 10 monclient(hunting): get_auth_request con 0x7fa324138cd0 auth_method 0 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa323fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa323fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa3237fe640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3240a5b80 0x7fa3241395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient(hunting): handle_auth_done global_id 65097 payload 274 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 -- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3240a5b80 msgr2=0x7fa3241395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3240a5b80 0x7fa3241395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 -- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa324138cd0 msgr2=0x7fa3241390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa324138cd0 0x7fa3241390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 monclient: found mon.noname-a 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa32899e640 1 -- 172.21.15.43:0/1078446972 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3240a40a0 con 0x7fa3241358b0 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa3237fe640 10 monclient: get_auth_request con 0x7fa3240a5b80 auth_method 0 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.291+0000 7fa3237fe640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3240a5b80 0x7fa3241395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:58:31.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32899e640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3241358b0 0x7fa324135d60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009870 tx=0x7fa31c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d661c61e80a84cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 1 -- 172.21.15.43:0/1078446972 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa31c0b0070 con 0x7fa3241358b0 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: dump: 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.299 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 1 -- 172.21.15.43:0/1078446972 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa31c0aa810 con 0x7fa3241358b0 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 1 -- 172.21.15.43:0/1078446972 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa31c0a9300 con 0x7fa3241358b0 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 5 monclient: authenticate success, global_id 65097 2024-04-19T00:58:31.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 4 set_mon_vals no callback set 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa322ffd640 10 monclient: dump: 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.301 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.302 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.302 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.302 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.302 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: shutdown 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/1078446972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3241358b0 msgr2=0x7fa324135d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3241358b0 0x7fa324135d60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009870 tx=0x7fa31c01c3c0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/1078446972 shutdown_connections 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa3241358b0 0x7fa324135d60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa3240a5b80 0x7fa3241395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- 172.21.15.43:0/1078446972 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa324138cd0 0x7fa3241390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/1078446972 >> 172.21.15.43:0/1078446972 conn(0x7fa3240ab2f0 msgr2=0x7fa3240ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/1078446972 shutdown_connections 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/1078446972 wait complete. 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: monmap: 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.310 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 librados: starting msgr at 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 librados: starting objecter 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 Processor -- start 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- start start 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 librados: setting wanted keys 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 librados: calling monclient init 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: init 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: _add_conns ranks=[6,1,5,7,4,2,0,8,3] 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3240a5b80 0x7fa3241d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): picked mon.noname-d con 0x7fa3240a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3241358b0 0x7fa3241d1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): picked mon.noname-f con 0x7fa3241358b0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa324138cd0 0x7fa3241d17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): picked mon.noname-h con 0x7fa324138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa3240a2010 con 0x7fa3241358b0 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa3240a1e90 con 0x7fa324138cd0 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa324142aa0 con 0x7fa3240a5b80 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient(hunting): authenticate will time out at 1781.426815s 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3240a5b80 0x7fa3241d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3240a5b80 0x7fa3241d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42696/0 (socket says 172.21.15.43:42696) 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 -- 172.21.15.43:0/2608518427 learned_addr learned my addr 172.21.15.43:0/2608518427 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): get_auth_request con 0x7fa3240a5b80 auth_method 0 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa3237fe640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3241358b0 0x7fa3241d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa3237fe640 10 monclient(hunting): get_auth_request con 0x7fa3241358b0 auth_method 0 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa3237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa3237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32899e640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa324138cd0 0x7fa3241d17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient(hunting): handle_auth_done global_id 74580 payload 824 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa324138cd0 msgr2=0x7fa3241d17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa324138cd0 0x7fa3241d17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3241358b0 msgr2=0x7fa3241d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa3241358b0 0x7fa3241d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 monclient: found mon.noname-d 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32413c940 con 0x7fa3240a5b80 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa323fff640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3240a5b80 0x7fa3241d0d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa314007ef0 tx=0x7fa31400e4b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 client.?.objecter ms_handle_connect 0x7fa3240a5b80 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa314014890 con 0x7fa3240a5b80 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: dump: 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.315 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 5 monclient: authenticate success, global_id 74580 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: _renew_subs 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa314019070 con 0x7fa3240a5b80 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3240a3590 con 0x7fa3240a5b80 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa314014a30 con 0x7fa3240a5b80 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa320ff9640 10 monclient: dump: 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.317 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa3227fc640 4 set_mon_vals no callback set 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 client.74580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:31.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: _renew_subs 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32413ec30 con 0x7fa3240a5b80 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.295+0000 7fa32a401640 1 librados: init done 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32a401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32a401640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 0x7fa32402c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa32406d140 con 0x7fa324135060 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 0x7fa32402c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient: get_auth_request con 0x7fa324135060 auth_method 0 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa314014bd0 con 0x7fa3240a5b80 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 10 monclient(hunting): handle_auth_done global_id 74589 payload 824 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa32413e800 con 0x7fa324135060 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa310038e10 0x7fa31003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(529..529 src has 1..529) ==== 6625+0+0 (secure 0 0 0) 0x7fa314049fd0 con 0x7fa3240a5b80 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 client.74580.objecter ms_dispatch 0x7fa324000ba0 osd_map(529..529 src has 1..529) 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 3 client.74580.objecter handle_osd_map got epochs [529,529] > 0 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 3 client.74580.objecter handle_osd_map decoding full epoch 529 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa3237fe640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 0x7fa32402c450 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa32413c420 tx=0x7fa31800ec00 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 20 client.74580.objecter dump_active .. 0 homeless 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 client.74580.objecter ms_handle_connect 0x7fa324135060 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 client.74580.objecter resend_mon_ops 2024-04-19T00:58:31.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa318019020 con 0x7fa324135060 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 monclient: dump: 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.321 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32899e640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa310038e10 0x7fa31003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32899e640 10 monclient: get_auth_request con 0x7fa310038e10 auth_method 0 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa31800a750 con 0x7fa324135060 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 msgr2=0x7fa32402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 0x7fa32402c450 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa32413c420 tx=0x7fa31800ec00 comp rx=0 tx=0).stop 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 msgr2=0x7fa32402c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135060 0x7fa32402c450 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa320ff9640 10 client.74580.objecter ms_handle_connect 0x7fa310038e10 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.299+0000 7fa32899e640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa310038e10 0x7fa31003b2d0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0ad000 tx=0x7fa31c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.303+0000 7fa32a401640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 0x7fa32402c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa324068240 con 0x7fa324135440 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 0x7fa32402c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient: get_auth_request con 0x7fa324135440 auth_method 0 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 10 monclient(hunting): handle_auth_done global_id 74598 payload 824 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 1 -- 172.21.15.43:0/2608518427 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa3240693b0 con 0x7fa324135440 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa3237fe640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 0x7fa32402c990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa318000c10 tx=0x7fa3180046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 client.74580.objecter ms_handle_connect 0x7fa324135440 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 client.74580.objecter resend_mon_ops 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa318019040 con 0x7fa324135440 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 monclient: dump: 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.324 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fa318020020 con 0x7fa324135440 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 msgr2=0x7fa32402c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 0x7fa32402c990 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa318000c10 tx=0x7fa3180046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 msgr2=0x7fa32402c990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa320ff9640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa324135440 0x7fa32402c990 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 290920, 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:31.326 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.327 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:31.329 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.330 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:31.331 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.332 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.335 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.337 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.340 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:31.344 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.347 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.348 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:31.348 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 10 librados: watch_flush enter 2024-04-19T00:58:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 10 librados: watch_flush exit 2024-04-19T00:58:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 20 client.74580.objecter shutdown clearing up homeless session... 2024-04-19T00:58:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 10 client.74580.objecter successfully canceled tick 2024-04-19T00:58:31.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa310038e10 msgr2=0x7fa31003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa310038e10 0x7fa31003b2d0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7fa31c0ad000 tx=0x7fa31c0a6000 comp rx=0 tx=0).stop 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 10 monclient: shutdown 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3240a5b80 msgr2=0x7fa3241d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 --2- 172.21.15.43:0/2608518427 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa3240a5b80 0x7fa3241d0d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa314007ef0 tx=0x7fa31400e4b0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32899e640 1 -- 172.21.15.43:0/2608518427 reap_dead start 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32899e640 1 -- 172.21.15.43:0/2608518427 reap_dead start 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 shutdown_connections 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 >> 172.21.15.43:0/2608518427 conn(0x7fa3240ab2f0 msgr2=0x7fa3240aa880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 shutdown_connections 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 -- 172.21.15.43:0/2608518427 wait complete. 2024-04-19T00:58:31.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.307+0000 7fa32a401640 1 librados: shutdown 2024-04-19T00:58:31.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:31.351 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:58:31.351 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:58:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient: get_monmap_and_config 2024-04-19T00:58:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient: monmap: 2024-04-19T00:58:31.471 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:31.474538+0000 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:31.474538+0000 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:31.472 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:31.473 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:31.474 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 Processor -- start 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 -- start start 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient: init 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient: _add_conns ranks=[7,4,5,1,2,0,3,6,10,9,8] 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f921809e4c0 0x7f9218136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): picked mon.noname-i con 0x7f921809e4c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 0x7f92180a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): picked mon.noname-c con 0x7f921809e8a0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f921809eca0 0x7f92180a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): picked mon.noname-h con 0x7f921809eca0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f92180a1c50 con 0x7f921809e8a0 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f92180a1e20 con 0x7f921809eca0 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f921806c760 con 0x7f921809e4c0 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921fbc3640 10 monclient(hunting): authenticate will time out at 1781.601883s 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 0x7f92180a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 0x7f92180a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:46268/0 (socket says 172.21.15.43:46268) 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 -- 172.21.15.43:0/332513859 learned_addr learned my addr 172.21.15.43:0/332513859 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): get_auth_request con 0x7f921809e8a0 auth_method 0 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921e160640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f921809eca0 0x7f92180a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d95f640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f921809e4c0 0x7f9218136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921e160640 10 monclient(hunting): get_auth_request con 0x7f921809eca0 auth_method 0 2024-04-19T00:58:31.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921e160640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921e160640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921e160640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d95f640 10 monclient(hunting): get_auth_request con 0x7f921809e4c0 auth_method 0 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d95f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d95f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d95f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient(hunting): handle_auth_done global_id 134140 payload 274 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 -- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f921809e4c0 msgr2=0x7f9218136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f921809e4c0 0x7f9218136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 -- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f921809eca0 msgr2=0x7f92180a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f921809eca0 0x7f92180a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 monclient: found mon.noname-c 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.471+0000 7f921d15e640 1 -- 172.21.15.43:0/332513859 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92180a5510 con 0x7f921809e8a0 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 0x7f92180a3940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92080098a0 tx=0x7f920801c4d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3977a59e5656b990 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 1 -- 172.21.15.43:0/332513859 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92080b0070 con 0x7f921809e8a0 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: dump: 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.488 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 5 monclient: authenticate success, global_id 134140 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 1 -- 172.21.15.43:0/332513859 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92080aa810 con 0x7f921809e8a0 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 1 -- 172.21.15.43:0/332513859 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92080a9300 con 0x7f921809e8a0 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 4 set_mon_vals no callback set 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921c95d640 10 monclient: dump: 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.490 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: shutdown 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- 172.21.15.43:0/332513859 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 msgr2=0x7f92180a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 0x7f92180a3940 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f92080098a0 tx=0x7f920801c4d0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- 172.21.15.43:0/332513859 shutdown_connections 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f921809eca0 0x7f92180a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f921809e8a0 0x7f92180a3940 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- 172.21.15.43:0/332513859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f921809e4c0 0x7f9218136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- 172.21.15.43:0/332513859 >> 172.21.15.43:0/332513859 conn(0x7f92180a85c0 msgr2=0x7f92180a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- 172.21.15.43:0/332513859 shutdown_connections 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- 172.21.15.43:0/332513859 wait complete. 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: monmap: 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:31.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 librados: starting msgr at 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 librados: starting objecter 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 Processor -- start 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- start start 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 librados: setting wanted keys 2024-04-19T00:58:31.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 librados: calling monclient init 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: init 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient: _add_conns ranks=[3,2,1,7,6,5,4,8,0] 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921809e4c0 0x7f92181d0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): picked mon.noname-g con 0x7f921809e4c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 0x7f92181d1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): picked mon.noname-b con 0x7f921809e8a0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921809eca0 0x7f921813be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): picked mon.noname-f con 0x7f921809eca0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92181406e0 con 0x7f921809eca0 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f92180a47d0 con 0x7f921809e8a0 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9218140860 con 0x7f921809e4c0 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921fbc3640 10 monclient(hunting): authenticate will time out at 1781.607232s 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 0x7f92181d1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 0x7f92181d1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35772/0 (socket says 172.21.15.43:35772) 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 -- 172.21.15.43:0/113383287 learned_addr learned my addr 172.21.15.43:0/113383287 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): get_auth_request con 0x7f921809e8a0 auth_method 0 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d95f640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921809e4c0 0x7f92181d0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d95f640 10 monclient(hunting): get_auth_request con 0x7f921809e4c0 auth_method 0 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d95f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d95f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d95f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921e160640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921809eca0 0x7f921813be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921e160640 10 monclient(hunting): get_auth_request con 0x7f921809eca0 auth_method 0 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921e160640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921e160640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921e160640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient(hunting): handle_auth_done global_id 74594 payload 824 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921809e4c0 msgr2=0x7f92181d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f921809e4c0 0x7f92181d0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921809eca0 msgr2=0x7f921813be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f921809eca0 0x7f921813be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 monclient: found mon.noname-b 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921806bda0 con 0x7f921809e8a0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f921d15e640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 0x7f92181d1490 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f92080ae040 tx=0x7f9208019270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f920e7fc640 10 client.?.objecter ms_handle_connect 0x7f921809e8a0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.475+0000 7f920e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92080b9070 con 0x7f921809e8a0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:58:31.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: dump: 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.498 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92080044b0 con 0x7f921809e8a0 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f92080b0040 con 0x7f921809e8a0 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 5 monclient: authenticate success, global_id 74594 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 monclient: _renew_subs 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92180a0830 con 0x7f921809e8a0 2024-04-19T00:58:31.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920ffff640 4 set_mon_vals no callback set 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: dump: 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.500 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 client.74594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 monclient: _renew_subs 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92180a0550 con 0x7f921809e8a0 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 1 librados: init done 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 0x7f921813c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f921813cf90 con 0x7f921813c3c0 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 0x7f921813c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9208004040 con 0x7f921809e8a0 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient: get_auth_request con 0x7f921813c3c0 auth_method 0 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91fc038e30 0x7f91fc03b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(530..530 src has 1..530) ==== 6625+0+0 (secure 0 0 0) 0x7f92080e2990 con 0x7f921809e8a0 2024-04-19T00:58:31.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 client.74594.objecter ms_dispatch 0x7f9218000ba0 osd_map(530..530 src has 1..530) 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 3 client.74594.objecter handle_osd_map got epochs [530,530] > 0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 3 client.74594.objecter handle_osd_map decoding full epoch 530 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 10 monclient(hunting): handle_auth_done global_id 114326 payload 824 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f921813cce0 con 0x7f921813c3c0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 20 client.74594.objecter dump_active .. 0 homeless 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921e160640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91fc038e30 0x7f91fc03b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921d95f640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 0x7f921813c7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92100045c0 tx=0x7f9210007700 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 client.74594.objecter ms_handle_connect 0x7f921813c3c0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 client.74594.objecter resend_mon_ops 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921e160640 10 monclient: get_auth_request con 0x7f91fc038e30 auth_method 0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9210014880 con 0x7f921813c3c0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:31.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: dump: 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.504 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9210019070 con 0x7f921813c3c0 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 msgr2=0x7f921813c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 0x7f921813c7a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92100045c0 tx=0x7f9210007700 comp rx=0 tx=0).stop 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 msgr2=0x7f921813c7a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f921813c3c0 0x7f921813c7a0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f92100045c0 tx=0x7f9210007700 comp rx=0 tx=0).stop 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f920e7fc640 10 client.74594.objecter ms_handle_connect 0x7f91fc038e30 2024-04-19T00:58:31.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.479+0000 7f921e160640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91fc038e30 0x7f91fc03b2f0 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7f921400d850 tx=0x7f9214008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 0x7f92180a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f921806d140 con 0x7f92180a5e10 2024-04-19T00:58:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 0x7f92180a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient: get_auth_request con 0x7f92180a5e10 auth_method 0 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 10 monclient(hunting): handle_auth_done global_id 114335 payload 824 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 1 -- 172.21.15.43:0/113383287 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f921809f730 con 0x7f92180a5e10 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 client.74594.objecter ms_handle_connect 0x7f92180a5e10 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 client.74594.objecter resend_mon_ops 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921d95f640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 0x7f92180a6220 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92100045c0 tx=0x7f9210004a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9210029400 con 0x7f92180a5e10 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 monclient: dump: 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.509 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f9210014ab0 con 0x7f92180a5e10 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 msgr2=0x7f92180a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 0x7f92180a6220 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92100045c0 tx=0x7f9210004a50 comp rx=0 tx=0).stop 2024-04-19T00:58:31.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 msgr2=0x7f92180a6220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f920e7fc640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f92180a5e10 0x7f92180a6220 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 148650, 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:31.511 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.512 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.513 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:31.514 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:31.515 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.516 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:31.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:31.522 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.526 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:31.528 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:31.530 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.531 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.532 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.533 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.533 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.533 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:31.533 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 10 librados: watch_flush enter 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 10 librados: watch_flush exit 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 20 client.74594.objecter shutdown clearing up homeless session... 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 10 client.74594.objecter successfully canceled tick 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91fc038e30 msgr2=0x7f91fc03b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f91fc038e30 0x7f91fc03b2f0 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7f921400d850 tx=0x7f9214008040 comp rx=0 tx=0).stop 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 10 monclient: shutdown 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 msgr2=0x7f92181d1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921e160640 1 -- 172.21.15.43:0/113383287 reap_dead start 2024-04-19T00:58:31.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 0x7f92181d1490 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f92080ae040 tx=0x7f9208019270 comp rx=0 tx=0).stop 2024-04-19T00:58:31.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 shutdown_connections 2024-04-19T00:58:31.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 --2- 172.21.15.43:0/113383287 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f921809e8a0 0x7f92181d1490 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 >> 172.21.15.43:0/113383287 conn(0x7f92180a85c0 msgr2=0x7f9218130df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 shutdown_connections 2024-04-19T00:58:31.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 -- 172.21.15.43:0/113383287 wait complete. 2024-04-19T00:58:31.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.487+0000 7f921fbc3640 1 librados: shutdown 2024-04-19T00:58:31.538 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:31.538 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:58:31.538 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:58:31.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.655+0000 7f013a8d1640 10 monclient: get_monmap_and_config 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.655+0000 7f013a8d1640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.655+0000 7f013a8d1640 10 monclient: monmap: 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:31.661086+0000 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:31.661086+0000 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.658 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:31.659 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 Processor -- start 2024-04-19T00:58:31.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 -- start start 2024-04-19T00:58:31.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient: init 2024-04-19T00:58:31.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient: _add_conns ranks=[9,4,1,0,8,5,2,3,6,10,7] 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0134135ca0 0x7f0134073000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): picked mon.noname-a con 0x7f0134135ca0 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0134136080 0x7f01340a6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): picked mon.noname-c con 0x7f0134136080 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f01341364e0 addr v1:172.21.15.161:6789/0 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0134135800 con 0x7f01341364e0 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f0134071210 con 0x7f0134135ca0 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0134071d80 con 0x7f0134136080 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f013a8d1640 10 monclient(hunting): authenticate will time out at 1781.788940s 2024-04-19T00:58:31.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0134135ca0 0x7f0134073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0134135ca0 0x7f0134073000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58220/0 (socket says 172.21.15.43:58220) 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 1 --2- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0134136080 0x7f01340a6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 -- 172.21.15.43:0/3604821668 learned_addr learned my addr 172.21.15.43:0/3604821668 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): get_auth_request con 0x7f0134135ca0 auth_method 0 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): get_auth_request con 0x7f0134136080 auth_method 0 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f012bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient(hunting): handle_auth_done global_id 65106 payload 274 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 -- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0134136080 msgr2=0x7f01340a6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 --2- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0134136080 0x7f01340a6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 -- 172.21.15.43:0/3604821668 >> v1:172.21.15.161:6789/0 conn(0x7f01341364e0 legacy=0x7f01340a6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 monclient: found mon.noname-a 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:31.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.659+0000 7f0133fff640 1 -- 172.21.15.43:0/3604821668 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0134072060 con 0x7f0134135ca0 2024-04-19T00:58:31.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f0133fff640 1 --2- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0134135ca0 0x7f0134073000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0124009870 tx=0x7f012401c210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01fa3b1f9ebd73e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 1 -- 172.21.15.43:0/3604821668 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01240b0070 con 0x7f0134135ca0 2024-04-19T00:58:31.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: dump: 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.666 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 1 -- 172.21.15.43:0/3604821668 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01240aa810 con 0x7f0134135ca0 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 1 -- 172.21.15.43:0/3604821668 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01240a9300 con 0x7f0134135ca0 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f01337fe640 10 monclient: dump: 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.667 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.668 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 5 monclient: authenticate success, global_id 65106 2024-04-19T00:58:31.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:31.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 4 set_mon_vals no callback set 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 10 monclient: shutdown 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 -- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0134135ca0 msgr2=0x7f0134073000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0134135ca0 0x7f0134073000 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0124009870 tx=0x7f012401c210 comp rx=0 tx=0).stop 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 -- 172.21.15.43:0/3604821668 shutdown_connections 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0134136080 0x7f01340a6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3604821668 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0134135ca0 0x7f0134073000 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 -- 172.21.15.43:0/3604821668 >> 172.21.15.43:0/3604821668 conn(0x7f01340a7ef0 msgr2=0x7f01340a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 -- 172.21.15.43:0/3604821668 shutdown_connections 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.663+0000 7f013a8d1640 1 -- 172.21.15.43:0/3604821668 wait complete. 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: monmap: 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.677 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 librados: starting msgr at 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 librados: starting objecter 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 Processor -- start 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 -- start start 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 librados: setting wanted keys 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 librados: calling monclient init 2024-04-19T00:58:31.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: init 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: _add_conns ranks=[5,1,3,2,8,4,6,0,7] 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0134135ca0 0x7f01341d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): picked mon.noname-h con 0x7f0134135ca0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 0x7f01341d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): picked mon.noname-f con 0x7f0134136080 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01341364e0 0x7f01341d1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): picked mon.noname-g con 0x7f01341364e0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f01340a75d0 con 0x7f0134136080 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f01340a7450 con 0x7f01341364e0 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0134140360 con 0x7f0134135ca0 2024-04-19T00:58:31.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient(hunting): authenticate will time out at 1781.797796s 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0134135ca0 0x7f01341d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 0x7f01341d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01341364e0 0x7f01341d1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 0x7f01341d0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:39988/0 (socket says 172.21.15.43:39988) 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01341364e0 0x7f01341d1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:39364/0 (socket says 172.21.15.43:39364) 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 -- 172.21.15.43:0/3897880282 learned_addr learned my addr 172.21.15.43:0/3897880282 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): get_auth_request con 0x7f0134136080 auth_method 0 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): get_auth_request con 0x7f01341364e0 auth_method 0 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): get_auth_request con 0x7f0134135ca0 auth_method 0 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient(hunting): handle_auth_done global_id 114337 payload 824 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0134135ca0 msgr2=0x7f01341d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0134135ca0 0x7f01341d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01341364e0 msgr2=0x7f01341d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01341364e0 0x7f01341d1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 monclient: found mon.noname-f 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01340713e0 con 0x7f0134136080 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0133fff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0134135ca0 0x7f01341d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0138e6e640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f01341364e0 0x7f01341d1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f012bfff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 0x7f01341d0ad0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f011c00e900 tx=0x7f011c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 client.?.objecter ms_handle_connect 0x7f0134136080 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f011c017070 con 0x7f0134136080 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: dump: 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.683 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f011c013470 con 0x7f0134136080 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 5 monclient: authenticate success, global_id 114337 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: _renew_subs 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:58:31.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013413c660 con 0x7f0134136080 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f011c013a90 con 0x7f0134136080 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f0132ffd640 4 set_mon_vals no callback set 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f01317fa640 10 monclient: dump: 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.685 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 client.114337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: _renew_subs 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f013413a370 con 0x7f0134136080 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.667+0000 7f013a8d1640 1 librados: init done 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f013a8d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 0x7f013412f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f013406d140 con 0x7f013412f1e0 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 0x7f013412f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f011c01a070 con 0x7f0134136080 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient: get_auth_request con 0x7f013412f1e0 auth_method 0 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0114038e10 0x7f011403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(530..530 src has 1..530) ==== 6625+0+0 (secure 0 0 0) 0x7f011c00f040 con 0x7f0134136080 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 client.114337.objecter ms_dispatch 0x7f0134000ba0 osd_map(530..530 src has 1..530) 2024-04-19T00:58:31.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 3 client.114337.objecter handle_osd_map got epochs [530,530] > 0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 3 client.114337.objecter handle_osd_map decoding full epoch 530 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 20 client.114337.objecter dump_active .. 0 homeless 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0138e6e640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0114038e10 0x7f011403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 10 monclient(hunting): handle_auth_done global_id 114346 payload 824 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01341387a0 con 0x7f013412f1e0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0138e6e640 10 monclient: get_auth_request con 0x7f0114038e10 auth_method 0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 client.114337.objecter ms_handle_connect 0x7f013412f1e0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 client.114337.objecter resend_mon_ops 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0133fff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 0x7f013412f5c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f01340a78a0 tx=0x7f0124018fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01240ba030 con 0x7f013412f1e0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 monclient: dump: 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.688 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 client.114337.objecter ms_handle_connect 0x7f0114038e10 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f0138e6e640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0114038e10 0x7f011403b2d0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f013406d960 tx=0x7f012c00d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f01240aa810 con 0x7f013412f1e0 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 msgr2=0x7f013412f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 0x7f013412f5c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f01340a78a0 tx=0x7f0124018fb0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 msgr2=0x7f013412f5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.671+0000 7f01317fa640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412f1e0 0x7f013412f5c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f013a8d1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 0x7f0134130270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f013a8d1640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f013413c000 con 0x7f013412fb00 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 0x7f0134130270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient: get_auth_request con 0x7f013412fb00 auth_method 0 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 10 monclient(hunting): handle_auth_done global_id 114355 payload 824 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 1 -- 172.21.15.43:0/3897880282 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f013406dab0 con 0x7f013412fb00 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f0133fff640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 0x7f0134130270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01240097c0 tx=0x7f01240a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 client.114337.objecter ms_handle_connect 0x7f013412fb00 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 client.114337.objecter resend_mon_ops 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f01240ba040 con 0x7f013412fb00 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 monclient: dump: 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.693 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f01240038f0 con 0x7f013412fb00 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 msgr2=0x7f0134130270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 0x7f0134130270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01240097c0 tx=0x7f01240a8000 comp rx=0 tx=0).stop 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 msgr2=0x7f0134130270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.679+0000 7f01317fa640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f013412fb00 0x7f0134130270 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f01240097c0 tx=0x7f01240a8000 comp rx=0 tx=0).stop 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:31.695 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 148849, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.696 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.697 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.698 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:31.699 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.700 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:31.701 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.704 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:31.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:31.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.716 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:31.718 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:31.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 10 librados: watch_flush enter 2024-04-19T00:58:31.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 10 librados: watch_flush exit 2024-04-19T00:58:31.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 20 client.114337.objecter shutdown clearing up homeless session... 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 10 client.114337.objecter successfully canceled tick 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0114038e10 msgr2=0x7f011403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0114038e10 0x7f011403b2d0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f013406d960 tx=0x7f012c00d040 comp rx=0 tx=0).stop 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 10 monclient: shutdown 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 msgr2=0x7f01341d0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f0138e6e640 1 -- 172.21.15.43:0/3897880282 reap_dead start 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 0x7f01341d0ad0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f011c00e900 tx=0x7f011c00edc0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 shutdown_connections 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 --2- 172.21.15.43:0/3897880282 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0134136080 0x7f01341d0ad0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 >> 172.21.15.43:0/3897880282 conn(0x7f01340a7ef0 msgr2=0x7f01340a2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 shutdown_connections 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 -- 172.21.15.43:0/3897880282 wait complete. 2024-04-19T00:58:31.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.683+0000 7f013a8d1640 1 librados: shutdown 2024-04-19T00:58:31.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:31.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:58:31.722 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:58:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient: get_monmap_and_config 2024-04-19T00:58:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient: monmap: 2024-04-19T00:58:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:31.839768+0000 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:31.839768+0000 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:31.838 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:31.838 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 Processor -- start 2024-04-19T00:58:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 -- start start 2024-04-19T00:58:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient: init 2024-04-19T00:58:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient: _add_conns ranks=[0,8,10,5,3,9,7,6,2,1,4] 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2328138a50 addr v1:172.21.15.43:6789/0 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328139540 0x7f232813dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): picked mon.noname-e con 0x7f2328139540 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f23280a5bb0 0x7f2328140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): picked mon.noname-f con 0x7f23280a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23280a0d40 con 0x7f2328138a50 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f23280a1020 con 0x7f23280a5bb0 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f23280a0780 con 0x7f2328139540 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f232d8cd640 10 monclient(hunting): authenticate will time out at 1781.967739s 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f2326ffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f2328138a50 0x7f2328138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38012/0 (socket says 172.21.15.43:38012) 2024-04-19T00:58:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f2326ffd640 1 -- 172.21.15.43:0/1936605408 learned_addr learned my addr 172.21.15.43:0/1936605408 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:58:31.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f23267fc640 1 --2- 172.21.15.43:0/1936605408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328139540 0x7f232813dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f23267fc640 10 monclient(hunting): get_auth_request con 0x7f2328139540 auth_method 0 2024-04-19T00:58:31.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f23267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f23267fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.835+0000 7f23267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23277fe640 1 --2- 172.21.15.43:0/1936605408 >> v2:172.21.15.161:3300/0 conn(0x7f23280a5bb0 0x7f2328140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23277fe640 10 monclient(hunting): get_auth_request con 0x7f23280a5bb0 auth_method 0 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23277fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 1 -- 172.21.15.43:0/1936605408 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1264838651 0 0) 0x7f23280a0d40 con 0x7f2328138a50 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient(hunting): _negotiate 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient(hunting): my global_id is 65115 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 1 -- 172.21.15.43:0/1936605408 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2310003530 con 0x7f2328138a50 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 10 monclient(hunting): handle_auth_done global_id 114344 payload 274 2024-04-19T00:58:31.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 1 -- 172.21.15.43:0/1936605408 >> v2:172.21.15.161:3300/0 conn(0x7f23280a5bb0 msgr2=0x7f2328140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 1 --2- 172.21.15.43:0/1936605408 >> v2:172.21.15.161:3300/0 conn(0x7f23280a5bb0 0x7f2328140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 1 -- 172.21.15.43:0/1936605408 >> v1:172.21.15.43:6789/0 conn(0x7f2328138a50 legacy=0x7f2328138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 1 monclient: found mon.noname-e 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:31.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 1 -- 172.21.15.43:0/1936605408 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23280a0a60 con 0x7f2328139540 2024-04-19T00:58:31.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f23267fc640 1 --2- 172.21.15.43:0/1936605408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328139540 0x7f232813dfe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f231c0097c0 tx=0x7f231c01c360 comp rx=0 tx=0).ready entity=mon.8 client_cookie=765e5b257fec4b00 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 1 -- 172.21.15.43:0/1936605408 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f231c0b0070 con 0x7f2328139540 2024-04-19T00:58:31.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: dump: 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.853 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 1 -- 172.21.15.43:0/1936605408 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231c0aa810 con 0x7f2328139540 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 5 monclient: authenticate success, global_id 114344 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 1 -- 172.21.15.43:0/1936605408 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f231c0aae30 con 0x7f2328139540 2024-04-19T00:58:31.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 4 set_mon_vals no callback set 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f2325ffb640 10 monclient: dump: 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.855 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 10 monclient: shutdown 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 -- 172.21.15.43:0/1936605408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328139540 msgr2=0x7f232813dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 --2- 172.21.15.43:0/1936605408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328139540 0x7f232813dfe0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f231c0097c0 tx=0x7f231c01c360 comp rx=0 tx=0).stop 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 -- 172.21.15.43:0/1936605408 shutdown_connections 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 --2- 172.21.15.43:0/1936605408 >> v2:172.21.15.161:3300/0 conn(0x7f23280a5bb0 0x7f2328140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 --2- 172.21.15.43:0/1936605408 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328139540 0x7f232813dfe0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 -- 172.21.15.43:0/1936605408 >> 172.21.15.43:0/1936605408 conn(0x7f23280a7cb0 msgr2=0x7f23280a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 -- 172.21.15.43:0/1936605408 shutdown_connections 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 1 -- 172.21.15.43:0/1936605408 wait complete. 2024-04-19T00:58:31.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 10 monclient: build_initial_monmap 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.839+0000 7f232d8cd640 10 monclient: monmap: 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:31.857 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 librados: starting msgr at 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 librados: starting objecter 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 Processor -- start 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- start start 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 librados: setting wanted keys 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 librados: calling monclient init 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: init 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: _add_conns ranks=[2,8,6,3,1,4,0,7,5] 2024-04-19T00:58:31.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23280a5bb0 0x7f2328121800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): picked mon.noname-b con 0x7f23280a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328138a50 0x7f232813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): picked mon.noname-e con 0x7f2328138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2328139540 0x7f2328137ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): picked mon.noname-d con 0x7f2328139540 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2328143620 con 0x7f23280a5bb0 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f23281434a0 con 0x7f2328139540 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f23281437a0 con 0x7f2328138a50 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): authenticate will time out at 1781.972718s 2024-04-19T00:58:31.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23280a5bb0 0x7f2328121800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2328139540 0x7f2328137ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328138a50 0x7f232813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23280a5bb0 0x7f2328121800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35788/0 (socket says 172.21.15.43:35788) 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 1 -- 172.21.15.43:0/962622812 learned_addr learned my addr 172.21.15.43:0/962622812 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): get_auth_request con 0x7f23280a5bb0 auth_method 0 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): get_auth_request con 0x7f2328138a50 auth_method 0 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): get_auth_request con 0x7f2328139540 auth_method 0 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient(hunting): handle_auth_done global_id 114353 payload 824 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2328139540 msgr2=0x7f2328137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2328139540 0x7f2328137ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23280a5bb0 msgr2=0x7f2328121800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23280a5bb0 0x7f2328121800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient: _finish_hunting 0 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 monclient: found mon.noname-e 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232809fc90 con 0x7f2328138a50 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f23280a5bb0 0x7f2328121800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23267fc640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328138a50 0x7f232813ced0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f23280a72d0 tx=0x7f231c019100 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 client.?.objecter ms_handle_connect 0x7f2328138a50 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f231c0b0040 con 0x7f2328138a50 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:31.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: dump: 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.863 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: _finish_auth 0 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231c0044b0 con 0x7f2328138a50 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 5 monclient: authenticate success, global_id 114353 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: _renew_subs 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232809ee60 con 0x7f2328138a50 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f231c0aac80 con 0x7f2328138a50 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 client.114353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:31.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 monclient: dump: 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.865 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: _renew_subs 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23257fa640 4 set_mon_vals no callback set 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f232813eef0 con 0x7f2328138a50 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 librados: init done 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 0x7f23281d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f232806d140 con 0x7f2328138020 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f231c004040 con 0x7f2328138a50 2024-04-19T00:58:31.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2314038e10 0x7f231403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(530..530 src has 1..530) ==== 6625+0+0 (secure 0 0 0) 0x7f231c0e23b0 con 0x7f2328138a50 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 10 client.114353.objecter ms_dispatch 0x7f2328000ba0 osd_map(530..530 src has 1..530) 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 3 client.114353.objecter handle_osd_map got epochs [530,530] > 0 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 3 client.114353.objecter handle_osd_map decoding full epoch 530 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 0x7f23281d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f230ffff640 20 client.114353.objecter dump_active .. 0 homeless 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient: get_auth_request con 0x7f2328138020 auth_method 0 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2314038e10 0x7f231403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f23277fe640 10 monclient: get_auth_request con 0x7f2314038e10 auth_method 0 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 10 monclient(hunting): handle_auth_done global_id 19596 payload 824 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.843+0000 7f2326ffd640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2328138e30 con 0x7f2328138020 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f2326ffd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 0x7f23281d09a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f231800f400 tx=0x7f231800f8c0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 client.114353.objecter ms_handle_connect 0x7f2328138020 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 client.114353.objecter resend_mon_ops 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 client.114353.objecter ms_handle_connect 0x7f2314038e10 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f23277fe640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2314038e10 0x7f231403b2d0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7f232813d4a0 tx=0x7f2320008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2318007750 con 0x7f2328138020 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 monclient: dump: 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.868 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2318018070 con 0x7f2328138020 2024-04-19T00:58:31.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:31.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:31.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 msgr2=0x7f23281d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 0x7f23281d09a0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f231800f400 tx=0x7f231800f8c0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 msgr2=0x7f23281d09a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.847+0000 7f230ffff640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2328138020 0x7f23281d09a0 secure :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f231800f400 tx=0x7f231800f8c0 comp rx=0 tx=0).stop 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.851+0000 7f232d8cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.851+0000 7f232d8cd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 0x7f23281d12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.851+0000 7f232d8cd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.851+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f23281d1830 con 0x7f23281d0ee0 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 0x7f23281d12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient: get_auth_request con 0x7f23281d0ee0 auth_method 0 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:31.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 10 monclient(hunting): handle_auth_done global_id 19605 payload 824 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 1 -- 172.21.15.43:0/962622812 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23280688a0 con 0x7f23281d0ee0 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f2326ffd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 0x7f23281d12f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f231800f400 tx=0x7f2318005b80 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 client.114353.objecter ms_handle_connect 0x7f23281d0ee0 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 client.114353.objecter resend_mon_ops 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23180079b0 con 0x7f23281d0ee0 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 monclient: dump: 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:31.876 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:31.877 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7f23180079b0 con 0x7f23281d0ee0 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 msgr2=0x7f23281d12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 0x7f23281d12f0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f231800f400 tx=0x7f2318005b80 comp rx=0 tx=0).stop 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 msgr2=0x7f23281d12f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f230ffff640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f23281d0ee0 0x7f23281d12f0 secure :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f231800f400 tx=0x7f2318005b80 comp rx=0 tx=0).stop 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:31.878 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 855342, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:31.879 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.880 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.881 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:31.882 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:31.883 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:31.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:31.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:31.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:31.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:31.899 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:31.900 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:31.901 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 10 librados: watch_flush enter 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 10 librados: watch_flush exit 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 20 client.114353.objecter shutdown clearing up homeless session... 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 10 client.114353.objecter successfully canceled tick 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2314038e10 msgr2=0x7f231403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2314038e10 0x7f231403b2d0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7f232813d4a0 tx=0x7f2320008040 comp rx=0 tx=0).stop 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 10 monclient: shutdown 2024-04-19T00:58:31.903 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328138a50 msgr2=0x7f232813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 --2- 172.21.15.43:0/962622812 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2328138a50 0x7f232813ced0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f23280a72d0 tx=0x7f231c019100 comp rx=0 tx=0).stop 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f23277fe640 1 -- 172.21.15.43:0/962622812 reap_dead start 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f23277fe640 1 -- 172.21.15.43:0/962622812 reap_dead start 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 shutdown_connections 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 >> 172.21.15.43:0/962622812 conn(0x7f23280a7cb0 msgr2=0x7f23280a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 shutdown_connections 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 -- 172.21.15.43:0/962622812 wait complete. 2024-04-19T00:58:31.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:31.855+0000 7f232d8cd640 1 librados: shutdown 2024-04-19T00:58:31.904 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:31.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:58:31.905 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:58:32.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient: get_monmap_and_config 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient: build_initial_monmap 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient: monmap: 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:32.030552+0000 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:32.030552+0000 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:32.028 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 Processor -- start 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 -- start start 2024-04-19T00:58:32.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient: init 2024-04-19T00:58:32.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient: _add_conns ranks=[1,9,5,6,8,3,7,4,2,0,10] 2024-04-19T00:58:32.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0d9c0a0670 addr v1:172.21.15.161:6789/0 2024-04-19T00:58:32.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d9c138f60 0x7f0d9c13ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): picked mon.noname-a con 0x7f0d9c138f60 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:32.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c0a11c0 0x7f0d9c140310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): picked mon.noname-h con 0x7f0d9c0a11c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9c09c6d0 con 0x7f0d9c0a0670 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9c09d190 con 0x7f0d9c138f60 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0d9c0a46e0 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0da1d2e640 10 monclient(hunting): authenticate will time out at 1782.157470s 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d9c138f60 0x7f0d9c13ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f0d9c138f60 0x7f0d9c13ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58230/0 (socket says 172.21.15.43:58230) 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 -- 172.21.15.43:0/3881866584 learned_addr learned my addr 172.21.15.43:0/3881866584 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): get_auth_request con 0x7f0d9c138f60 auth_method 0 2024-04-19T00:58:32.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9bfff640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c0a11c0 0x7f0d9c140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9bfff640 10 monclient(hunting): get_auth_request con 0x7f0d9c0a11c0 auth_method 0 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient(hunting): handle_auth_done global_id 65124 payload 274 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 -- 172.21.15.43:0/3881866584 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c0a11c0 msgr2=0x7f0d9c140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c0a11c0 0x7f0d9c140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 -- 172.21.15.43:0/3881866584 >> v1:172.21.15.161:6789/0 conn(0x7f0d9c0a0670 legacy=0x7f0d9c0a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 monclient: found mon.noname-a 2024-04-19T00:58:32.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:32.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9affd640 1 -- 172.21.15.43:0/3881866584 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c09bc30 con 0x7f0d9c138f60 2024-04-19T00:58:32.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.027+0000 7f0d9bfff640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c0a11c0 0x7f0d9c140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:58:32.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9c138f60 0x7f0d9c13ddc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009870 tx=0x7f0d9001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c052aa0364e803a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 1 -- 172.21.15.43:0/3881866584 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d900b0070 con 0x7f0d9c138f60 2024-04-19T00:58:32.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: dump: 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.041 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 1 -- 172.21.15.43:0/3881866584 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d900aa810 con 0x7f0d9c138f60 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 5 monclient: authenticate success, global_id 65124 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 4 set_mon_vals no callback set 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 1 -- 172.21.15.43:0/3881866584 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d900a9300 con 0x7f0d9c138f60 2024-04-19T00:58:32.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9a7fc640 10 monclient: dump: 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.043 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: shutdown 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3881866584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9c138f60 msgr2=0x7f0d9c13ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9c138f60 0x7f0d9c13ddc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009870 tx=0x7f0d9001c420 comp rx=0 tx=0).stop 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3881866584 shutdown_connections 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c0a11c0 0x7f0d9c140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3881866584 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0d9c138f60 0x7f0d9c13ddc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3881866584 >> 172.21.15.43:0/3881866584 conn(0x7f0d9c0a7520 msgr2=0x7f0d9c0a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3881866584 shutdown_connections 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3881866584 wait complete. 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: build_initial_monmap 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: monmap: 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:32.044 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 librados: starting msgr at 2024-04-19T00:58:32.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 librados: starting objecter 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 Processor -- start 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- start start 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 librados: setting wanted keys 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 librados: calling monclient init 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: init 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient: _add_conns ranks=[3,6,5,2,4,1,8,0,7] 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0d9c0a0670 0x7f0d9c136770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): picked mon.noname-g con 0x7f0d9c0a0670 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d9c0a11c0 0x7f0d9c11d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): picked mon.noname-d con 0x7f0d9c0a11c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c1c8410 0x7f0d9c11dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): picked mon.noname-h con 0x7f0d9c1c8410 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:32.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9c143400 con 0x7f0d9c0a0670 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0d9c143280 con 0x7f0d9c1c8410 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0d9c143580 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0da1d2e640 10 monclient(hunting): authenticate will time out at 1782.163657s 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d9c0a11c0 0x7f0d9c11d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d9c0a11c0 0x7f0d9c11d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42736/0 (socket says 172.21.15.43:42736) 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 1 -- 172.21.15.43:0/3083826334 learned_addr learned my addr 172.21.15.43:0/3083826334 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): get_auth_request con 0x7f0d9c0a11c0 auth_method 0 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.031+0000 7f0d9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c1c8410 0x7f0d9c11dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 10 monclient(hunting): get_auth_request con 0x7f0d9c1c8410 auth_method 0 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0d9c0a0670 0x7f0d9c136770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): get_auth_request con 0x7f0d9c0a0670 auth_method 0 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 10 monclient(hunting): handle_auth_done global_id 74616 payload 824 2024-04-19T00:58:32.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c1c8410 msgr2=0x7f0d9c11dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c1c8410 0x7f0d9c11dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0d9c0a0670 msgr2=0x7f0d9c136770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0d9c0a0670 0x7f0d9c136770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 monclient: found mon.noname-d 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9c09c3a0 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9affd640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d9c0a11c0 0x7f0d9c11d570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0d900ae040 tx=0x7f0d900191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 client.?.objecter ms_handle_connect 0x7f0d9c0a11c0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d900b9070 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:32.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: dump: 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.050 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d900044b0 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 5 monclient: authenticate success, global_id 74616 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 monclient: _renew_subs 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d9c11ee10 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d900b0040 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:32.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d997fa640 4 set_mon_vals no callback set 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 client.74616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: dump: 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.052 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 monclient: _renew_subs 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9c121100 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 1 librados: init done 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 0x7f0d9c11dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0d9c06d140 con 0x7f0d9c120860 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0d90004040 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 0x7f0d9c11dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient: get_auth_request con 0x7f0d9c120860 auth_method 0 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d88038e10 0x7f0d8803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(530..530 src has 1..530) ==== 6625+0+0 (secure 0 0 0) 0x7f0d900e2820 con 0x7f0d9c0a11c0 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 client.74616.objecter ms_dispatch 0x7f0d9c000ba0 osd_map(530..530 src has 1..530) 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 3 client.74616.objecter handle_osd_map got epochs [530,530] > 0 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 3 client.74616.objecter handle_osd_map decoding full epoch 530 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 20 client.74616.objecter dump_active .. 0 homeless 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d88038e10 0x7f0d8803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 10 monclient: get_auth_request con 0x7f0d88038e10 auth_method 0 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_done global_id 74561 payload 824 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d9c1d13c0 con 0x7f0d9c120860 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 client.74616.objecter ms_handle_connect 0x7f0d9c120860 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 client.74616.objecter resend_mon_ops 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9b7fe640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 0x7f0d9c11dff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00e4d0 tx=0x7f0d8c00e990 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d8c019070 con 0x7f0d9c120860 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: dump: 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.055 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 client.74616.objecter ms_handle_connect 0x7f0d88038e10 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d9bfff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d88038e10 0x7f0d8803b2d0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0738e0 tx=0x7f0d94003cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d8c0052e0 con 0x7f0d9c120860 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:32.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 msgr2=0x7f0d9c11dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 0x7f0d9c11dff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00e4d0 tx=0x7f0d8c00e990 comp rx=0 tx=0).stop 2024-04-19T00:58:32.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 msgr2=0x7f0d9c11dff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:32.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.035+0000 7f0d83fff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c120860 0x7f0d9c11dff0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00e4d0 tx=0x7f0d8c00e990 comp rx=0 tx=0).stop 2024-04-19T00:58:32.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0da1d2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:32.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 0x7f0d9c11e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0da1d2e640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0d9c068240 con 0x7f0d9c138f60 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 0x7f0d9c11e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient: get_auth_request con 0x7f0d9c138f60 auth_method 0 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 10 monclient(hunting): handle_auth_done global_id 74570 payload 824 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 1 -- 172.21.15.43:0/3083826334 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d9c09c8f0 con 0x7f0d9c138f60 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d9b7fe640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 0x7f0d9c11e530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00e4d0 tx=0x7f0d8c000ee0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d83fff640 10 client.74616.objecter ms_handle_connect 0x7f0d9c138f60 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d83fff640 10 client.74616.objecter resend_mon_ops 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0d8c019070 con 0x7f0d9c138f60 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d83fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.043+0000 7f0d83fff640 10 monclient: dump: 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.060 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f0d8c0067e0 con 0x7f0d9c138f60 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 msgr2=0x7f0d9c11e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 0x7f0d9c11e530 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00e4d0 tx=0x7f0d8c000ee0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 msgr2=0x7f0d9c11e530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:32.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d83fff640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0d9c138f60 0x7f0d9c11e530 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00e4d0 tx=0x7f0d8c000ee0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 291667, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:32.062 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:32.063 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:32.064 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:32.065 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:32.066 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:32.071 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:32.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.079 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.081 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:32.082 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 10 librados: watch_flush enter 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 10 librados: watch_flush exit 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 20 client.74616.objecter shutdown clearing up homeless session... 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 10 client.74616.objecter successfully canceled tick 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d88038e10 msgr2=0x7f0d8803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0d88038e10 0x7f0d8803b2d0 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0738e0 tx=0x7f0d94003cf0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 10 monclient: shutdown 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d9c0a11c0 msgr2=0x7f0d9c11d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 --2- 172.21.15.43:0/3083826334 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0d9c0a11c0 0x7f0d9c11d570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0d900ae040 tx=0x7f0d900191c0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d9bfff640 1 -- 172.21.15.43:0/3083826334 reap_dead start 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0d9bfff640 1 -- 172.21.15.43:0/3083826334 reap_dead start 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 shutdown_connections 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 >> 172.21.15.43:0/3083826334 conn(0x7f0d9c0a7520 msgr2=0x7f0d9c09e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 shutdown_connections 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 -- 172.21.15.43:0/3083826334 wait complete. 2024-04-19T00:58:32.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.047+0000 7f0da1d2e640 1 librados: shutdown 2024-04-19T00:58:32.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:32.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:58:32.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:58:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient: get_monmap_and_config 2024-04-19T00:58:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient: build_initial_monmap 2024-04-19T00:58:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient: monmap: 2024-04-19T00:58:32.200 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:32.204020+0000 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:32.204020+0000 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:32.201 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:32.202 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:32.202 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 Processor -- start 2024-04-19T00:58:32.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 -- start start 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient: init 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient: _add_conns ranks=[3,0,5,8,7,10,4,2,6,1,9] 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43b809e4c0 0x7f43b8136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): picked mon.noname-g con 0x7f43b809e4c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43b809e8a0 addr v1:172.21.15.43:6789/0 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43b809eca0 0x7f43b80a4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): picked mon.noname-h con 0x7f43b809eca0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43b80a1e20 con 0x7f43b809e8a0 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f43b80a1c50 con 0x7f43b809e4c0 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f43b806c760 con 0x7f43b809eca0 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43bda9d640 10 monclient(hunting): authenticate will time out at 1782.331831s 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43b6ffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f43b809e8a0 0x7f43b80a3940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38028/0 (socket says 172.21.15.43:38028) 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.199+0000 7f43b6ffd640 1 -- 172.21.15.43:0/3009568919 learned_addr learned my addr 172.21.15.43:0/3009568919 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43b809eca0 0x7f43b80a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43b809e4c0 0x7f43b8136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): get_auth_request con 0x7f43b809eca0 auth_method 0 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): get_auth_request con 0x7f43b809e4c0 auth_method 0 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 651392967 0 0) 0x7f43b80a1e20 con 0x7f43b809e8a0 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): _negotiate 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): my global_id is 65133 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43a0003530 con 0x7f43b809e8a0 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1111860549 0 0) 0x7f43a0003530 con 0x7f43b809e8a0 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43b809eca0 msgr2=0x7f43b80a4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43b809eca0 0x7f43b80a4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43b809e4c0 msgr2=0x7f43b8136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43b809e4c0 0x7f43b8136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b80a5510 con 0x7f43b809e8a0 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43bda9d640 5 monclient: authenticate success, global_id 65133 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f43ac001970 con 0x7f43b809e8a0 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: dump: 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.211 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:58:32.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 >> v1:172.21.15.43:6789/0 conn(0x7f43b809e8a0 legacy=0x7f43b80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: _add_conns ranks=[4,6,2,3,8,5,1,7,0] 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004260 0x7f43a0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): picked mon.c con 0x7f43a0004260 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43a0004d50 0x7f43a00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): picked mon.d con 0x7f43a0004d50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f43a0009a60 0x7f43a000bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): picked mon.b con 0x7f43a0009a60 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f43b806c760 con 0x7f43a0009a60 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f43a000c7d0 con 0x7f43a0004260 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f43a000c980 con 0x7f43a0004d50 2024-04-19T00:58:32.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43bda9d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43a0004d50 0x7f43a00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f43a0009a60 0x7f43a000bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): get_auth_request con 0x7f43a0004d50 auth_method 0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004260 0x7f43a0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): get_auth_request con 0x7f43a0009a60 auth_method 0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): get_auth_request con 0x7f43a0004260 auth_method 0 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_done global_id 65133 payload 293 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 -- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004260 msgr2=0x7f43a0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004260 0x7f43a0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 -- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f43a0009a60 msgr2=0x7f43a000bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f43a0009a60 0x7f43a000bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 monclient: found mon.d 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 -- 172.21.15.43:0/3009568919 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a0003530 con 0x7f43a0004d50 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f43a0009a60 0x7f43a000bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b7fff640 1 -- 172.21.15.43:0/3009568919 reap_dead start 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b77fe640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004260 0x7f43a0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43a0004d50 0x7f43a00051d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b80a8fd0 tx=0x7f43ac01eff0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8b0f7a891e31ac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43ac007ac0 con 0x7f43a0004d50 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: dump: 2024-04-19T00:58:32.216 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.217 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ac0ac020 con 0x7f43a0004d50 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43bda9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43bda9d640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 1 -- 172.21.15.43:0/3009568919 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43ac005d50 con 0x7f43a0004d50 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43bda9d640 4 set_mon_vals no callback set 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.203+0000 7f43b67fc640 10 monclient: dump: 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: shutdown 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43a0004d50 msgr2=0x7f43a00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43a0004d50 0x7f43a00051d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f43b80a8fd0 tx=0x7f43ac01eff0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3009568919 shutdown_connections 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 --2- 172.21.15.43:0/3009568919 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43a0004d50 0x7f43a00051d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3009568919 >> 172.21.15.43:0/3009568919 conn(0x7f43b80a85c0 msgr2=0x7f43b80a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3009568919 shutdown_connections 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3009568919 wait complete. 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: build_initial_monmap 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: monmap: 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:32.220 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 librados: starting msgr at 2024-04-19T00:58:32.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 librados: starting objecter 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 Processor -- start 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- start start 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 librados: setting wanted keys 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 librados: calling monclient init 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: init 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: _add_conns ranks=[4,1,6,7,3,2,8,5,0] 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004d50 0x7f43b8135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): picked mon.noname-c con 0x7f43a0004d50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:32.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b81c86a0 0x7f43b8135780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): picked mon.noname-f con 0x7f43b81c86a0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43b81cb170 0x7f43b81cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): picked mon.noname-d con 0x7f43b81cb170 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43b8136fe0 con 0x7f43b81c86a0 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f43b8136e60 con 0x7f43a0004d50 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f43b8137160 con 0x7f43b81cb170 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): authenticate will time out at 1782.337926s 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004d50 0x7f43b8135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004d50 0x7f43b8135240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:46292/0 (socket says 172.21.15.43:46292) 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 -- 172.21.15.43:0/3049245536 learned_addr learned my addr 172.21.15.43:0/3049245536 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:32.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): get_auth_request con 0x7f43a0004d50 auth_method 0 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43b81cb170 0x7f43b81cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): get_auth_request con 0x7f43b81cb170 auth_method 0 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b81c86a0 0x7f43b8135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b6ffd640 10 monclient(hunting): get_auth_request con 0x7f43b81c86a0 auth_method 0 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient(hunting): handle_auth_done global_id 134158 payload 824 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43b81cb170 msgr2=0x7f43b81cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f43b81cb170 0x7f43b81cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b81c86a0 msgr2=0x7f43b8135780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b81c86a0 0x7f43b8135780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 monclient: found mon.noname-c 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b8120910 con 0x7f43a0004d50 2024-04-19T00:58:32.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b77fe640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004d50 0x7f43b8135240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43a800f440 tx=0x7f43a800f900 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 client.?.objecter ms_handle_connect 0x7f43a0004d50 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43a8009c10 con 0x7f43a0004d50 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: dump: 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.232 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a80158a0 con 0x7f43a0004d50 2024-04-19T00:58:32.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43a801c3b0 con 0x7f43a0004d50 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 10 monclient: dump: 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.234 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 5 monclient: authenticate success, global_id 134158 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: _renew_subs 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43b57fa640 4 set_mon_vals no callback set 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b8122c00 con 0x7f43a0004d50 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 client.134158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: _renew_subs 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:58:32.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b8121b70 con 0x7f43a0004d50 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 librados: init done 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 0x7f43b81cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f43b806bda0 con 0x7f43b802d5d0 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.207+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f43a8015430 con 0x7f43a0004d50 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 0x7f43b81cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient: get_auth_request con 0x7f43b802d5d0 auth_method 0 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43a4038e10 0x7f43a403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(530..530 src has 1..530) ==== 6625+0+0 (secure 0 0 0) 0x7f43a804a040 con 0x7f43a0004d50 2024-04-19T00:58:32.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 client.134158.objecter ms_dispatch 0x7f43b8000ba0 osd_map(530..530 src has 1..530) 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 3 client.134158.objecter handle_osd_map got epochs [530,530] > 0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 3 client.134158.objecter handle_osd_map decoding full epoch 530 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 20 client.134158.objecter dump_active .. 0 homeless 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b7fff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43a4038e10 0x7f43a403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b7fff640 10 monclient: get_auth_request con 0x7f43a4038e10 auth_method 0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_done global_id 74446 payload 824 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43b806b9e0 con 0x7f43b802d5d0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 client.134158.objecter ms_handle_connect 0x7f43b802d5d0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 client.134158.objecter resend_mon_ops 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 0x7f43b81cbb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43a0004790 tx=0x7f43a0005840 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f43b7fff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43a4038e10 0x7f43a403b2d0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f43b80a54e0 tx=0x7f43b0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 client.134158.objecter ms_handle_connect 0x7f43a4038e10 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43ac0bf070 con 0x7f43b802d5d0 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 monclient: dump: 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.238 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f43ac0ac040 con 0x7f43b802d5d0 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 msgr2=0x7f43b81cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 0x7f43b81cbb30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43a0004790 tx=0x7f43a0005840 comp rx=0 tx=0).stop 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 msgr2=0x7f43b81cbb30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:32.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.211+0000 7f439ffff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b802d5d0 0x7f43b81cbb30 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f43a0004790 tx=0x7f43a0005840 comp rx=0 tx=0).stop 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43bda9d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43bda9d640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 0x7f43b8063010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43bda9d640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f43b806d140 con 0x7f43b8062c00 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 0x7f43b8063010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient: get_auth_request con 0x7f43b8062c00 auth_method 0 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 10 monclient(hunting): handle_auth_done global_id 74455 payload 824 2024-04-19T00:58:32.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 1 -- 172.21.15.43:0/3049245536 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43b802d9b0 con 0x7f43b8062c00 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f43b6ffd640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 0x7f43b8063010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f43ac01f030 tx=0x7f43ac0aa070 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f439ffff640 10 client.134158.objecter ms_handle_connect 0x7f43b8062c00 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f439ffff640 10 client.134158.objecter resend_mon_ops 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43ac0bf040 con 0x7f43b8062c00 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f439ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f439ffff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.219+0000 7f439ffff640 10 monclient: dump: 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.241 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f43ac0c5070 con 0x7f43b8062c00 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 msgr2=0x7f43b8063010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 0x7f43b8063010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f43ac01f030 tx=0x7f43ac0aa070 comp rx=0 tx=0).stop 2024-04-19T00:58:32.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 msgr2=0x7f43b8063010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f439ffff640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f43b8062c00 0x7f43b8063010 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f43ac01f030 tx=0x7f43ac0aa070 comp rx=0 tx=0).stop 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 291839, 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:32.243 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:32.244 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:32.245 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:32.246 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:32.247 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:32.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.251 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:32.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:32.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:32.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:32.257 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:32.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:32.262 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:32.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:32.264 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 10 librados: watch_flush enter 2024-04-19T00:58:32.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 10 librados: watch_flush exit 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 20 client.134158.objecter shutdown clearing up homeless session... 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 10 client.134158.objecter successfully canceled tick 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43a4038e10 msgr2=0x7f43a403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43a4038e10 0x7f43a403b2d0 secure :-1 s=READY pgs=3565 cs=0 l=1 rev1=1 crypto rx=0x7f43b80a54e0 tx=0x7f43b0009190 comp rx=0 tx=0).stop 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 10 monclient: shutdown 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004d50 msgr2=0x7f43b8135240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 --2- 172.21.15.43:0/3049245536 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f43a0004d50 0x7f43b8135240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43a800f440 tx=0x7f43a800f900 comp rx=0 tx=0).stop 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43b7fff640 1 -- 172.21.15.43:0/3049245536 reap_dead start 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43b7fff640 1 -- 172.21.15.43:0/3049245536 reap_dead start 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 shutdown_connections 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 >> 172.21.15.43:0/3049245536 conn(0x7f43b80a85c0 msgr2=0x7f43b80a6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:32.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 shutdown_connections 2024-04-19T00:58:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 -- 172.21.15.43:0/3049245536 wait complete. 2024-04-19T00:58:32.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.223+0000 7f43bda9d640 1 librados: shutdown 2024-04-19T00:58:32.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:32.267 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T00:58:32.267 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient: get_monmap_and_config 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient: build_initial_monmap 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient: monmap: 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:32.403620+0000 2024-04-19T00:58:32.400 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:32.403620+0000 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:32.401 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 1 Processor -- start 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 1 -- start start 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient: init 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient: _add_conns ranks=[1,0,3,5,8,7,10,2,9,4,6] 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb2c0138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2c00a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2c01419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient(hunting): picked mon.noname-g con 0x7fb2c01397c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c00a3dc0 con 0x7fb2c00a5b80 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2c00a40a0 con 0x7fb2c0138cd0 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c00a4410 con 0x7fb2c01397c0 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 10 monclient(hunting): authenticate will time out at 1782.530684s 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bffff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb2c00a5b80 0x7fb2c013e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38044/0 (socket says 172.21.15.43:38044) 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bffff640 1 -- 172.21.15.43:0/1856303371 learned_addr learned my addr 172.21.15.43:0/1856303371 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2c01419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): get_auth_request con 0x7fb2c01397c0 auth_method 0 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043132924 0 0) 0x7fb2c00a3dc0 con 0x7fb2c00a5b80 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): _negotiate 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): my global_id is 65142 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2a8003530 con 0x7fb2c00a5b80 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 332614158 0 0) 0x7fb2a8003530 con 0x7fb2c00a5b80 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 msgr2=0x7fb2c01419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2c01419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 >> v1:172.21.15.161:6789/0 conn(0x7fb2c0138cd0 legacy=0x7fb2c01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 monclient: found mon.noname-a-legacy 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c00a2ae0 con 0x7fb2c00a5b80 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fb2b0001990 con 0x7fb2c00a5b80 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 5 monclient: authenticate success, global_id 65142 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: dump: 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 >> v1:172.21.15.43:6789/0 conn(0x7fb2c00a5b80 legacy=0x7fb2c013e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient: _add_conns ranks=[3,5,6,8,4,2,0,7,1] 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2a8004240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): picked mon.g con 0x7fb2c01397c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb2a8004860 0x7fb2a8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): picked mon.h con 0x7fb2a8004860 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2a80099c0 0x7fb2a800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): picked mon.d con 0x7fb2a80099c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c00a4410 con 0x7fb2c01397c0 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb2a800c480 con 0x7fb2a8004860 2024-04-19T00:58:32.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb2a800c630 con 0x7fb2a80099c0 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2bf7fe640 10 monclient(hunting): _finish_auth 0 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c6c81640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.399+0000 7fb2c521e640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2a80099c0 0x7fb2a800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): get_auth_request con 0x7fb2a80099c0 auth_method 0 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c4a1d640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2a8004240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:58:32.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c4a1d640 10 monclient(hunting): get_auth_request con 0x7fb2c01397c0 auth_method 0 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c4a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c4a1d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c4a1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bffff640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb2a8004860 0x7fb2a8004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bffff640 10 monclient(hunting): get_auth_request con 0x7fb2a8004860 auth_method 0 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient(hunting): handle_auth_done global_id 65142 payload 293 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 -- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb2a8004860 msgr2=0x7fb2a8004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb2a8004860 0x7fb2a8004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 -- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 msgr2=0x7fb2a8004240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:32.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2a8004240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 monclient: found mon.d 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 -- 172.21.15.43:0/1856303371 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a8003530 con 0x7fb2a80099c0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c4a1d640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c01397c0 0x7fb2a8004240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2a80099c0 0x7fb2a800bdb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800e170 tx=0x7fb2b801cc40 comp rx=0 tx=0).ready entity=mon.6 client_cookie=777e1c9313153e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb2b80b2070 con 0x7fb2a80099c0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: dump: 2024-04-19T00:58:32.419 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.420 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b80a5d80 con 0x7fb2a80099c0 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 4 set_mon_vals no callback set 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 1 -- 172.21.15.43:0/1856303371 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb2b8009c10 con 0x7fb2a80099c0 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2bf7fe640 10 monclient: dump: 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.421 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 10 monclient: shutdown 2024-04-19T00:58:32.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 1 -- 172.21.15.43:0/1856303371 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2a80099c0 msgr2=0x7fb2a800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/1856303371 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2a80099c0 0x7fb2a800bdb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800e170 tx=0x7fb2b801cc40 comp rx=0 tx=0).stop 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.403+0000 7fb2c521e640 1 -- 172.21.15.43:0/1856303371 reap_dead start 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/1856303371 shutdown_connections 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/1856303371 >> 172.21.15.43:0/1856303371 conn(0x7fb2c00ab2f0 msgr2=0x7fb2c00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/1856303371 shutdown_connections 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/1856303371 wait complete. 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: build_initial_monmap 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: monmap: 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:32.423 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 librados: starting msgr at 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 librados: starting objecter 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 Processor -- start 2024-04-19T00:58:32.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- start start 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 librados: setting wanted keys 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 librados: calling monclient init 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: init 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _add_conns ranks=[6,2,3,5,0,7,1,8,4] 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 0x7fb2c0137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): picked mon.noname-d con 0x7fb2c01d0750 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb2c01d0b30 0x7fb2c009ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): picked mon.noname-b con 0x7fb2c01d0b30 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c0137580 0x7fb2c009f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): picked mon.noname-g con 0x7fb2c0137580 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c01423c0 con 0x7fb2c01d0b30 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb2c00a20a0 con 0x7fb2c0137580 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb2c0142540 con 0x7fb2c01d0750 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient(hunting): authenticate will time out at 1782.537904s 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 0x7fb2c0137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 0x7fb2c0137040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42762/0 (socket says 172.21.15.43:42762) 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 -- 172.21.15.43:0/863506326 learned_addr learned my addr 172.21.15.43:0/863506326 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): get_auth_request con 0x7fb2c01d0750 auth_method 0 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb2c01d0b30 0x7fb2c009ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): get_auth_request con 0x7fb2c01d0b30 auth_method 0 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c521e640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c0137580 0x7fb2c009f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c521e640 10 monclient(hunting): get_auth_request con 0x7fb2c0137580 auth_method 0 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c521e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:32.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c521e640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c521e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient(hunting): handle_auth_done global_id 74634 payload 824 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 -- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c0137580 msgr2=0x7fb2c009f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c0137580 0x7fb2c009f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 -- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb2c01d0b30 msgr2=0x7fb2c009ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb2c01d0b30 0x7fb2c009ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient: _finish_hunting 0 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 monclient: found mon.noname-d 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 -- 172.21.15.43:0/863506326 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c00a09f0 con 0x7fb2c01d0750 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bffff640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb2c01d0b30 0x7fb2c009ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:32.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c4a1d640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 0x7fb2c0137040 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e7e0 tx=0x7fb2b400eca0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 client.?.objecter ms_handle_connect 0x7fb2c01d0750 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb2b400c6c0 con 0x7fb2c01d0750 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: dump: 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.429 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: _finish_auth 0 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: _check_auth_tickets 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b400c880 con 0x7fb2c01d0750 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 5 monclient: authenticate success, global_id 74634 2024-04-19T00:58:32.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _renew_subs 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c00a3040 con 0x7fb2c01d0750 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb2b400cea0 con 0x7fb2c01d0750 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2bd7fa640 10 monclient: dump: 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:32.431 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 client.74634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _renew_subs 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c0122140 con 0x7fb2c01d0750 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2be7fc640 4 set_mon_vals no callback set 2024-04-19T00:58:32.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 librados: init done 2024-04-19T00:58:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.407+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c00693b0 con 0x7fb2c01d0750 2024-04-19T00:58:32.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb2b401a030 con 0x7fb2c01d0750 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb2ac038e10 0x7fb2ac03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(531..531 src has 1..531) ==== 6617+0+0 (secure 0 0 0) 0x7fb2b4018020 con 0x7fb2c01d0750 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 10 client.74634.objecter ms_dispatch 0x7fb2c0000ba0 osd_map(531..531 src has 1..531) 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 3 client.74634.objecter handle_osd_map got epochs [531,531] > 0 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 3 client.74634.objecter handle_osd_map decoding full epoch 531 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 20 client.74634.objecter dump_active .. 0 homeless 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bffff640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb2ac038e10 0x7fb2ac03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:32.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bffff640 10 monclient: get_auth_request con 0x7fb2ac038e10 auth_method 0 2024-04-19T00:58:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bd7fa640 10 client.74634.objecter ms_handle_connect 0x7fb2ac038e10 2024-04-19T00:58:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.411+0000 7fb2bffff640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb2ac038e10 0x7fb2ac03b2d0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00a0480 tx=0x7fb2b0007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.415+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb2b4021060 con 0x7fb2c01d0750 2024-04-19T00:58:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.415+0000 7fb2bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:32.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.415+0000 7fb2bd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:32.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.639+0000 7fb2c6c81640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T00:58:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.639+0000 7fb2c6c81640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T00:58:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.639+0000 7fb2c6c81640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:32.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.639+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb2c00a6f40 con 0x7fb2c01d0750 2024-04-19T00:58:32.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.643+0000 7fb2bd7fa640 1 -- 172.21.15.43:0/863506326 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb2b40135e0 con 0x7fb2c01d0750 2024-04-19T00:58:32.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.643+0000 7fb2bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T00:58:32.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.643+0000 7fb2bd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:32.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=22,config=2,health=12,kv=11,logm=53} crc {auth=1657984828,config=2586702516,health=3020086537,kv=4171600807,logm=3922461586}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=10,kv=12,logm=54} crc {auth=1657984828,config=2586702516,health=3972666825,kv=4196696685,logm=4162321983}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=503613565,config=2586702516,health=3020086537,kv=4196696685,logm=1477643139}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=1660182692,config=2586702516,health=1207457705,kv=1646202747,logm=3773053362}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=22,config=2,health=12,kv=12,logm=52} crc {auth=1657984828,config=2586702516,health=3020086537,kv=4196696685,logm=3159747558}) 2024-04-19T00:58:32.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=1876532013,config=2586702516,health=3020086537,kv=4196696685,logm=3744039790}) 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=418551765,config=2586702516,health=3020086537,kv=4196696685,logm=2825941719}) 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=22,config=1,health=10,kv=10,logm=57} crc {auth=1657984828,config=1747058506,health=2731312575,kv=117869517,logm=1875229361}) 2024-04-19T00:58:32.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=22,config=2,health=10,kv=12,logm=54} crc {auth=1657984828,config=2586702516,health=1020772047,kv=4196696685,logm=1727036431}) 2024-04-19T00:58:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 10 librados: watch_flush enter 2024-04-19T00:58:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 10 librados: watch_flush exit 2024-04-19T00:58:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 20 client.74634.objecter shutdown clearing up homeless session... 2024-04-19T00:58:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 10 client.74634.objecter successfully canceled tick 2024-04-19T00:58:32.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb2ac038e10 msgr2=0x7fb2ac03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb2ac038e10 0x7fb2ac03b2d0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00a0480 tx=0x7fb2b0007040 comp rx=0 tx=0).stop 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 10 monclient: shutdown 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 msgr2=0x7fb2c0137040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 0x7fb2c0137040 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400e7e0 tx=0x7fb2b400eca0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 shutdown_connections 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb2ac038e10 0x7fb2ac03b2d0 secure :-1 s=CLOSED pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7fb2c00a0480 tx=0x7fb2b0007040 comp rx=0 tx=0).stop 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb2c0137580 0x7fb2c009f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb2c01d0b30 0x7fb2c009ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 --2- 172.21.15.43:0/863506326 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb2c01d0750 0x7fb2c0137040 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 >> 172.21.15.43:0/863506326 conn(0x7fb2c00ab2f0 msgr2=0x7fb2c00a9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 shutdown_connections 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 -- 172.21.15.43:0/863506326 wait complete. 2024-04-19T00:58:32.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:32.647+0000 7fb2c6c81640 1 librados: shutdown 2024-04-19T00:58:32.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545226938}) 2024-04-19T00:58:32.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=916536741}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082696850}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1887386893}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=861580313}) 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=265247840}) 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2867422929}) 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546272119}) 2024-04-19T00:58:32.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=179779036}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3026088577}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1267112457}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=60674594}) 2024-04-19T00:58:32.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2625433017}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2359410067}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4275385697}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1681628598}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3493230608}) 2024-04-19T00:58:32.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1971806754}) 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965037869}) 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2616787054}) 2024-04-19T00:58:32.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084329995}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=289395613}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2226410116}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3689418287}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.669 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3875697345}) 2024-04-19T00:58:32.670 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.670 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2285334252}) 2024-04-19T00:58:32.670 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3298555490}) 2024-04-19T00:58:32.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T00:58:32.673 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.673 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.673 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175785880}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937760195}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082225635}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2917376868}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2121538928}) 2024-04-19T00:58:32.674 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1082843276}) 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3902588972}) 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2902863561}) 2024-04-19T00:58:32.675 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=481984730}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468147230}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1510388857}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046553143}) 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.679 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673235586}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=329775633}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=472026664}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1060978524}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241553663}) 2024-04-19T00:58:32.680 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3711264476}) 2024-04-19T00:58:32.683 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.683 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063630020}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2368849298}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635167165}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1707272954}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=802843281}) 2024-04-19T00:58:32.684 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2199256874}) 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2589523146}) 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739390563}) 2024-04-19T00:58:32.685 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4140067198}) 2024-04-19T00:58:32.687 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582363880}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135111690}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348322059}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=894987893}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.688 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4246143562}) 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1605272659}) 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=116612027}) 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3042482625}) 2024-04-19T00:58:32.689 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2578224657}) 2024-04-19T00:58:32.690 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3956593227}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750626256}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3015346854}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=855731285}) 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.691 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=686792538}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4258760665}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2901131877}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596454289}) 2024-04-19T00:58:32.692 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2522827831}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790926128}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3182177664}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1074714827}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.695 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2344108878}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=704670123}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1690085245}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4019720353}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2955697284}) 2024-04-19T00:58:32.696 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3232833753}) 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903511935}) 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889276118}) 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.698 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969703227}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3296638250}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2112881322}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1670433294}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=273489485}) 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.699 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=511698186}) 2024-04-19T00:58:32.700 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1664012511}) 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034654672}) 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71593441}) 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.702 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945227159}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2024870102}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3263836792}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2265540637}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.703 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=324772715}) 2024-04-19T00:58:32.704 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.704 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3032978703}) 2024-04-19T00:58:32.704 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3746585861}) 2024-04-19T00:58:32.706 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.706 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.706 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717261079}) 2024-04-19T00:58:32.706 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.706 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.706 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2416375244}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=643305827}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=609952741}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3438232225}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2163867622}) 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.707 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.708 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2621161217}) 2024-04-19T00:58:32.708 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.708 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1018861164}) 2024-04-19T00:58:32.708 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=402688137}) 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305361522}) 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943059911}) 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.710 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341454340}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2532268943}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2772163438}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=230350851}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.711 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1158926034}) 2024-04-19T00:58:32.712 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.712 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=512616031}) 2024-04-19T00:58:32.712 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3940191158}) 2024-04-19T00:58:32.713 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3274216958}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742392830}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532792266}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1031687042}) 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.714 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3698447168}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4166255768}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=584206101}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4214129720}) 2024-04-19T00:58:32.715 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1332872484}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=271329886}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=611206607}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1827252937}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4168627668}) 2024-04-19T00:58:32.717 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1756127733}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2098841313}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2858720508}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986040628}) 2024-04-19T00:58:32.718 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3826196736}) 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127556382}) 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100650641}) 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.720 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3816707884}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2209619727}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4035044691}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3113918578}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4099780449}) 2024-04-19T00:58:32.721 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.722 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121504975}) 2024-04-19T00:58:32.722 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1292972750}) 2024-04-19T00:58:32.724 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917702770}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504760696}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=902514113}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1721351156}) 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.725 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1076585871}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1173347927}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1241082228}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1153298904}) 2024-04-19T00:58:32.726 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4024503579}) 2024-04-19T00:58:32.729 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.729 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.729 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278844094}) 2024-04-19T00:58:32.729 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.729 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.729 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2174378482}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448911666}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3095055164}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=537739133}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3943678355}) 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.730 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.731 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2570408629}) 2024-04-19T00:58:32.731 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.731 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2566744618}) 2024-04-19T00:58:32.731 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4132545521}) 2024-04-19T00:58:32.733 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.733 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.733 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119248629}) 2024-04-19T00:58:32.733 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786151339}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407442528}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2085342468}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1104317357}) 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.734 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4029266623}) 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4072385846}) 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=874550817}) 2024-04-19T00:58:32.735 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3044959219}) 2024-04-19T00:58:32.737 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=102423796}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205238002}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536778944}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1592356177}) 2024-04-19T00:58:32.738 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3433999793}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2939815662}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3678287147}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2596538258}) 2024-04-19T00:58:32.739 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1333747464}) 2024-04-19T00:58:32.741 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346098380}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2280726448}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1828472731}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2289822601}) 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.742 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1988295375}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3440036905}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3662544915}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198217552}) 2024-04-19T00:58:32.743 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1783356072}) 2024-04-19T00:58:32.745 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.745 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.745 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977781575}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1056197778}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017991682}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1492499929}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3916921178}) 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.746 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3298494734}) 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2107164578}) 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1487219934}) 2024-04-19T00:58:32.747 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3393428208}) 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3997388079}) 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=165681156}) 2024-04-19T00:58:32.749 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396632632}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1962822649}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=58349757}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=330601824}) 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.750 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.751 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1815505431}) 2024-04-19T00:58:32.751 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.751 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1003642878}) 2024-04-19T00:58:32.751 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3375414104}) 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671428976}) 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2900801995}) 2024-04-19T00:58:32.753 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1759796473}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2360991423}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2732299382}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3199775140}) 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.754 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.755 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1669152490}) 2024-04-19T00:58:32.755 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.755 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1385003753}) 2024-04-19T00:58:32.755 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1594316817}) 2024-04-19T00:58:32.757 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.757 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.757 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267550799}) 2024-04-19T00:58:32.757 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.757 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3017324117}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=407152954}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4173184731}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4269687961}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3844498795}) 2024-04-19T00:58:32.758 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.759 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.759 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3845359346}) 2024-04-19T00:58:32.759 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.759 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1027324301}) 2024-04-19T00:58:32.759 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=155996601}) 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808621600}) 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055339364}) 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.761 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495356983}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1265439366}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=646416555}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3390502386}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1602688704}) 2024-04-19T00:58:32.762 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.763 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2088309688}) 2024-04-19T00:58:32.763 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1622728188}) 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347368953}) 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2243932064}) 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.764 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1992379255}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4105074515}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4066446889}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1293570536}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1694391832}) 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.765 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2994282987}) 2024-04-19T00:58:32.766 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1206657309}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231608563}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983104551}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914107202}) 2024-04-19T00:58:32.768 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=131069056}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2017268875}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1749754184}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1259272035}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=121500964}) 2024-04-19T00:58:32.769 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4132476963}) 2024-04-19T00:58:32.771 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.771 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.771 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087951684}) 2024-04-19T00:58:32.771 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.771 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210629115}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1868804541}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1289877947}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=242429237}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.772+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2512008178}) 2024-04-19T00:58:32.772 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.773 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.773 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1008312295}) 2024-04-19T00:58:32.773 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.773 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3681849564}) 2024-04-19T00:58:32.773 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2523471047}) 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695181049}) 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509290682}) 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.775 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710928110}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3000481811}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3732340295}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4268206119}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.776+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2722103970}) 2024-04-19T00:58:32.776 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.777 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2458265032}) 2024-04-19T00:58:32.777 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2138121626}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249757308}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691337440}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2125252548}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.779 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2146246550}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=807526631}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=242946894}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=584956998}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=383064851}) 2024-04-19T00:58:32.780 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1919773851}) 2024-04-19T00:58:32.782 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.782 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.782 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223223894}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306578995}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=850730555}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=174570757}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3454126216}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.783 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=582900249}) 2024-04-19T00:58:32.784 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.784 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.784 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3656635532}) 2024-04-19T00:58:32.784 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.784 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1516854311}) 2024-04-19T00:58:32.784 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.784+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3245166480}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3481600203}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2547044014}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638099161}) 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.786 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=261244327}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=646230408}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3873727719}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=298262738}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2775847729}) 2024-04-19T00:58:32.787 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.788+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2513915868}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501634547}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922716523}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2114830123}) 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.790 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1878861903}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2824183000}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1027452512}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4229948807}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3036454700}) 2024-04-19T00:58:32.791 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.792+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=235575772}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2421888866}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757460344}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147123991}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.794 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2296673312}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1702398430}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2611982029}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=146953220}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=376889041}) 2024-04-19T00:58:32.795 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.796+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=108555070}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673101752}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3759902734}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3639040849}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.798 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1631566851}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3242563938}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3449331126}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1316387880}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3502517343}) 2024-04-19T00:58:32.799 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.800+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1473068654}) 2024-04-19T00:58:32.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3273734800}) 2024-04-19T00:58:32.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225598000}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871344270}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2084998256}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3804322257}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3154136428}) 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2614887321}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3920386480}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1641238516}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224119217}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11040815}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=84983550}) 2024-04-19T00:58:32.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1838472529}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.804+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2528969345}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1006173746}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=114834102}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758795519}) 2024-04-19T00:58:32.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1974985193}) 2024-04-19T00:58:32.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.808+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786949215}) 2024-04-19T00:58:32.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986420301}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151482944}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3610418368}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1340825035}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3192421637}) 2024-04-19T00:58:32.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1397256049}) 2024-04-19T00:58:32.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3151422830}) 2024-04-19T00:58:32.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.812+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1159138258}) 2024-04-19T00:58:32.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138547632}) 2024-04-19T00:58:32.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161824585}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316357494}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2889255450}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=122417482}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3926362101}) 2024-04-19T00:58:32.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2984966716}) 2024-04-19T00:58:32.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=887309374}) 2024-04-19T00:58:32.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.816+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2223374587}) 2024-04-19T00:58:32.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310482123}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3804435640}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305505305}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2973856561}) 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2725510138}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2202079733}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2166831161}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821319293}) 2024-04-19T00:58:32.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.828+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=394660208}) 2024-04-19T00:58:32.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226723241}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1976437169}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1011404920}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3142574293}) 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2513187104}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=839756237}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2533174397}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1200518862}) 2024-04-19T00:58:32.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.836+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1139949097}) 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252019775}) 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409695919}) 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4143512353}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3833292614}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2868558722}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1400457508}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2077202848}) 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2732318859}) 2024-04-19T00:58:32.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.840+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=690920875}) 2024-04-19T00:58:32.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861726528}) 2024-04-19T00:58:32.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971709219}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421084815}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1186127343}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1549118656}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2073041038}) 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=748670078}) 2024-04-19T00:58:32.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4132538722}) 2024-04-19T00:58:32.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.844+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2730015069}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=286228541}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2603513558}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864748437}) 2024-04-19T00:58:32.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=924219587}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4064000236}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1342273200}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1776181734}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3648347171}) 2024-04-19T00:58:32.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.848+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3307783410}) 2024-04-19T00:58:32.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220199314}) 2024-04-19T00:58:32.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939433271}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556685492}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3632099693}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=229291631}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1170216228}) 2024-04-19T00:58:32.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1120688736}) 2024-04-19T00:58:32.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=167928251}) 2024-04-19T00:58:32.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.852+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=63947108}) 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397492848}) 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1924578583}) 2024-04-19T00:58:32.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3203290360}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=420893366}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2425677248}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1751349858}) 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=661918305}) 2024-04-19T00:58:32.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3398080288}) 2024-04-19T00:58:32.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.856+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4164366711}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1678512684}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2598848057}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94833632}) 2024-04-19T00:58:32.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=477873588}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2569858907}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=214923113}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4137208278}) 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3286668540}) 2024-04-19T00:58:32.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.860+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=163504080}) 2024-04-19T00:58:32.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466479429}) 2024-04-19T00:58:32.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802300092}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703363911}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3630171461}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4032714147}) 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2954320334}) 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3348728723}) 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2709309090}) 2024-04-19T00:58:32.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4108353412}) 2024-04-19T00:58:32.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864462876}) 2024-04-19T00:58:32.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3528727197}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16136186}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=905042878}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1068587836}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1433770794}) 2024-04-19T00:58:32.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2926415912}) 2024-04-19T00:58:32.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=134579114}) 2024-04-19T00:58:32.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.864+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3099902340}) 2024-04-19T00:58:32.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1333173244}) 2024-04-19T00:58:32.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893661889}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763146081}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2958536618}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=485366723}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1213897728}) 2024-04-19T00:58:32.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2439194974}) 2024-04-19T00:58:32.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=300802601}) 2024-04-19T00:58:32.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.868+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4170828416}) 2024-04-19T00:58:32.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246957932}) 2024-04-19T00:58:32.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787825199}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420930883}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1917941351}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=54874593}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=184005318}) 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3592397001}) 2024-04-19T00:58:32.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3216947262}) 2024-04-19T00:58:32.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.872+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=615758748}) 2024-04-19T00:58:32.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403672716}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857618105}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3838094656}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2662588557}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2560301650}) 2024-04-19T00:58:32.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=805978437}) 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2358067638}) 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3690246822}) 2024-04-19T00:58:32.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.876+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3619674214}) 2024-04-19T00:58:32.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2853512374}) 2024-04-19T00:58:32.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3801103537}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043398717}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3552275689}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1676708873}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2613391392}) 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2031153839}) 2024-04-19T00:58:32.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1080057903}) 2024-04-19T00:58:32.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.880+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2563239123}) 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.916+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.916+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.916+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504434190}) 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1263321136}) 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.916 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139704089}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1170457595}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2385385539}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3015017509}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1835140585}) 2024-04-19T00:58:32.917 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.918 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3569489090}) 2024-04-19T00:58:32.918 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.920+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=33864183}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744364061}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3455329942}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513733357}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=72719313}) 2024-04-19T00:58:32.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=347033956}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3890184359}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=487605618}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=679687729}) 2024-04-19T00:58:32.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1211934621}) 2024-04-19T00:58:32.935 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.935 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434382693}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2474250801}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1152627038}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3540331533}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.924+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3260405217}) 2024-04-19T00:58:32.936 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2223817961}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2813570310}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1243433002}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1137729231}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3139930922}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.937 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2606854733}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1730682428}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2593531429}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=593661862}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2804314613}) 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.938 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2680676280}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1279362993}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.928+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=571810283}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62970824}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208821003}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093283729}) 2024-04-19T00:58:32.939 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2890006663}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3970559766}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1587739026}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3912425697}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1040527515}) 2024-04-19T00:58:32.940 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3917746596}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750993406}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134810459}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008154455}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2086704781}) 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.941 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1665627626}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=808373322}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3541783861}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4210799356}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.932+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=208290189}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687736582}) 2024-04-19T00:58:32.942 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1491718789}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3518980551}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1525552945}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=499306983}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2312917179}) 2024-04-19T00:58:32.943 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1197806916}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2768272649}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1314028725}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398390494}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1084482595}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.944 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1117761128}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4285612534}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3310941179}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.936+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3677345144}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=995039738}) 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.945 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1896553990}) 2024-04-19T00:58:32.946 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3424353504}) 2024-04-19T00:58:32.974 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.974 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1397779045}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555412686}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265748411}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4148156800}) 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.975 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=987782695}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4217767984}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3687539225}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=782999222}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.940+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=617404142}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270769856}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.976 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3744047065}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711980658}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3613257617}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2522635885}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1353121222}) 2024-04-19T00:58:32.977 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2481419044}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434104391}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2751852611}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1097459701}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034073796}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.978 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1819745624}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2047751909}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.944+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1880926111}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1513096859}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1631931223}) 2024-04-19T00:58:32.979 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2917246925}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4008871662}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975689769}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259557388}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3187998328}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3530954098}) 2024-04-19T00:58:32.980 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3603087007}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=117524917}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3943052348}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1262684262}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.948+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3757963250}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.981 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1518476778}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194304382}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4150415575}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2943242120}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4069963192}) 2024-04-19T00:58:32.982 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4160162776}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3172068457}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=921451908}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1784388643}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2364497797}) 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.983 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.952+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700354842}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87605941}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2981089181}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1765866333}) 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.984 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3919058607}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4224671632}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3214507482}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=924340268}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346848957}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.985 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947303489}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2447559747}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1992050898}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=153825640}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=539728375}) 2024-04-19T00:58:32.986 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1484897297}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=298763947}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.956+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2240123147}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052961076}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=902925253}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4023126682}) 2024-04-19T00:58:32.987 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=346771732}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1185037816}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1465769643}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4046809422}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2097894082}) 2024-04-19T00:58:32.988 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.960+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1106963312}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456056400}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1860018297}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999760866}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=32494969}) 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.989 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2646991545}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3488427013}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2610134041}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3342439819}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.964+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3843220041}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.990 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764563813}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1610449156}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650492634}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4250850505}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1552884435}) 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.991 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=463041902}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2262038085}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1173995298}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2230043530}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409493920}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2788297022}) 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.992 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=879836959}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2709887893}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3214576050}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3663508727}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=990247354}) 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.993 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2919618286}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.968+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1027044289}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534997163}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3550992219}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798991453}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3985241795}) 2024-04-19T00:58:32.994 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=358859783}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=575383696}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3322782487}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831381292}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1656478918}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.995 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2310207685}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235561353}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1621442050}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=28172462}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1952468743}) 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.996 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2350262824}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1883482201}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2990467176}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.972+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3072464135}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622816629}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.997 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938052977}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2558069661}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3028763868}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2158119602}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=296583572}) 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.998 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.999 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3564437640}) 2024-04-19T00:58:32.999 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:32.999 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=192268584}) 2024-04-19T00:58:32.999 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.976+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2403704079}) 2024-04-19T00:58:33.113 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981208723}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3450464507}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=89573627}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=809109972}) 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.114 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=237794239}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3900556147}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2871035629}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2755839307}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.980+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=432516417}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205911587}) 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.115 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1222447491}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007647080}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1370102896}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1947556711}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=856502682}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.116 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=49671480}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4115439559}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=228755950}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604257020}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101866857}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426837715}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.117 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2292792983}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3496241805}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1330315007}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=943270966}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3412848709}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.984+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3707062768}) 2024-04-19T00:58:33.118 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951711302}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631980221}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=758399480}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1606567915}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.119 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2053189165}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4137261458}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3968340427}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3873487766}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.988+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1257624137}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2123301630}) 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.120 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2173602590}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845195965}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3702196600}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1449143058}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.121 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=135075115}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1329743869}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1052611133}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1415294731}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685632775}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397709681}) 2024-04-19T00:58:33.122 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112797983}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1171122265}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=638497666}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3460084640}) 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.123 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3415867653}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1666010348}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.992+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4215793606}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502122202}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4023055567}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=990893780}) 2024-04-19T00:58:33.124 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=193588146}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=815822326}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=11401808}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=942171399}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1489081567}) 2024-04-19T00:58:33.125 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:32.996+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=235258633}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332575250}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3277749932}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2607133282}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1091510922}) 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.126 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3844943695}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=752592933}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=868388560}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2337657698}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2392092831}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360718769}) 2024-04-19T00:58:33.127 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2320117569}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506425408}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2328762959}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=592480175}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.128 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3361258367}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4021607250}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1467871039}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.000+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=175734436}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764502329}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023760161}) 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.129 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769879036}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4123314953}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=661309699}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1182545272}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2946784832}) 2024-04-19T00:58:33.130 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3580692769}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2010431025}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512286268}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344883899}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793900253}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.131 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=636776497}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3336775072}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=36239912}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1643526939}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=447524080}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.004+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3531152359}) 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.132 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3976699750}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3471220296}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4095214214}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1312686476}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.133 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2064196665}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3722060969}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2329821913}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1995782201}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1045080076}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781508767}) 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.134 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110663906}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=273297232}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3624305232}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1890175150}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3023020137}) 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.135 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2612500441}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2132109846}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.008+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3057520764}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333596893}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3179347059}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581995656}) 2024-04-19T00:58:33.136 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2313841355}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1308318316}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3579661274}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3419237357}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1606652095}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2842496201}) 2024-04-19T00:58:33.137 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2126181177}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3095769153}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.012+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90826321}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1689467981}) 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.138 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2607582743}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=99685147}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1860074636}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941046396}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1733462926}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986289632}) 2024-04-19T00:58:33.139 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=225506318}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098150120}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1812350703}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3382941790}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1020347043}) 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.140 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2330677939}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4014927461}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.016+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3535671213}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215718399}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1383175286}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890497708}) 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.141 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=463085456}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1831631296}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2488216280}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3517745428}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=620471706}) 2024-04-19T00:58:33.142 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.020+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=579229145}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=926279843}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346219076}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=986711704}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3766533582}) 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.143 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1984205238}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1526963682}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=941336000}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1362207273}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.024+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=871025134}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1667566769}) 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.144 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386382587}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81291899}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=619478340}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1041884414}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2814800895}) 2024-04-19T00:58:33.145 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=712086037}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1565121677}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2352423619}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487404183}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880837112}) 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.146 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456645600}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2064309025}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=495300955}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.028+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=695639762}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.147 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=519663241}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112908452}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3240288823}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861448603}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243374571}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032055780}) 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.148 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2893248513}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2588678859}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3293538710}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3291442246}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=338349034}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1253388955}) 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.149 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181289886}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78579155}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=871862829}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.032+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1722678495}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=64728716}) 2024-04-19T00:58:33.150 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3318450911}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2633390880}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=136069985}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4182950022}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605318498}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.151 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311059903}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729680462}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3241601810}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=62152426}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1580286826}) 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.152 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=79626893}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1520792497}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3518198468}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055561870}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167603399}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3761956090}) 2024-04-19T00:58:33.153 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2866775566}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.036+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=450668748}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=624984789}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2926338375}) 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.154 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2764013007}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2710471066}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=744365718}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1269339655}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1687755393}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=640511084}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.155 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2204089842}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2406239766}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2789063405}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1978186797}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=852290610}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358474481}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.156 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=144973251}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032740445}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1722130578}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4263366620}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.040+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1740628947}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1842683943}) 2024-04-19T00:58:33.157 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2860914151}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=730857196}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526247029}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1227357448}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1971222315}) 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.158 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2660058495}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1400739095}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1626283647}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4129118073}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2983365212}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1710592854}) 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.159 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384711863}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4069782024}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3819833988}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1957161990}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4051505668}) 2024-04-19T00:58:33.160 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1076855958}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=13531740}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2133505405}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.044+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1899650044}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516011909}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2681101950}) 2024-04-19T00:58:33.161 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213222841}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2646959383}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1928969428}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1438550921}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.162 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3300542050}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=919059340}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=826868006}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740824642}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860491434}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679865285}) 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.163 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3291290819}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=318574754}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=98332374}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2819955412}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171540860}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.048+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2132500371}) 2024-04-19T00:58:33.164 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1554048443}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2983735698}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1829402993}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=229303252}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.165 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3270823533}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=300344807}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3584611958}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4182705817}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=536715610}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340955437}) 2024-04-19T00:58:33.166 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076038607}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808430362}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3578005113}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1811977484}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.167 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2721078948}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=888831746}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092119587}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.052+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2020276453}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683281648}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=843821253}) 2024-04-19T00:58:33.168 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3284302234}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1294916115}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1161290983}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2114757676}) 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.169 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=752377982}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=375855403}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1343235852}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913523680}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800046151}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264818637}) 2024-04-19T00:58:33.170 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1591373564}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2378013116}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2095086660}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.056+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2040531815}) 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.171 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3473975829}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2142516387}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735336838}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=555491036}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2829314288}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.172 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2990492907}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3883024894}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3818645934}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=151088517}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2151426737}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1877174710}) 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.173 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.060+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819292528}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=43321256}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318695944}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=86258238}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1079790267}) 2024-04-19T00:58:33.174 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=49995247}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=265505069}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3853387011}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3340993223}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737242305}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.175 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3164161779}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3056802889}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3831335382}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1669230419}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.176 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=761707069}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2271403831}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2380315816}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.064+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=162054079}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467645690}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96875806}) 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.177 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=676214306}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4141814439}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=996170653}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3158140124}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418247767}) 2024-04-19T00:58:33.178 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3749548184}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1789198568}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030116031}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578869982}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4154812981}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2518438220}) 2024-04-19T00:58:33.179 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4280541324}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2698857486}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=656880158}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=840639013}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.068+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3830185408}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056431123}) 2024-04-19T00:58:33.180 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333908280}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2678052144}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1520678022}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2328533020}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3578649541}) 2024-04-19T00:58:33.181 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=259082438}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=410548880}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=487447507}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600040384}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727028402}) 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.182 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3674363347}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2810789314}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2334081699}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3145081670}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3517954618}) 2024-04-19T00:58:33.183 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4241283797}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.072+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3604270215}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4118982039}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2359394462}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1666690223}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4001220882}) 2024-04-19T00:58:33.184 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4087737042}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=798604452}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2874513897}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2958173058}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1409465305}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043788982}) 2024-04-19T00:58:33.185 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=426868065}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1952532654}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=885512313}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=335146427}) 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.186 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=973253360}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3130509660}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3738097169}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.076+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1498466498}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595095401}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1448778851}) 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.187 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2075253677}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1982194559}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=766635205}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3366246548}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1696480531}) 2024-04-19T00:58:33.188 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2562578667}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=569933359}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=8} crc {logm=577926972,mdsmap=2840952586,mgr=1318271769,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=2354127186,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=3124346916,osdmap=3059081192}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=13} crc {logm=1754945070,mdsmap=3140514009,mgr=1318271769,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1635703141,mon_config_key=771028450,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=3091462074,osdmap=3691066747}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=13} crc {logm=1391502909,mdsmap=2840952586,mgr=1170334752,mgr_metadata=2750671369,mgrstat=3281840969,mon_config_key=306580907,monmap=4080741109,osd_metadata=3683448940,osd_pg_creating=522254654,osd_snap=3953324641,osdmap=1541349029}) 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.189 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.080+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=11,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=12} crc {logm=577926972,mdsmap=3140514009,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2644004551,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=310994766,osdmap=1810408589}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=11,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=9} crc {logm=2868354033,mdsmap=2840952586,mgr=1318271769,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=54308681,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=3884565907,osdmap=1483771343}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,osd_metadata=5,osd_snap=31,osdmap=12} crc {logm=147206182,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1474155290,mon_config_key=3068527461,monmap=4080741109,osd_metadata=3305883572,osd_snap=3124346916,osdmap=1810408589}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=11,mdsmap=1,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=11} crc {logm=577926972,mdsmap=3842495371,mgr=4237663773,mgr_metadata=2750671369,mgrstat=3566304126,mon_config_key=4212284759,monmap=4080741110,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=3124346916,osdmap=3937667900}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=10} crc {logm=182107287,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2415727083,osdmap=2797090574}) 2024-04-19T00:58:33.190 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=9,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=11} crc {logm=4168335265,mdsmap=3140514009,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1893801906,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=522254654,osd_snap=2863464936,osdmap=2421511247}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4201189386}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2237749024}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1876793892}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=202572862}) 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.191 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4162285336}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1963907483}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=939455956}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3573245317}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.084+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2056904900}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2261480339}) 2024-04-19T00:58:33.192 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2502289927}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4111196937}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2145342163}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=2572675589}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1756711843}) 2024-04-19T00:58:33.193 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2810287217}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2162675296}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1816838756}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3736828625}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=734891113}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.194 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1518571795}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1427748427}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=4291382902}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1866459033}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1441774733}) 2024-04-19T00:58:33.195 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2280807618}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.088+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=3300463705}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2099480286}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=599451975}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1832157777}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.196 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3008776797}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3411746936}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1030836790}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=3865803018}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2128178230}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4251348863}) 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.197 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3895006678}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3504134065}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2026596547}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2073066117}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.198 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3156003922}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.092+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=1332890070}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1684673197}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3887061606}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=1417893824}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3772681948}) 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.199 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=966388595}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=135815304}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=171151757}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3769647477}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2078954117}) 2024-04-19T00:58:33.200 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=2464142592}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=3001772849}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.096+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2328482892}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=591293101}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3646646930}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.201 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1537491182}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=1069474848}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=1887966141}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3181780223}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=1232022331}) 2024-04-19T00:58:33.202 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=1193743844}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.100+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2899282575}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=1564963618}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=4058922873}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=64} crc {osdmap=1306923623}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=66} crc {osdmap=3793148982}) 2024-04-19T00:58:33.203 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=65} crc {osdmap=863243944}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=63} crc {osdmap=3141821739}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=64} crc {osdmap=3936917045}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=64} crc {osdmap=3283499519}) 2024-04-19T00:58:33.204 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T00:58:33.112+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=66} crc {osdmap=3898394157}) 2024-04-19T00:58:33.671 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T00:58:33.671 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient: get_monmap_and_config 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient: build_initial_monmap 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient: monmap: 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:33.800351+0000 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:33.800351+0000 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:33.797 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:33.798 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 Processor -- start 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 -- start start 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient: init 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient: _add_conns ranks=[5,7,10,9,4,6,3,8,2,1,0] 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5898138a50 0x7f5898138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): picked mon.noname-h con 0x7f5898138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f5898139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): picked mon.noname-i con 0x7f58980a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5898137960 0x7f5898137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): picked mon.noname-f con 0x7f5898137960 addr v2:172.21.15.161:3300/0 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f58980a1020 con 0x7f5898137960 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f58981398b0 con 0x7f5898138a50 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f58980a0780 con 0x7f58980a5bb0 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f589d470640 10 monclient(hunting): authenticate will time out at 1783.927442s 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f5898139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f5898139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:42564/0 (socket says 172.21.15.43:42564) 2024-04-19T00:58:33.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 1 -- 172.21.15.43:0/3968067426 learned_addr learned my addr 172.21.15.43:0/3968067426 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 10 monclient(hunting): get_auth_request con 0x7f58980a5bb0 auth_method 0 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f5896ffd640 1 --2- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5898138a50 0x7f5898138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58977fe640 1 --2- 172.21.15.43:0/3968067426 >> v2:172.21.15.161:3300/0 conn(0x7f5898137960 0x7f5898137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f5896ffd640 10 monclient(hunting): get_auth_request con 0x7f5898138a50 auth_method 0 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f5896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f5896ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f5896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58977fe640 10 monclient(hunting): get_auth_request con 0x7f5898137960 auth_method 0 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:33.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.795+0000 7f58967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 10 monclient(hunting): handle_auth_done global_id 74464 payload 274 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 -- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5898138a50 msgr2=0x7f5898138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 --2- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5898138a50 0x7f5898138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 -- 172.21.15.43:0/3968067426 >> v2:172.21.15.161:3300/0 conn(0x7f5898137960 msgr2=0x7f5898137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:33.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 --2- 172.21.15.43:0/3968067426 >> v2:172.21.15.161:3300/0 conn(0x7f5898137960 0x7f5898137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 10 monclient: _finish_hunting 0 2024-04-19T00:58:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 monclient: found mon.noname-i 2024-04-19T00:58:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:58:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 -- 172.21.15.43:0/3968067426 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980a0d40 con 0x7f58980a5bb0 2024-04-19T00:58:33.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f58967fc640 1 --2- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f5898139370 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f588c009870 tx=0x7f588c01c3f0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=b3852f6d8bc7837c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:33.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 1 -- 172.21.15.43:0/3968067426 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f588c0b2070 con 0x7f58980a5bb0 2024-04-19T00:58:33.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: dump: 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:33.812 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: _finish_auth 0 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 1 -- 172.21.15.43:0/3968067426 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c0ab7b0 con 0x7f58980a5bb0 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 5 monclient: authenticate success, global_id 74464 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 1 -- 172.21.15.43:0/3968067426 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f588c0abe50 con 0x7f58980a5bb0 2024-04-19T00:58:33.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 4 set_mon_vals no callback set 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f5895ffb640 10 monclient: dump: 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:33.814 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 10 monclient: shutdown 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 -- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 msgr2=0x7f5898139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 --2- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f5898139370 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f588c009870 tx=0x7f588c01c3f0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 -- 172.21.15.43:0/3968067426 shutdown_connections 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 --2- 172.21.15.43:0/3968067426 >> v2:172.21.15.161:3300/0 conn(0x7f5898137960 0x7f5898137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 --2- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f5898139370 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 --2- 172.21.15.43:0/3968067426 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5898138a50 0x7f5898138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 -- 172.21.15.43:0/3968067426 >> 172.21.15.43:0/3968067426 conn(0x7f58980a7cb0 msgr2=0x7f58980a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:33.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 -- 172.21.15.43:0/3968067426 shutdown_connections 2024-04-19T00:58:33.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.799+0000 7f589d470640 1 -- 172.21.15.43:0/3968067426 wait complete. 2024-04-19T00:58:33.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: build_initial_monmap 2024-04-19T00:58:33.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: monmap: 2024-04-19T00:58:33.816 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:33.817 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:33.818 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 librados: starting msgr at 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 librados: starting objecter 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 Processor -- start 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- start start 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 librados: setting wanted keys 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 librados: calling monclient init 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: init 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _add_conns ranks=[7,2,8,0,5,6,1,4,3] 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f58981d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): picked mon.noname-i con 0x7f58980a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 0x7f58981d1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): picked mon.noname-b con 0x7f5898137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:33.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5898138a50 0x7f589813c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): picked mon.noname-e con 0x7f5898138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f58980a2100 con 0x7f5898137960 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f58980a1f80 con 0x7f58980a5bb0 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f5898142810 con 0x7f5898138a50 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient(hunting): authenticate will time out at 1783.933271s 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 0x7f58981d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5898138a50 0x7f589813c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 0x7f58981d1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35810/0 (socket says 172.21.15.43:35810) 2024-04-19T00:58:33.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 1 -- 172.21.15.43:0/3901839294 learned_addr learned my addr 172.21.15.43:0/3901839294 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): get_auth_request con 0x7f5898137960 auth_method 0 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): get_auth_request con 0x7f5898138a50 auth_method 0 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f58981d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:33.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): get_auth_request con 0x7f58980a5bb0 auth_method 0 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient(hunting): handle_auth_done global_id 114362 payload 824 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 -- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 msgr2=0x7f58981d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f58981d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 -- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 msgr2=0x7f58981d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 0x7f58981d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:33.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 monclient: found mon.noname-e 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 -- 172.21.15.43:0/3901839294 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58981221a0 con 0x7f5898138a50 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58967fc640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 0x7f58981d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58977fe640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5898138a50 0x7f589813c950 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5890009d90 tx=0x7f589000c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 client.?.objecter ms_handle_connect 0x7f5898138a50 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5890018020 con 0x7f5898138a50 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: dump: 2024-04-19T00:58:33.823 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:33.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589000cd30 con 0x7f5898138a50 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5890014d60 con 0x7f5898138a50 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 5 monclient: authenticate success, global_id 114362 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _renew_subs 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589809fc90 con 0x7f5898138a50 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f58957fa640 4 set_mon_vals no callback set 2024-04-19T00:58:33.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 monclient: dump: 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:33.826 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 client.114362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _renew_subs 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f5898122480 con 0x7f5898138a50 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 librados: init done 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58980008e0 con 0x7f5898138a50 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f589001b020 con 0x7f5898138a50 2024-04-19T00:58:33.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5884038e40 0x7f588403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(532..532 src has 1..532) ==== 6244+0+0 (secure 0 0 0) 0x7f589004a140 con 0x7f5898138a50 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 10 client.114362.objecter ms_dispatch 0x7f5898000ba0 osd_map(532..532 src has 1..532) 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 3 client.114362.objecter handle_osd_map got epochs [532,532] > 0 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 3 client.114362.objecter handle_osd_map decoding full epoch 532 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f587f7fe640 20 client.114362.objecter dump_active .. 0 homeless 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5884038e40 0x7f588403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.803+0000 7f5896ffd640 10 monclient: get_auth_request con 0x7f5884038e40 auth_method 0 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.807+0000 7f587f7fe640 10 client.114362.objecter ms_handle_connect 0x7f5884038e40 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.807+0000 7f5896ffd640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5884038e40 0x7f588403b300 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f5888009a40 tx=0x7f5888009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.811+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5890022030 con 0x7f5898138a50 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.811+0000 7f587f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:33.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:33.811+0000 7f587f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:34.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f589d470640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:58:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f589d470640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:58:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f589d470640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f589813f230 con 0x7f5898138a50 2024-04-19T00:58:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f587f7fe640 1 -- 172.21.15.43:0/3901839294 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3192 (secure 0 0 0) 0x7f5890010040 con 0x7f5898138a50 2024-04-19T00:58:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f587f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:58:34.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.035+0000 7f587f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:34.038 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":94,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:58:34.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 10 librados: watch_flush enter 2024-04-19T00:58:34.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 10 librados: watch_flush exit 2024-04-19T00:58:34.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 20 client.114362.objecter shutdown clearing up homeless session... 2024-04-19T00:58:34.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 10 client.114362.objecter successfully canceled tick 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5884038e40 msgr2=0x7f588403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5884038e40 0x7f588403b300 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f5888009a40 tx=0x7f5888009190 comp rx=0 tx=0).stop 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 10 monclient: shutdown 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5898138a50 msgr2=0x7f589813c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5898138a50 0x7f589813c950 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5890009d90 tx=0x7f589000c490 comp rx=0 tx=0).stop 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 shutdown_connections 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5884038e40 0x7f588403b300 unknown :-1 s=CLOSED pgs=3567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5898138a50 0x7f589813c950 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5898137960 0x7f58981d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 --2- 172.21.15.43:0/3901839294 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f58980a5bb0 0x7f58981d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 >> 172.21.15.43:0/3901839294 conn(0x7f58980a7cb0 msgr2=0x7f589806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 shutdown_connections 2024-04-19T00:58:34.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 -- 172.21.15.43:0/3901839294 wait complete. 2024-04-19T00:58:34.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.039+0000 7f589d470640 1 librados: shutdown 2024-04-19T00:58:34.057 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T00:58:34.057 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T00:58:34.057 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T00:58:34.057 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:58:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient: get_monmap_and_config 2024-04-19T00:58:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient: monmap: 2024-04-19T00:58:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.179 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:34.182575+0000 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:34.182575+0000 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:34.180 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 Processor -- start 2024-04-19T00:58:34.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 -- start start 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient: init 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient: _add_conns ranks=[10,6,5,4,0,1,8,7,3,9,2] 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4fec135d20 0x7f4fec136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): picked mon.noname-f con 0x7f4fec135d20 addr v2:172.21.15.161:3300/0 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 0x7f4fec0a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): picked mon.noname-d con 0x7f4fec135860 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4fec0a3e80 0x7f4fec0a4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): picked mon.noname-h con 0x7f4fec0a3e80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4fec0a1780 con 0x7f4fec135d20 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4fec0a2400 con 0x7f4fec0a3e80 2024-04-19T00:58:34.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f4fec0a1130 con 0x7f4fec135860 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4ff0cab640 10 monclient(hunting): authenticate will time out at 1784.309876s 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 0x7f4fec0a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 0x7f4fec0a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42766/0 (socket says 172.21.15.43:42766) 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 -- 172.21.15.43:0/3534077512 learned_addr learned my addr 172.21.15.43:0/3534077512 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): get_auth_request con 0x7f4fec135860 auth_method 0 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 1 --2- 172.21.15.43:0/3534077512 >> v2:172.21.15.161:3300/0 conn(0x7f4fec135d20 0x7f4fec136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 1 --2- 172.21.15.43:0/3534077512 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4fec0a3e80 0x7f4fec0a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): get_auth_request con 0x7f4fec135d20 auth_method 0 2024-04-19T00:58:34.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): get_auth_request con 0x7f4fec0a3e80 auth_method 0 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient(hunting): handle_auth_done global_id 74643 payload 274 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 -- 172.21.15.43:0/3534077512 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4fec0a3e80 msgr2=0x7f4fec0a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 --2- 172.21.15.43:0/3534077512 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4fec0a3e80 0x7f4fec0a4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 -- 172.21.15.43:0/3534077512 >> v2:172.21.15.161:3300/0 conn(0x7f4fec135d20 msgr2=0x7f4fec136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 --2- 172.21.15.43:0/3534077512 >> v2:172.21.15.161:3300/0 conn(0x7f4fec135d20 0x7f4fec136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 monclient: found mon.noname-d 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.179+0000 7f4fe9d9b640 1 -- 172.21.15.43:0/3534077512 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec0a1410 con 0x7f4fec135860 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe9d9b640 1 --2- 172.21.15.43:0/3534077512 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 0x7f4fec0a3940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009870 tx=0x7f4fe001e6c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=32a11d139474195e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 1 -- 172.21.15.43:0/3534077512 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fe00a9070 con 0x7f4fec135860 2024-04-19T00:58:34.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: dump: 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.189 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 1 -- 172.21.15.43:0/3534077512 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe00b2070 con 0x7f4fec135860 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 1 -- 172.21.15.43:0/3534077512 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fe00add50 con 0x7f4fec135860 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe959a640 10 monclient: dump: 2024-04-19T00:58:34.190 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.191 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 5 monclient: authenticate success, global_id 74643 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 4 set_mon_vals no callback set 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: shutdown 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- 172.21.15.43:0/3534077512 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 msgr2=0x7f4fec0a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/3534077512 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 0x7f4fec0a3940 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4fe0009870 tx=0x7f4fe001e6c0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- 172.21.15.43:0/3534077512 shutdown_connections 2024-04-19T00:58:34.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/3534077512 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4fec0a3e80 0x7f4fec0a4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/3534077512 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135860 0x7f4fec0a3940 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/3534077512 >> v2:172.21.15.161:3300/0 conn(0x7f4fec135d20 0x7f4fec136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- 172.21.15.43:0/3534077512 >> 172.21.15.43:0/3534077512 conn(0x7f4fec0a85c0 msgr2=0x7f4fec0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- 172.21.15.43:0/3534077512 shutdown_connections 2024-04-19T00:58:34.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- 172.21.15.43:0/3534077512 wait complete. 2024-04-19T00:58:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: monmap: 2024-04-19T00:58:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:34.200 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 librados: starting msgr at 2024-04-19T00:58:34.201 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 librados: starting objecter 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 Processor -- start 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- start start 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 librados: setting wanted keys 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 librados: calling monclient init 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: init 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient: _add_conns ranks=[1,3,6,0,4,8,2,7,5] 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fec0a3e80 0x7f4fec12f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): picked mon.noname-f con 0x7f4fec0a3e80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4fec135860 0x7f4fec12fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): picked mon.noname-g con 0x7f4fec135860 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 0x7f4fec070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): picked mon.noname-d con 0x7f4fec135d20 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec06c1f0 con 0x7f4fec0a3e80 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4fec073460 con 0x7f4fec135860 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f4fec06c370 con 0x7f4fec135d20 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4ff0cab640 10 monclient(hunting): authenticate will time out at 1784.315405s 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 0x7f4fec070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 0x7f4fec070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42780/0 (socket says 172.21.15.43:42780) 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 -- 172.21.15.43:0/203066311 learned_addr learned my addr 172.21.15.43:0/203066311 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): get_auth_request con 0x7f4fec135d20 auth_method 0 2024-04-19T00:58:34.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fe9d9b640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4fec135860 0x7f4fec12fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fea59c640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fec0a3e80 0x7f4fec12f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient(hunting): handle_auth_done global_id 74652 payload 824 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4fec135860 msgr2=0x7f4fec12fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4fec135860 0x7f4fec12fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fec0a3e80 msgr2=0x7f4fec12f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4fec0a3e80 0x7f4fec12f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.204 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 monclient: found mon.noname-d 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.183+0000 7f4fead9d640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec11eb40 con 0x7f4fec135d20 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fead9d640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 0x7f4fec070020 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400e930 tx=0x7f4fe400edf0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 client.?.objecter ms_handle_connect 0x7f4fec135d20 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fe4017070 con 0x7f4fec135d20 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: dump: 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.205 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe400ce10 con 0x7f4fec135d20 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 5 monclient: authenticate success, global_id 74652 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 monclient: _renew_subs 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fec072290 con 0x7f4fec135d20 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fe40138e0 con 0x7f4fec135d20 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdbfff640 4 set_mon_vals no callback set 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 client.74652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:34.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: dump: 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.208 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 monclient: _renew_subs 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fec11e860 con 0x7f4fec135d20 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 1 librados: init done 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 0x7f4fec070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec06d140 con 0x7f4fec070560 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4fe401a070 con 0x7f4fec135d20 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 0x7f4fec070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient: get_auth_request con 0x7f4fec070560 auth_method 0 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4fcc038e10 0x7f4fcc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(532..532 src has 1..532) ==== 6244+0+0 (secure 0 0 0) 0x7f4fe400f040 con 0x7f4fec135d20 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 client.74652.objecter ms_dispatch 0x7f4fec000ba0 osd_map(532..532 src has 1..532) 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 3 client.74652.objecter handle_osd_map got epochs [532,532] > 0 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 3 client.74652.objecter handle_osd_map decoding full epoch 532 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 10 monclient(hunting): handle_auth_done global_id 65151 payload 824 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 20 client.74652.objecter dump_active .. 0 homeless 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fec0a0660 con 0x7f4fec070560 2024-04-19T00:58:34.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fe9d9b640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4fcc038e10 0x7f4fcc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fe9d9b640 10 monclient: get_auth_request con 0x7f4fcc038e10 auth_method 0 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 client.74652.objecter ms_handle_connect 0x7f4fec070560 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 client.74652.objecter resend_mon_ops 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fea59c640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 0x7f4fec070940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009ee0 tx=0x7f4fdc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fdc01c070 con 0x7f4fec070560 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: dump: 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.211 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4fdc00fdc0 con 0x7f4fec070560 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:34.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 msgr2=0x7f4fec070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 0x7f4fec070940 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009ee0 tx=0x7f4fdc00f4a0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 msgr2=0x7f4fec070940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070560 0x7f4fec070940 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009ee0 tx=0x7f4fdc00f4a0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fdaffd640 10 client.74652.objecter ms_handle_connect 0x7f4fcc038e10 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.187+0000 7f4fe9d9b640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4fcc038e10 0x7f4fcc03b2d0 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b1000 tx=0x7f4fe00a9000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 0x7f4fec0a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec09f730 con 0x7f4fec070e80 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 0x7f4fec0a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient: get_auth_request con 0x7f4fec070e80 auth_method 0 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 10 monclient(hunting): handle_auth_done global_id 65160 payload 824 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 1 -- 172.21.15.43:0/203066311 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4fec071730 con 0x7f4fec070e80 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 client.74652.objecter ms_handle_connect 0x7f4fec070e80 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 client.74652.objecter resend_mon_ops 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fea59c640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 0x7f4fec0a5e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009ee0 tx=0x7f4fdc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4fdc01c050 con 0x7f4fec070e80 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 monclient: dump: 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.214 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7f4fdc01d040 con 0x7f4fec070e80 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 msgr2=0x7f4fec0a5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 0x7f4fec0a5e30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009ee0 tx=0x7f4fdc004a50 comp rx=0 tx=0).stop 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 msgr2=0x7f4fec0a5e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fdaffd640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4fec070e80 0x7f4fec0a5e30 secure :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009ee0 tx=0x7f4fdc004a50 comp rx=0 tx=0).stop 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:34.216 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 440467, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:34.217 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.218 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.219 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.220 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.221 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:34.222 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.223 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.224 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.225 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.226 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.227 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.228 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.229 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.230 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.231 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.232 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.233 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.234 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.235 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:34.236 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.237 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.238 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:34.239 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:34.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 10 librados: watch_flush enter 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 10 librados: watch_flush exit 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 20 client.74652.objecter shutdown clearing up homeless session... 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 10 client.74652.objecter successfully canceled tick 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4fcc038e10 msgr2=0x7f4fcc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4fcc038e10 0x7f4fcc03b2d0 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b1000 tx=0x7f4fe00a9000 comp rx=0 tx=0).stop 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 10 monclient: shutdown 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 msgr2=0x7f4fec070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 0x7f4fec070020 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4fe400e930 tx=0x7f4fe400edf0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4fead9d640 1 -- 172.21.15.43:0/203066311 reap_dead start 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 shutdown_connections 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 --2- 172.21.15.43:0/203066311 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4fec135d20 0x7f4fec070020 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 >> 172.21.15.43:0/203066311 conn(0x7f4fec0a85c0 msgr2=0x7f4fec133c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 shutdown_connections 2024-04-19T00:58:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 -- 172.21.15.43:0/203066311 wait complete. 2024-04-19T00:58:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.195+0000 7f4ff0cab640 1 librados: shutdown 2024-04-19T00:58:34.243 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:34.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T00:58:34.243 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T00:58:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient: get_monmap_and_config 2024-04-19T00:58:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient: monmap: 2024-04-19T00:58:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.368 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:34.371779+0000 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:34.371779+0000 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.369 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.370 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.370 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:34.370 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:34.370 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 Processor -- start 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 -- start start 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient: init 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient: _add_conns ranks=[1,10,9,0,6,4,2,8,3,5,7] 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff9600a2a70 addr v1:172.21.15.161:6789/0 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff960136eb0 0x7ff96013bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): picked mon.noname-f con 0x7ff960136eb0 addr v2:172.21.15.161:3300/0 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff96013c250 0x7ff96013e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): picked mon.noname-a con 0x7ff96013c250 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9601372f0 con 0x7ff9600a2a70 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7ff960073890 con 0x7ff96013c250 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff960071d40 con 0x7ff960136eb0 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff964a68640 10 monclient(hunting): authenticate will time out at 1784.499647s 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff96013c250 0x7ff96013e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7ff96013c250 0x7ff96013e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58260/0 (socket says 172.21.15.43:58260) 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 1 -- 172.21.15.43:0/6846653 learned_addr learned my addr 172.21.15.43:0/6846653 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 10 monclient(hunting): get_auth_request con 0x7ff96013c250 auth_method 0 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.367+0000 7ff95ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff955d9b640 1 --2- 172.21.15.43:0/6846653 >> v2:172.21.15.161:3300/0 conn(0x7ff960136eb0 0x7ff96013bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff955d9b640 10 monclient(hunting): get_auth_request con 0x7ff960136eb0 auth_method 0 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff955d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff955d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 10 monclient(hunting): handle_auth_done global_id 65169 payload 274 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 1 -- 172.21.15.43:0/6846653 >> v2:172.21.15.161:3300/0 conn(0x7ff960136eb0 msgr2=0x7ff96013bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 1 --2- 172.21.15.43:0/6846653 >> v2:172.21.15.161:3300/0 conn(0x7ff960136eb0 0x7ff96013bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 1 -- 172.21.15.43:0/6846653 >> v1:172.21.15.161:6789/0 conn(0x7ff9600a2a70 legacy=0x7ff9600a2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 1 monclient: found mon.noname-a 2024-04-19T00:58:34.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:34.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 1 -- 172.21.15.43:0/6846653 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff960072ff0 con 0x7ff96013c250 2024-04-19T00:58:34.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95ed9d640 1 --2- 172.21.15.43:0/6846653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff96013c250 0x7ff96013e640 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff96006c8f0 tx=0x7ff95801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e870dcc4f2227f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 1 -- 172.21.15.43:0/6846653 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9580090d0 con 0x7ff96013c250 2024-04-19T00:58:34.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:34.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: dump: 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.383 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 1 -- 172.21.15.43:0/6846653 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9580a5af0 con 0x7ff96013c250 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 5 monclient: authenticate success, global_id 65169 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 1 -- 172.21.15.43:0/6846653 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff9580a5ce0 con 0x7ff96013c250 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff95dd9b640 10 monclient: dump: 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.384 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 4 set_mon_vals no callback set 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 10 monclient: shutdown 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 -- 172.21.15.43:0/6846653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff96013c250 msgr2=0x7ff96013e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 --2- 172.21.15.43:0/6846653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff96013c250 0x7ff96013e640 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff96006c8f0 tx=0x7ff95801c8b0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 -- 172.21.15.43:0/6846653 shutdown_connections 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 --2- 172.21.15.43:0/6846653 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7ff96013c250 0x7ff96013e640 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 --2- 172.21.15.43:0/6846653 >> v2:172.21.15.161:3300/0 conn(0x7ff960136eb0 0x7ff96013bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 -- 172.21.15.43:0/6846653 >> 172.21.15.43:0/6846653 conn(0x7ff9600a6270 msgr2=0x7ff96006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 -- 172.21.15.43:0/6846653 shutdown_connections 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 1 -- 172.21.15.43:0/6846653 wait complete. 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.371+0000 7ff964a68640 10 monclient: monmap: 2024-04-19T00:58:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.387 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 librados: starting msgr at 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 librados: starting objecter 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 Processor -- start 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- start start 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 librados: setting wanted keys 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 librados: calling monclient init 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: init 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: _add_conns ranks=[8,6,5,3,7,4,1,0,2] 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9600a2a70 0x7ff9601cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): picked mon.noname-e con 0x7ff9600a2a70 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff960136eb0 0x7ff9601cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): picked mon.noname-d con 0x7ff960136eb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff96013c250 0x7ff960139050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): picked mon.noname-h con 0x7ff96013c250 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ff960141730 con 0x7ff96013c250 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ff9601415b0 con 0x7ff960136eb0 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ff9601418b0 con 0x7ff9600a2a70 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): authenticate will time out at 1784.504914s 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9600a2a70 0x7ff9601cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff960136eb0 0x7ff9601cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9600a2a70 0x7ff9601cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:32856/0 (socket says 172.21.15.43:32856) 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 -- 172.21.15.43:0/4154977026 learned_addr learned my addr 172.21.15.43:0/4154977026 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): get_auth_request con 0x7ff9600a2a70 auth_method 0 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): get_auth_request con 0x7ff960136eb0 auth_method 0 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff96013c250 0x7ff960139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 10 monclient(hunting): get_auth_request con 0x7ff96013c250 auth_method 0 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient(hunting): handle_auth_done global_id 114371 payload 824 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff960136eb0 msgr2=0x7ff9601cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff960136eb0 0x7ff9601cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff96013c250 msgr2=0x7ff960139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff96013c250 0x7ff960139050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 monclient: found mon.noname-e 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96009cc30 con 0x7ff9600a2a70 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff960136eb0 0x7ff9601cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff96013c250 0x7ff960139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95e59c640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9600a2a70 0x7ff9601cea20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff95000c500 tx=0x7ff95000c9c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 client.?.objecter ms_handle_connect 0x7ff9600a2a70 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff950017070 con 0x7ff9600a2a70 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: dump: 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.392 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff950003600 con 0x7ff9600a2a70 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 5 monclient: authenticate success, global_id 114371 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: _renew_subs 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96013b830 con 0x7ff9600a2a70 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff950003ca0 con 0x7ff9600a2a70 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95d59a640 4 set_mon_vals no callback set 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: dump: 2024-04-19T00:58:34.394 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.395 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 client.114371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: _renew_subs 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96009cf10 con 0x7ff9600a2a70 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 librados: init done 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff950003830 con 0x7ff9600a2a70 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 0x7ff960139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff96009ba70 con 0x7ff960139590 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff940038e10 0x7ff94003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 0x7ff960139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.396 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(532..532 src has 1..532) ==== 6244+0+0 (secure 0 0 0) 0x7ff9500485c0 con 0x7ff9600a2a70 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 client.114371.objecter ms_dispatch 0x7ff960000ba0 osd_map(532..532 src has 1..532) 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient: get_auth_request con 0x7ff960139590 auth_method 0 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 3 client.114371.objecter handle_osd_map got epochs [532,532] > 0 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 3 client.114371.objecter handle_osd_map decoding full epoch 532 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 20 client.114371.objecter dump_active .. 0 homeless 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 10 monclient(hunting): handle_auth_done global_id 74630 payload 824 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff940038e10 0x7ff94003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff960072650 con 0x7ff960139590 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff95ed9d640 10 monclient: get_auth_request con 0x7ff940038e10 auth_method 0 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 client.114371.objecter ms_handle_connect 0x7ff960139590 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 client.114371.objecter resend_mon_ops 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff955d9b640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 0x7ff960139970 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff948010b60 tx=0x7ff94800e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff948018020 con 0x7ff960139590 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.375+0000 7ff9577fe640 10 monclient: dump: 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff948010e60 con 0x7ff960139590 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 msgr2=0x7ff960139970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 0x7ff960139970 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff948010b60 tx=0x7ff94800e5a0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 msgr2=0x7ff960139970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139590 0x7ff960139970 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff9577fe640 10 client.114371.objecter ms_handle_connect 0x7ff940038e10 2024-04-19T00:58:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.379+0000 7ff95ed9d640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff940038e10 0x7ff94003b2d0 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7ff9580ae000 tx=0x7ff9580a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 0x7ff9600a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff96006d140 con 0x7ff960139eb0 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 0x7ff9600a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient: get_auth_request con 0x7ff960139eb0 auth_method 0 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 10 monclient(hunting): handle_auth_done global_id 74639 payload 824 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 1 -- 172.21.15.43:0/4154977026 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff960076440 con 0x7ff960139eb0 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 client.114371.objecter ms_handle_connect 0x7ff960139eb0 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff955d9b640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 0x7ff9600a07d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff948010b60 tx=0x7ff9480046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 client.114371.objecter resend_mon_ops 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff948018040 con 0x7ff960139eb0 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 monclient: dump: 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9613 (secure 0 0 0) 0x7ff948003c00 con 0x7ff960139eb0 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 msgr2=0x7ff9600a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 0x7ff9600a07d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff948010b60 tx=0x7ff9480046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 msgr2=0x7ff9600a07d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff9577fe640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff960139eb0 0x7ff9600a07d0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff948010b60 tx=0x7ff9480046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:34.405 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 294004, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.406 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.407 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.408 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:34.409 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.410 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.411 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.412 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.413 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:34.414 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.415 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.416 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.417 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.418 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:34.419 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.420 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:34.421 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.422 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.423 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:34.424 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.425 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.426 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.427 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:34.428 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:34.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 10 librados: watch_flush enter 2024-04-19T00:58:34.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 10 librados: watch_flush exit 2024-04-19T00:58:34.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 20 client.114371.objecter shutdown clearing up homeless session... 2024-04-19T00:58:34.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 10 client.114371.objecter successfully canceled tick 2024-04-19T00:58:34.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff940038e10 msgr2=0x7ff94003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff940038e10 0x7ff94003b2d0 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7ff9580ae000 tx=0x7ff9580a7070 comp rx=0 tx=0).stop 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 10 monclient: shutdown 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9600a2a70 msgr2=0x7ff9601cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 --2- 172.21.15.43:0/4154977026 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff9600a2a70 0x7ff9601cea20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff95000c500 tx=0x7ff95000c9c0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff95ed9d640 1 -- 172.21.15.43:0/4154977026 reap_dead start 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff95ed9d640 1 -- 172.21.15.43:0/4154977026 reap_dead start 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 shutdown_connections 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.383+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 >> 172.21.15.43:0/4154977026 conn(0x7ff9600a6270 msgr2=0x7ff96012faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.387+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 shutdown_connections 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.387+0000 7ff964a68640 1 -- 172.21.15.43:0/4154977026 wait complete. 2024-04-19T00:58:34.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.387+0000 7ff964a68640 1 librados: shutdown 2024-04-19T00:58:34.432 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:34.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T00:58:34.432 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:58:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient: get_monmap_and_config 2024-04-19T00:58:34.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient: monmap: 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:34.558979+0000 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:34.558979+0000 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.556 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:34.557 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 Processor -- start 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 -- start start 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient: init 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient: _add_conns ranks=[9,4,1,6,2,0,7,8,5,3,10] 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 0x7f333c1392d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): picked mon.noname-a con 0x7f333c138ef0 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c09c9c0 0x7f333c0a2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): picked mon.noname-c con 0x7f333c09c9c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f333c0a2e70 addr v1:172.21.15.161:6789/0 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f333c138aa0 con 0x7f333c0a2e70 2024-04-19T00:58:34.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f333c09be00 con 0x7f333c138ef0 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f333c0a0de0 con 0x7f333c09c9c0 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3341616640 10 monclient(hunting): authenticate will time out at 1784.685890s 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 0x7f333c1392d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c09c9c0 0x7f333c0a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 0x7f333c1392d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58274/0 (socket says 172.21.15.43:58274) 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 1 -- 172.21.15.43:0/250102296 learned_addr learned my addr 172.21.15.43:0/250102296 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): get_auth_request con 0x7f333c138ef0 auth_method 0 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): get_auth_request con 0x7f333c09c9c0 auth_method 0 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient(hunting): handle_auth_done global_id 134167 payload 274 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 1 -- 172.21.15.43:0/250102296 >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 msgr2=0x7f333c1392d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 1 --2- 172.21.15.43:0/250102296 >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 0x7f333c1392d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 1 -- 172.21.15.43:0/250102296 >> v1:172.21.15.161:6789/0 conn(0x7f333c0a2e70 legacy=0x7f333c13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 1 monclient: found mon.noname-c 2024-04-19T00:58:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f3333fff640 1 -- 172.21.15.43:0/250102296 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c09d0e0 con 0x7f333c09c9c0 2024-04-19T00:58:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.555+0000 7f333affd640 1 --2- 172.21.15.43:0/250102296 >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 0x7f333c1392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3333fff640 1 --2- 172.21.15.43:0/250102296 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c09c9c0 0x7f333c0a2930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33240097c0 tx=0x7f332401c480 comp rx=0 tx=0).ready entity=mon.4 client_cookie=730e5d559beb797a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 1 -- 172.21.15.43:0/250102296 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33240b2070 con 0x7f333c09c9c0 2024-04-19T00:58:34.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: dump: 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.570 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 1 -- 172.21.15.43:0/250102296 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33240ab7b0 con 0x7f333c09c9c0 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 5 monclient: authenticate success, global_id 134167 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 1 -- 172.21.15.43:0/250102296 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33240abe50 con 0x7f333c09c9c0 2024-04-19T00:58:34.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 4 set_mon_vals no callback set 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333a7fc640 10 monclient: dump: 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.572 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: shutdown 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- 172.21.15.43:0/250102296 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c09c9c0 msgr2=0x7f333c0a2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 --2- 172.21.15.43:0/250102296 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c09c9c0 0x7f333c0a2930 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f33240097c0 tx=0x7f332401c480 comp rx=0 tx=0).stop 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- 172.21.15.43:0/250102296 shutdown_connections 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 --2- 172.21.15.43:0/250102296 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c09c9c0 0x7f333c0a2930 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 --2- 172.21.15.43:0/250102296 >> v2:172.21.15.43:3300/0 conn(0x7f333c138ef0 0x7f333c1392d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- 172.21.15.43:0/250102296 >> 172.21.15.43:0/250102296 conn(0x7f333c0a77d0 msgr2=0x7f333c0a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- 172.21.15.43:0/250102296 shutdown_connections 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- 172.21.15.43:0/250102296 wait complete. 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: monmap: 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.574 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 librados: starting msgr at 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 librados: starting objecter 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 Processor -- start 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- start start 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 librados: setting wanted keys 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 librados: calling monclient init 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: init 2024-04-19T00:58:34.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient: _add_conns ranks=[1,3,2,5,4,7,0,8,6] 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f333c09c9c0 0x7f333c12d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): picked mon.noname-f con 0x7f333c09c9c0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f333c0a2e70 0x7f333c12de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): picked mon.noname-g con 0x7f333c0a2e70 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f333c138ef0 0x7f333c134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): picked mon.noname-b con 0x7f333c138ef0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f333c09df20 con 0x7f333c09c9c0 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f333c09dda0 con 0x7f333c138ef0 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f333c139810 con 0x7f333c0a2e70 2024-04-19T00:58:34.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3341616640 10 monclient(hunting): authenticate will time out at 1784.691205s 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f333c138ef0 0x7f333c134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f333c138ef0 0x7f333c134200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35832/0 (socket says 172.21.15.43:35832) 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 -- 172.21.15.43:0/3043340042 learned_addr learned my addr 172.21.15.43:0/3043340042 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): get_auth_request con 0x7f333c138ef0 auth_method 0 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f333c09c9c0 0x7f333c12d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): get_auth_request con 0x7f333c09c9c0 auth_method 0 2024-04-19T00:58:34.577 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3333fff640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f333c0a2e70 0x7f333c12de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3333fff640 10 monclient(hunting): get_auth_request con 0x7f333c0a2e70 auth_method 0 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3333fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3333fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f3333fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient(hunting): handle_auth_done global_id 74648 payload 824 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f333c0a2e70 msgr2=0x7f333c12de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f333c0a2e70 0x7f333c12de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.578 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f333c09c9c0 msgr2=0x7f333c12d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f333c09c9c0 0x7f333c12d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 monclient: found mon.noname-b 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c139b70 con 0x7f333c138ef0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.559+0000 7f333b7fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f333c138ef0 0x7f333c134200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f333400c7d0 tx=0x7f333400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 client.?.objecter ms_handle_connect 0x7f333c138ef0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3334003df0 con 0x7f333c138ef0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:58:34.579 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: dump: 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.580 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f333400a850 con 0x7f333c138ef0 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f33340050d0 con 0x7f333c138ef0 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 5 monclient: authenticate success, global_id 74648 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 monclient: _renew_subs 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333c0a1000 con 0x7f333c138ef0 2024-04-19T00:58:34.581 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3339ffb640 4 set_mon_vals no callback set 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: dump: 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.582 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 client.74648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 monclient: _renew_subs 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f333c135d90 con 0x7f333c138ef0 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 1 librados: init done 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 0x7f333c13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f333c136720 con 0x7f333c12e380 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3334003980 con 0x7f333c138ef0 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 0x7f333c13afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient: get_auth_request con 0x7f333c12e380 auth_method 0 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f331c038e10 0x7f331c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(532..532 src has 1..532) ==== 6244+0+0 (secure 0 0 0) 0x7f33340491f0 con 0x7f333c138ef0 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 client.74648.objecter ms_dispatch 0x7f333c000ba0 osd_map(532..532 src has 1..532) 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 3 client.74648.objecter handle_osd_map got epochs [532,532] > 0 2024-04-19T00:58:34.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 3 client.74648.objecter handle_osd_map decoding full epoch 532 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 10 monclient(hunting): handle_auth_done global_id 134176 payload 824 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 20 client.74648.objecter dump_active .. 0 homeless 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f333c0693b0 con 0x7f333c12e380 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3333fff640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f331c038e10 0x7f331c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 client.74648.objecter ms_handle_connect 0x7f333c12e380 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 client.74648.objecter resend_mon_ops 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f333affd640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 0x7f333c13afd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f332c011860 tx=0x7f332c011d20 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3333fff640 10 monclient: get_auth_request con 0x7f331c038e10 auth_method 0 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f332c014440 con 0x7f333c12e380 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: dump: 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.585 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.586 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f332c00faa0 con 0x7f333c12e380 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 msgr2=0x7f333c13afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 0x7f333c13afd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f332c011860 tx=0x7f332c011d20 comp rx=0 tx=0).stop 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 msgr2=0x7f333c13afd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c12e380 0x7f333c13afd0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f332c011860 tx=0x7f332c011d20 comp rx=0 tx=0).stop 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f33337fe640 10 client.74648.objecter ms_handle_connect 0x7f331c038e10 2024-04-19T00:58:34.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.563+0000 7f3333fff640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f331c038e10 0x7f331c03b2d0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7f33240af000 tx=0x7f33240a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 0x7f333c134b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f333c06d140 con 0x7f333c134740 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 0x7f333c134b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient: get_auth_request con 0x7f333c134740 auth_method 0 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 10 monclient(hunting): handle_auth_done global_id 134185 payload 824 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 1 -- 172.21.15.43:0/3043340042 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f333c076520 con 0x7f333c134740 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333affd640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 0x7f333c134b50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f332c011860 tx=0x7f332c0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 client.74648.objecter ms_handle_connect 0x7f333c134740 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 client.74648.objecter resend_mon_ops 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f332c004490 con 0x7f333c134740 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 monclient: dump: 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.590 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9272 (secure 0 0 0) 0x7f332c02a420 con 0x7f333c134740 2024-04-19T00:58:34.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 msgr2=0x7f333c134b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 0x7f333c134b50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f332c011860 tx=0x7f332c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 msgr2=0x7f333c134b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f33337fe640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f333c134740 0x7f333c134b50 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f332c011860 tx=0x7f332c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:34.592 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19275, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.593 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.594 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.595 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:34.596 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.597 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.598 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.599 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.600 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:34.601 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.602 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:34.603 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.604 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:34.605 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.606 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.607 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.608 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.609 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.610 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.611 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.612 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.613 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:34.614 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 10 librados: watch_flush enter 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 10 librados: watch_flush exit 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 20 client.74648.objecter shutdown clearing up homeless session... 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 10 client.74648.objecter successfully canceled tick 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f331c038e10 msgr2=0x7f331c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f331c038e10 0x7f331c03b2d0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7f33240af000 tx=0x7f33240a7070 comp rx=0 tx=0).stop 2024-04-19T00:58:34.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 10 monclient: shutdown 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f333c138ef0 msgr2=0x7f333c134200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 --2- 172.21.15.43:0/3043340042 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f333c138ef0 0x7f333c134200 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f333400c7d0 tx=0x7f333400cc90 comp rx=0 tx=0).stop 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333b7fe640 1 -- 172.21.15.43:0/3043340042 reap_dead start 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f333b7fe640 1 -- 172.21.15.43:0/3043340042 reap_dead start 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 shutdown_connections 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 >> 172.21.15.43:0/3043340042 conn(0x7f333c0a77d0 msgr2=0x7f333c0a5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 shutdown_connections 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 -- 172.21.15.43:0/3043340042 wait complete. 2024-04-19T00:58:34.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.571+0000 7f3341616640 1 librados: shutdown 2024-04-19T00:58:34.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:34.618 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T00:58:34.618 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient: get_monmap_and_config 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient: monmap: 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:34.743398+0000 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:34.743398+0000 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:34.741 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 Processor -- start 2024-04-19T00:58:34.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 -- start start 2024-04-19T00:58:34.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient: init 2024-04-19T00:58:34.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient: _add_conns ranks=[7,3,6,10,9,4,2,0,5,1,8] 2024-04-19T00:58:34.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd12c138a50 0x7fd12c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): picked mon.noname-i con 0x7fd12c138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): picked mon.noname-g con 0x7fd12c0a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 0x7fd12c137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): picked mon.noname-d con 0x7fd12c137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd12c0a1020 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd12c1398b0 con 0x7fd12c137960 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd12c0a0780 con 0x7fd12c138a50 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd133624640 10 monclient(hunting): authenticate will time out at 1784.870660s 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 0x7fd12c137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 0x7fd12c137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42790/0 (socket says 172.21.15.43:42790) 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 -- 172.21.15.43:0/2327578897 learned_addr learned my addr 172.21.15.43:0/2327578897 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): get_auth_request con 0x7fd12c137960 auth_method 0 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd130bbf640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd12c138a50 0x7fd12c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd130bbf640 10 monclient(hunting): get_auth_request con 0x7fd12c0a5bb0 auth_method 0 2024-04-19T00:58:34.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd130bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd130bbf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd130bbf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd1313c0640 10 monclient(hunting): get_auth_request con 0x7fd12c138a50 auth_method 0 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd1313c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd1313c0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd1313c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient(hunting): handle_auth_done global_id 74670 payload 274 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 -- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd12c138a50 msgr2=0x7fd12c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd12c138a50 0x7fd12c138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 -- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c0a5bb0 msgr2=0x7fd12c139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 monclient: found mon.noname-d 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd131bc1640 1 -- 172.21.15.43:0/2327578897 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12c0a0d40 con 0x7fd12c137960 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.739+0000 7fd130bbf640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 0x7fd12c137dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd124009e80 tx=0x7fd12401c8e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=2477b38e4b7c39c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 1 -- 172.21.15.43:0/2327578897 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1240090d0 con 0x7fd12c137960 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: dump: 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.749 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 1 -- 172.21.15.43:0/2327578897 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1240a5b50 con 0x7fd12c137960 2024-04-19T00:58:34.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 5 monclient: authenticate success, global_id 74670 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 4 set_mon_vals no callback set 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 1 -- 172.21.15.43:0/2327578897 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1240a5d40 con 0x7fd12c137960 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd123fff640 10 monclient: dump: 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.751 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 10 monclient: shutdown 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 -- 172.21.15.43:0/2327578897 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 msgr2=0x7fd12c137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 0x7fd12c137dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fd124009e80 tx=0x7fd12401c8e0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 -- 172.21.15.43:0/2327578897 shutdown_connections 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c137960 0x7fd12c137dc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 --2- 172.21.15.43:0/2327578897 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd12c138a50 0x7fd12c138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 -- 172.21.15.43:0/2327578897 >> 172.21.15.43:0/2327578897 conn(0x7fd12c0a7cb0 msgr2=0x7fd12c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 -- 172.21.15.43:0/2327578897 shutdown_connections 2024-04-19T00:58:34.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 -- 172.21.15.43:0/2327578897 wait complete. 2024-04-19T00:58:34.765 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 10 monclient: monmap: 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:34.766 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 librados: starting msgr at 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 librados: starting objecter 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 Processor -- start 2024-04-19T00:58:34.767 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 -- start start 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 librados: setting wanted keys 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 1 librados: calling monclient init 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.743+0000 7fd133624640 10 monclient: init 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient: _add_conns ranks=[2,5,3,7,4,8,6,0,1] 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): picked mon.noname-b con 0x7fd12c0a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd12c137960 0x7fd12c0a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): picked mon.noname-h con 0x7fd12c137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c138a50 0x7fd12c0a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): picked mon.noname-g con 0x7fd12c138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd12c0a2100 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd12c0a1f80 con 0x7fd12c138a50 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fd12c142810 con 0x7fd12c137960 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): authenticate will time out at 1784.876366s 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35838/0 (socket says 172.21.15.43:35838) 2024-04-19T00:58:34.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 learned_addr learned my addr 172.21.15.43:0/2389736565 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): get_auth_request con 0x7fd12c0a5bb0 auth_method 0 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c138a50 0x7fd12c0a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): get_auth_request con 0x7fd12c138a50 auth_method 0 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd12c137960 0x7fd12c0a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.770 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): get_auth_request con 0x7fd12c137960 auth_method 0 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_done global_id 74657 payload 824 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd12c137960 msgr2=0x7fd12c0a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd12c137960 0x7fd12c0a4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c138a50 msgr2=0x7fd12c0a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd12c138a50 0x7fd12c0a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 monclient: found mon.noname-b 2024-04-19T00:58:34.771 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd12c09fc90 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd12800d5f0 tx=0x7fd12800dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 client.?.objecter ms_handle_connect 0x7fd12c0a5bb0 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd128012a60 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: dump: 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.772 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd128019070 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd128012c20 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: dump: 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.774 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1237fe640 4 set_mon_vals no callback set 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 5 monclient: authenticate success, global_id 74657 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient: _renew_subs 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd12c0a3230 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 client.74657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient: _renew_subs 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd12c09e680 con 0x7fd12c0a5bb0 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 librados: init done 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 msgr2=0x7fd12c09dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 msgr2=0x7fd12c09dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd12800d5f0 tx=0x7fd12800dab0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd12800d5f0 tx=0x7fd12800dab0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 msgr2=0x7fd12c09dc00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd12c0a5bb0 0x7fd12c09dc00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: _add_conns ranks=[1,0,6,3,4,5,8,2,7] 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd110006720 0x7fd110008b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): picked mon.f con 0x7fd110006720 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd110009260 0x7fd11000b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): picked mon.a con 0x7fd110009260 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd11000c120 0x7fd11000e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): picked mon.d con 0x7fd11000c120 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd11000f160 con 0x7fd110009260 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd11000f310 con 0x7fd110006720 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd11000f4c0 con 0x7fd11000c120 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd110009260 0x7fd11000b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd11000c120 0x7fd11000e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): get_auth_request con 0x7fd110009260 auth_method 0 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): get_auth_request con 0x7fd11000c120 auth_method 0 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd110006720 0x7fd110008b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd130bbf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): get_auth_request con 0x7fd110006720 auth_method 0 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient(hunting): handle_auth_done global_id 74657 payload 293 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd110006720 msgr2=0x7fd110008b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd110006720 0x7fd110008b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd110009260 msgr2=0x7fd11000b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd110009260 0x7fd11000b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 monclient: found mon.d 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fd11000f840 con 0x7fd11000c120 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 0x7fd12c135a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd12c06d140 con 0x7fd12c135670 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 -- 172.21.15.43:0/2389736565 reap_dead start 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 0x7fd12c135a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.780 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd131bc1640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd11000c120 0x7fd11000e5b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd1240ab000 tx=0x7fd128008040 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient: get_auth_request con 0x7fd12c135670 auth_method 0 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 client.74657.objecter ms_handle_connect 0x7fd11000c120 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 client.74657.objecter resend_mon_ops 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 10 monclient(hunting): handle_auth_done global_id 74679 payload 824 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd12c136900 con 0x7fd12c135670 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd124009510 con 0x7fd11000c120 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:34.781 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: dump: 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.782 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd124009250 con 0x7fd11000c120 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd122ffd640 4 set_mon_vals no callback set 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 client.74657.objecter ms_handle_connect 0x7fd12c135670 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.747+0000 7fd121ffb640 10 client.74657.objecter resend_mon_ops 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd1240a56e0 con 0x7fd11000c120 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 0x7fd12c135a50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd12801d000 tx=0x7fd128004740 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd11003da70 0x7fd11003ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 4 ==== osd_map(533..533 src has 1..533) ==== 6617+0+0 (secure 0 0 0) 0x7fd1240e17f0 con 0x7fd11000c120 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 10 client.74657.objecter ms_dispatch 0x7fd12c000ba0 osd_map(533..533 src has 1..533) 2024-04-19T00:58:34.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 3 client.74657.objecter handle_osd_map got epochs [533,533] > 0 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 3 client.74657.objecter handle_osd_map decoding full epoch 533 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 20 client.74657.objecter dump_active .. 0 homeless 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd128007380 con 0x7fd12c135670 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 10 monclient: dump: 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.784 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd128003620 con 0x7fd12c135670 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 msgr2=0x7fd12c135a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 0x7fd12c135a50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd12801d000 tx=0x7fd128004740 comp rx=0 tx=0).stop 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 msgr2=0x7fd12c135a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.751+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 0x7fd12c135a50 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd12801d000 tx=0x7fd128004740 comp rx=0 tx=0).stop 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd130bbf640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd11003da70 0x7fd11003ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd130bbf640 10 monclient: get_auth_request con 0x7fd11003da70 auth_method 0 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd130bbf640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd11003da70 0x7fd11003ff30 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7fd1100056a0 tx=0x7fd11c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 10 client.74657.objecter ms_handle_connect 0x7fd11003da70 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd133624640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 0x7fd12c135f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd133624640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd12c1364d0 con 0x7fd12c09dc00 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 0x7fd12c135f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient: get_auth_request con 0x7fd12c09dc00 auth_method 0 2024-04-19T00:58:34.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 10 monclient(hunting): handle_auth_done global_id 74688 payload 824 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 1 -- 172.21.15.43:0/2389736565 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd12c136d80 con 0x7fd12c09dc00 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd1313c0640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 0x7fd12c135f90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd12800d5f0 tx=0x7fd128014000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 10 client.74657.objecter ms_handle_connect 0x7fd12c09dc00 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 10 client.74657.objecter resend_mon_ops 2024-04-19T00:58:34.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd128039610 con 0x7fd12c09dc00 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.755+0000 7fd121ffb640 10 monclient: dump: 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.788 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9443 (secure 0 0 0) 0x7fd128003a80 con 0x7fd12c09dc00 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 msgr2=0x7fd12c135f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 0x7fd12c135f90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd12800d5f0 tx=0x7fd128014000 comp rx=0 tx=0).stop 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 msgr2=0x7fd12c135f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd121ffb640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 0x7fd12c135f90 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd12800d5f0 tx=0x7fd128014000 comp rx=0 tx=0).stop 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:34.790 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 294368, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.791 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.792 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.793 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:34.794 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:34.795 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.796 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.797 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.798 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.799 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.800 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:34.801 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.802 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:34.803 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.804 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.805 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.806 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.807 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.808 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.809 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.810 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.811 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.812 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.813 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.814 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:34.815 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:34.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 10 librados: watch_flush enter 2024-04-19T00:58:34.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 10 librados: watch_flush exit 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 20 client.74657.objecter shutdown clearing up homeless session... 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 10 client.74657.objecter successfully canceled tick 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd11003da70 msgr2=0x7fd11003ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd11003da70 0x7fd11003ff30 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7fd1100056a0 tx=0x7fd11c008040 comp rx=0 tx=0).stop 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 10 monclient: shutdown 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd11000c120 msgr2=0x7fd11000e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd11000c120 0x7fd11000e5b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd1240ab000 tx=0x7fd128008040 comp rx=0 tx=0).stop 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 shutdown_connections 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd11003da70 0x7fd11003ff30 unknown :-1 s=CLOSED pgs=3571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd11000c120 0x7fd11000e5b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c09dc00 0x7fd12c135f90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 --2- 172.21.15.43:0/2389736565 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd12c135670 0x7fd12c135a50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 >> 172.21.15.43:0/2389736565 conn(0x7fd12c0a7cb0 msgr2=0x7fd12c06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 shutdown_connections 2024-04-19T00:58:34.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 -- 172.21.15.43:0/2389736565 wait complete. 2024-04-19T00:58:34.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.759+0000 7fd133624640 1 librados: shutdown 2024-04-19T00:58:34.818 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:34.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T00:58:34.819 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T00:58:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient: get_monmap_and_config 2024-04-19T00:58:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient: monmap: 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:34.938969+0000 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:34.938969+0000 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.936 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:34.937 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:34.937 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 Processor -- start 2024-04-19T00:58:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 -- start start 2024-04-19T00:58:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient: init 2024-04-19T00:58:34.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient: _add_conns ranks=[7,9,4,3,1,6,5,10,8,0,2] 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa31c138cd0 0x7fa31c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): picked mon.noname-i con 0x7fa31c138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa31c0a5b80 0x7fa31c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): picked mon.noname-a con 0x7fa31c0a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa31c1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fa31c0a4410 con 0x7fa31c0a5b80 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa31c139b30 con 0x7fa31c1358b0 2024-04-19T00:58:34.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa31c0a3dc0 con 0x7fa31c138cd0 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa3212b3640 10 monclient(hunting): authenticate will time out at 1785.066093s 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa31c0a5b80 0x7fa31c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa31c0a5b80 0x7fa31c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58282/0 (socket says 172.21.15.43:58282) 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 -- 172.21.15.43:0/4263104639 learned_addr learned my addr 172.21.15.43:0/4263104639 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): get_auth_request con 0x7fa31c0a5b80 auth_method 0 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): get_auth_request con 0x7fa31c1358b0 auth_method 0 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31affd640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa31c138cd0 0x7fa31c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31affd640 10 monclient(hunting): get_auth_request con 0x7fa31c138cd0 auth_method 0 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_done global_id 65187 payload 274 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 -- 172.21.15.43:0/4263104639 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa31c138cd0 msgr2=0x7fa31c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa31c138cd0 0x7fa31c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 -- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 msgr2=0x7fa31c135d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c135d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 monclient: found mon.noname-a 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.935+0000 7fa31a7fc640 1 -- 172.21.15.43:0/4263104639 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c0a40a0 con 0x7fa31c0a5b80 2024-04-19T00:58:34.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31c0a5b80 0x7fa31c1395f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009870 tx=0x7fa30c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56f69ed356c9293 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 1 -- 172.21.15.43:0/4263104639 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa30c0b0070 con 0x7fa31c0a5b80 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: dump: 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.953 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 1 -- 172.21.15.43:0/4263104639 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa30c0aa810 con 0x7fa31c0a5b80 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 1 -- 172.21.15.43:0/4263104639 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa30c0a9300 con 0x7fa31c0a5b80 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 5 monclient: authenticate success, global_id 65187 2024-04-19T00:58:34.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 4 set_mon_vals no callback set 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa319ffb640 10 monclient: dump: 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.955 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: shutdown 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31c0a5b80 msgr2=0x7fa31c1395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31c0a5b80 0x7fa31c1395f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009870 tx=0x7fa30c01c3c0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- 172.21.15.43:0/4263104639 shutdown_connections 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa31c0a5b80 0x7fa31c1395f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c135d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- 172.21.15.43:0/4263104639 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa31c138cd0 0x7fa31c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- 172.21.15.43:0/4263104639 >> 172.21.15.43:0/4263104639 conn(0x7fa31c0ab2f0 msgr2=0x7fa31c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- 172.21.15.43:0/4263104639 shutdown_connections 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- 172.21.15.43:0/4263104639 wait complete. 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: build_initial_monmap 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: monmap: 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:34.957 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 librados: starting msgr at 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 librados: starting objecter 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 Processor -- start 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- start start 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 librados: setting wanted keys 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 librados: calling monclient init 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: init 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:34.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient: _add_conns ranks=[3,4,2,8,7,5,6,1,0] 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa31c0a5b80 0x7fa31c139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): picked mon.noname-g con 0x7fa31c0a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): picked mon.noname-c con 0x7fa31c1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa31c138cd0 0x7fa31c13eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): picked mon.noname-b con 0x7fa31c138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fa31c0a2010 con 0x7fa31c138cd0 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa31c0a1e90 con 0x7fa31c0a5b80 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa31c142aa0 con 0x7fa31c1358b0 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3212b3640 10 monclient(hunting): authenticate will time out at 1785.071128s 2024-04-19T00:58:34.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa31c138cd0 0x7fa31c13eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c133db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:46344/0 (socket says 172.21.15.43:46344) 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 -- 172.21.15.43:0/505672162 learned_addr learned my addr 172.21.15.43:0/505672162 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): get_auth_request con 0x7fa31c1358b0 auth_method 0 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): get_auth_request con 0x7fa31c138cd0 auth_method 0 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31affd640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa31c0a5b80 0x7fa31c139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient(hunting): handle_auth_done global_id 134203 payload 824 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa31c0a5b80 msgr2=0x7fa31c139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa31c0a5b80 0x7fa31c139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa31c138cd0 msgr2=0x7fa31c13eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fa31c138cd0 0x7fa31c13eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient: _finish_hunting 0 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 monclient: found mon.noname-c 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c13b950 con 0x7fa31c1358b0 2024-04-19T00:58:34.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31affd640 10 monclient: get_auth_request con 0x7fa31c0a5b80 auth_method 0 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31affd640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa31c0a5b80 0x7fa31c139fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa31a7fc640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c133db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009870 tx=0x7fa30c0191c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3137fe640 10 client.?.objecter ms_handle_connect 0x7fa31c1358b0 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.939+0000 7fa3137fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa30c0b0040 con 0x7fa31c1358b0 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: dump: 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.962 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:34.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 5 monclient: authenticate success, global_id 134203 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 monclient: _renew_subs 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa30c0044a0 con 0x7fa31c1358b0 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c13b670 con 0x7fa31c1358b0 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa30c0aabc0 con 0x7fa31c1358b0 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 client.134203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3197fa640 4 set_mon_vals no callback set 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: dump: 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.964 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.965 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 monclient: _renew_subs 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c13f3e0 con 0x7fa31c1358b0 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 1 librados: init done 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 0x7fa31c1d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa31c06d140 con 0x7fa31c1d0aa0 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa30c004030 con 0x7fa31c1358b0 2024-04-19T00:58:34.966 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 0x7fa31c1d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient: get_auth_request con 0x7fa31c1d0aa0 auth_method 0 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 0x7fa30003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(533..533 src has 1..533) ==== 6617+0+0 (secure 0 0 0) 0x7fa30c0e2b80 con 0x7fa31c1358b0 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 client.134203.objecter ms_dispatch 0x7fa31c000ba0 osd_map(533..533 src has 1..533) 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 3 client.134203.objecter handle_osd_map got epochs [533,533] > 0 2024-04-19T00:58:34.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 3 client.134203.objecter handle_osd_map decoding full epoch 533 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 10 monclient(hunting): handle_auth_done global_id 114380 payload 824 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa31c1344c0 con 0x7fa31c1d0aa0 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 20 client.134203.objecter dump_active .. 0 homeless 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 client.134203.objecter ms_handle_connect 0x7fa31c1d0aa0 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 client.134203.objecter resend_mon_ops 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31affd640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 0x7fa31c1d0e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa304007d90 tx=0x7fa30400f520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31b7fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 0x7fa30003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31b7fe640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 msgr2=0x7fa30003b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31b7fe640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 msgr2=0x7fa30003b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31b7fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 0x7fa30003b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:34.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa31b7fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 0x7fa30003b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa304003e10 con 0x7fa31c1d0aa0 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: dump: 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.969 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa304019070 con 0x7fa31c1d0aa0 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 msgr2=0x7fa31c1d0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 0x7fa31c1d0e80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa304007d90 tx=0x7fa30400f520 comp rx=0 tx=0).stop 2024-04-19T00:58:34.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 msgr2=0x7fa31c1d0e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.943+0000 7fa3137fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d0aa0 0x7fa31c1d0e80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 0x7fa31c12cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:34.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa31c068240 con 0x7fa31c1d13c0 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 0x7fa31c12cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient: get_auth_request con 0x7fa31c1d13c0 auth_method 0 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 10 monclient(hunting): handle_auth_done global_id 114389 payload 824 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 1 -- 172.21.15.43:0/505672162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa31c13b490 con 0x7fa31c1d13c0 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 client.134203.objecter ms_handle_connect 0x7fa31c1d13c0 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31affd640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 0x7fa31c12cc70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa304007d90 tx=0x7fa304004a50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:34.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 client.134203.objecter resend_mon_ops 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa3040046b0 con 0x7fa31c1d13c0 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 monclient: dump: 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:34.974 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa304004240 con 0x7fa31c1d13c0 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:34.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 msgr2=0x7fa31c12cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 0x7fa31c12cc70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa304007d90 tx=0x7fa304004a50 comp rx=0 tx=0).stop 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 msgr2=0x7fa31c12cc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3137fe640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa31c1d13c0 0x7fa31c12cc70 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa304007d90 tx=0x7fa304004a50 comp rx=0 tx=0).stop 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:34.976 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 152114, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:34.977 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.978 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.979 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.980 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:34.981 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:34.982 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:34.983 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.984 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:34.985 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.986 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.987 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.988 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:34.989 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.990 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.991 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.992 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.993 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.994 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.995 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.996 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:34.997 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.998 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:34.999 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.000 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:35.001 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 10 librados: watch_flush enter 2024-04-19T00:58:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 10 librados: watch_flush exit 2024-04-19T00:58:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 20 client.134203.objecter shutdown clearing up homeless session... 2024-04-19T00:58:35.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 10 client.134203.objecter successfully canceled tick 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 msgr2=0x7fa30003b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa300038e10 0x7fa30003b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 10 monclient: shutdown 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 msgr2=0x7fa31c133db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 --2- 172.21.15.43:0/505672162 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa31c1358b0 0x7fa31c133db0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa30c009870 tx=0x7fa30c0191c0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31b7fe640 1 -- 172.21.15.43:0/505672162 reap_dead start 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa31b7fe640 1 -- 172.21.15.43:0/505672162 reap_dead start 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 shutdown_connections 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 >> 172.21.15.43:0/505672162 conn(0x7fa31c0ab2f0 msgr2=0x7fa31c0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 shutdown_connections 2024-04-19T00:58:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 -- 172.21.15.43:0/505672162 wait complete. 2024-04-19T00:58:35.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:34.951+0000 7fa3212b3640 1 librados: shutdown 2024-04-19T00:58:35.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:35.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T00:58:35.004 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.127+0000 7f8a7209b640 10 monclient: get_monmap_and_config 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.127+0000 7f8a7209b640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.127+0000 7f8a7209b640 10 monclient: monmap: 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:35.134404+0000 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:35.134404+0000 2024-04-19T00:58:35.131 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:35.132 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:35.133 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.135 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 Processor -- start 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 -- start start 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient: init 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient: _add_conns ranks=[3,9,6,4,10,1,2,5,7,0,8] 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a6c138cd0 0x7f8a6c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): picked mon.noname-g con 0x7f8a6c138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 0x7f8a6c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a6c0a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 0x7f8a6c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): picked mon.noname-d con 0x7f8a6c1358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.136 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f8a6c0a4410 con 0x7f8a6c0a5b80 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c139b30 con 0x7f8a6c138cd0 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f8a6c0a3dc0 con 0x7f8a6c1358b0 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a7209b640 10 monclient(hunting): authenticate will time out at 1785.261696s 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 0x7f8a6c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 0x7f8a6c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 0x7f8a6c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58286/0 (socket says 172.21.15.43:58286) 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 1 -- 172.21.15.43:0/3690212520 learned_addr learned my addr 172.21.15.43:0/3690212520 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): get_auth_request con 0x7f8a6c0a5b80 auth_method 0 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 0x7f8a6c135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42818/0 (socket says 172.21.15.43:42818) 2024-04-19T00:58:35.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): get_auth_request con 0x7f8a6c1358b0 auth_method 0 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 1 --2- 172.21.15.43:0/3690212520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a6c138cd0 0x7f8a6c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): get_auth_request con 0x7f8a6c138cd0 auth_method 0 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_done global_id 74697 payload 274 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 -- 172.21.15.43:0/3690212520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a6c138cd0 msgr2=0x7f8a6c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 --2- 172.21.15.43:0/3690212520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a6c138cd0 0x7f8a6c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 -- 172.21.15.43:0/3690212520 >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 msgr2=0x7f8a6c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 --2- 172.21.15.43:0/3690212520 >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 0x7f8a6c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 monclient: found mon.noname-d 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6bfff640 1 -- 172.21.15.43:0/3690212520 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c0a40a0 con 0x7f8a6c1358b0 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.131+0000 7f8a6affd640 1 --2- 172.21.15.43:0/3690212520 >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 0x7f8a6c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 --2- 172.21.15.43:0/3690212520 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 0x7f8a6c135d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009d90 tx=0x7f8a6401c7d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=49f5fb4ac0b3157f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 1 -- 172.21.15.43:0/3690212520 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8a640b2070 con 0x7f8a6c1358b0 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: dump: 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.140 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 1 -- 172.21.15.43:0/3690212520 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a640a5990 con 0x7f8a6c1358b0 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 1 -- 172.21.15.43:0/3690212520 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8a640a5bc0 con 0x7f8a6c1358b0 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6a7fc640 10 monclient: dump: 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.142 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 5 monclient: authenticate success, global_id 74697 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:35.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 4 set_mon_vals no callback set 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: shutdown 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- 172.21.15.43:0/3690212520 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 msgr2=0x7f8a6c135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- 172.21.15.43:0/3690212520 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 0x7f8a6c135d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009d90 tx=0x7f8a6401c7d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- 172.21.15.43:0/3690212520 shutdown_connections 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- 172.21.15.43:0/3690212520 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f8a6c1358b0 0x7f8a6c135d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- 172.21.15.43:0/3690212520 >> v2:172.21.15.43:3300/0 conn(0x7f8a6c0a5b80 0x7f8a6c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- 172.21.15.43:0/3690212520 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8a6c138cd0 0x7f8a6c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- 172.21.15.43:0/3690212520 >> 172.21.15.43:0/3690212520 conn(0x7f8a6c0ab2f0 msgr2=0x7f8a6c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- 172.21.15.43:0/3690212520 shutdown_connections 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- 172.21.15.43:0/3690212520 wait complete. 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: monmap: 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:35.152 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 librados: starting msgr at 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 librados: starting objecter 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 Processor -- start 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- start start 2024-04-19T00:58:35.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 librados: setting wanted keys 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 librados: calling monclient init 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: init 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient: _add_conns ranks=[7,2,0,8,3,4,1,6,5] 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8a6c0a5b80 0x7f8a6c134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): picked mon.noname-i con 0x7f8a6c0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a6c1358b0 0x7f8a6c135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): picked mon.noname-b con 0x7f8a6c1358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a6c138cd0 0x7f8a6c1d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): picked mon.noname-a con 0x7f8a6c138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:58:35.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0a2010 con 0x7f8a6c138cd0 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c0a1e90 con 0x7f8a6c1358b0 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f8a6c142aa0 con 0x7f8a6c0a5b80 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a7209b640 10 monclient(hunting): authenticate will time out at 1785.267357s 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a6c138cd0 0x7f8a6c1d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a6c138cd0 0x7f8a6c1d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:58292/0 (socket says 172.21.15.43:58292) 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 -- 172.21.15.43:0/1784745649 learned_addr learned my addr 172.21.15.43:0/1784745649 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): get_auth_request con 0x7f8a6c138cd0 auth_method 0 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a6c1358b0 0x7f8a6c135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): get_auth_request con 0x7f8a6c1358b0 auth_method 0 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6b7fe640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8a6c0a5b80 0x7f8a6c134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient(hunting): handle_auth_done global_id 65205 payload 824 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8a6c0a5b80 msgr2=0x7f8a6c134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8a6c0a5b80 0x7f8a6c134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a6c1358b0 msgr2=0x7f8a6c135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8a6c1358b0 0x7f8a6c135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 monclient: found mon.noname-a 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.135+0000 7f8a6bfff640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c13c4a0 con 0x7f8a6c138cd0 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6bfff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a6c138cd0 0x7f8a6c1d0a30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a640b0040 tx=0x7f8a640195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 client.?.objecter ms_handle_connect 0x7f8a6c138cd0 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8a640b2050 con 0x7f8a6c138cd0 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: dump: 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.158 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.159 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a640be070 con 0x7f8a6c138cd0 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 5 monclient: authenticate success, global_id 65205 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 monclient: _renew_subs 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6c0a3590 con 0x7f8a6c138cd0 2024-04-19T00:58:35.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8a64003e10 con 0x7f8a6c138cd0 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: dump: 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.161 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 client.65205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a697fa640 4 set_mon_vals no callback set 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 monclient: _renew_subs 2024-04-19T00:58:35.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6c13e520 con 0x7f8a6c138cd0 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 1 librados: init done 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 0x7f8a6c1d1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c13dfe0 con 0x7f8a6c1d0f70 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8a640039a0 con 0x7f8a6c138cd0 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 0x7f8a5803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(533..533 src has 1..533) ==== 6617+0+0 (secure 0 0 0) 0x7f8a640e4420 con 0x7f8a6c138cd0 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 client.65205.objecter ms_dispatch 0x7f8a6c000ba0 osd_map(533..533 src has 1..533) 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 3 client.65205.objecter handle_osd_map got epochs [533,533] > 0 2024-04-19T00:58:35.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 3 client.65205.objecter handle_osd_map decoding full epoch 533 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 0x7f8a6c1d1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient: get_auth_request con 0x7f8a6c1d0f70 auth_method 0 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 20 client.65205.objecter dump_active .. 0 homeless 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6affd640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 0x7f8a5803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6affd640 10 monclient: get_auth_request con 0x7f8a58038e10 auth_method 0 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_done global_id 114445 payload 824 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a6c1d1890 con 0x7f8a6c1d0f70 2024-04-19T00:58:35.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6affd640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 msgr2=0x7f8a5803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6affd640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 msgr2=0x7f8a5803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6affd640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 0x7f8a5803b2d0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c13c0a0 tx=0x7f8a60009240 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6affd640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 0x7f8a5803b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a6b7fe640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 0x7f8a6c1d1350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010570 tx=0x7f8a5c010a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 client.65205.objecter ms_handle_connect 0x7f8a6c1d0f70 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 client.65205.objecter resend_mon_ops 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8a5c00f040 con 0x7f8a6c1d0f70 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: dump: 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.165 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8a5c01c400 con 0x7f8a6c1d0f70 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 msgr2=0x7f8a6c1d1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 0x7f8a6c1d1350 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010570 tx=0x7f8a5c010a30 comp rx=0 tx=0).stop 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 msgr2=0x7f8a6c1d1350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.139+0000 7f8a53fff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c1d0f70 0x7f8a6c1d1350 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010570 tx=0x7f8a5c010a30 comp rx=0 tx=0).stop 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a7209b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a7209b640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 0x7f8a6c09de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a7209b640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c06d140 con 0x7f8a6c09da80 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 0x7f8a6c09de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient: get_auth_request con 0x7f8a6c09da80 auth_method 0 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 10 monclient(hunting): handle_auth_done global_id 114454 payload 824 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 1 -- 172.21.15.43:0/1784745649 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a6c0693b0 con 0x7f8a6c09da80 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a6b7fe640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 0x7f8a6c09de90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010570 tx=0x7f8a5c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 client.65205.objecter ms_handle_connect 0x7f8a6c09da80 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 client.65205.objecter resend_mon_ops 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8a5c01ccb0 con 0x7f8a6c09da80 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 monclient: dump: 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.168 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f8a5c025070 con 0x7f8a6c09da80 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 msgr2=0x7f8a6c09de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 0x7f8a6c09de90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010570 tx=0x7f8a5c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 msgr2=0x7f8a6c09de90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.147+0000 7f8a53fff640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8a6c09da80 0x7f8a6c09de90 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c010570 tx=0x7f8a5c0046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:35.170 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 152317, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:58:35.171 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.172 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.173 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:35.174 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.175 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.176 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.177 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.178 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.179 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:35.180 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.181 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:35.182 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.183 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:35.184 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.185 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.186 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.187 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.188 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.189 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.190 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.191 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.192 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.193 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.194 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:35.195 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 10 librados: watch_flush enter 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 10 librados: watch_flush exit 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 20 client.65205.objecter shutdown clearing up homeless session... 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 10 client.65205.objecter successfully canceled tick 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 msgr2=0x7f8a5803b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8a58038e10 0x7f8a5803b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 10 monclient: shutdown 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a6c138cd0 msgr2=0x7f8a6c1d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 --2- 172.21.15.43:0/1784745649 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8a6c138cd0 0x7f8a6c1d0a30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a640b0040 tx=0x7f8a640195d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a6bfff640 1 -- 172.21.15.43:0/1784745649 reap_dead start 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a6bfff640 1 -- 172.21.15.43:0/1784745649 reap_dead start 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 shutdown_connections 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 >> 172.21.15.43:0/1784745649 conn(0x7f8a6c0ab2f0 msgr2=0x7f8a6c1368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 shutdown_connections 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 -- 172.21.15.43:0/1784745649 wait complete. 2024-04-19T00:58:35.198 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.151+0000 7f8a7209b640 1 librados: shutdown 2024-04-19T00:58:35.198 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:35.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T00:58:35.199 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:58:35.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient: get_monmap_and_config 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient: monmap: 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:35.323109+0000 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:35.323109+0000 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:35.320 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:35.321 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 Processor -- start 2024-04-19T00:58:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 -- start start 2024-04-19T00:58:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient: init 2024-04-19T00:58:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient: _add_conns ranks=[0,8,4,5,1,2,6,9,7,3,10] 2024-04-19T00:58:35.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac0c0a1f20 addr v1:172.21.15.43:6789/0 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c0a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): picked mon.noname-e con 0x7fac0c0a1270 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac0c13eba0 0x7fac0c140f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): picked mon.noname-c con 0x7fac0c13eba0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac0c09fff0 con 0x7fac0c0a1f20 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fac0c0a0920 con 0x7fac0c13eba0 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fac0c06bf40 con 0x7fac0c0a1270 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac13524640 10 monclient(hunting): authenticate will time out at 1785.450331s 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac0c13eba0 0x7fac0c140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac112c0640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fac0c0a1f20 0x7fac0c0a39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:38048/0 (socket says 172.21.15.43:38048) 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac112c0640 1 -- 172.21.15.43:0/2944329545 learned_addr learned my addr 172.21.15.43:0/2944329545 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 1 --2- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c0a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): get_auth_request con 0x7fac0c13eba0 auth_method 0 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): get_auth_request con 0x7fac0c0a1270 auth_method 0 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac11ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fabfffff640 1 -- 172.21.15.43:0/2944329545 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202155634 0 0) 0x7fac0c09fff0 con 0x7fac0c0a1f20 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fabfffff640 10 monclient(hunting): _negotiate 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fabfffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fabfffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fabfffff640 10 monclient(hunting): my global_id is 65214 2024-04-19T00:58:35.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fabfffff640 1 -- 172.21.15.43:0/2944329545 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabf8003530 con 0x7fac0c0a1f20 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient(hunting): handle_auth_done global_id 114398 payload 274 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 1 -- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac0c13eba0 msgr2=0x7fac0c140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 1 --2- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac0c13eba0 0x7fac0c140f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 1 -- 172.21.15.43:0/2944329545 >> v1:172.21.15.43:6789/0 conn(0x7fac0c0a1f20 legacy=0x7fac0c0a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 1 monclient: found mon.noname-e 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.319+0000 7fac10abf640 1 -- 172.21.15.43:0/2944329545 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0c0a02d0 con 0x7fac0c0a1270 2024-04-19T00:58:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac10abf640 1 --2- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c0a40c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac080097f0 tx=0x7fac0801c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8b3d29e40c202246 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 1 -- 172.21.15.43:0/2944329545 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac080b0070 con 0x7fac0c0a1270 2024-04-19T00:58:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: dump: 2024-04-19T00:58:35.335 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.336 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 1 -- 172.21.15.43:0/2944329545 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac080aa810 con 0x7fac0c0a1270 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 5 monclient: authenticate success, global_id 114398 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.337 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 1 -- 172.21.15.43:0/2944329545 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac080a9300 con 0x7fac0c0a1270 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fabfffff640 10 monclient: dump: 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.338 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 4 set_mon_vals no callback set 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: shutdown 2024-04-19T00:58:35.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 msgr2=0x7fac0c0a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 --2- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c0a40c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac080097f0 tx=0x7fac0801c3a0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- 172.21.15.43:0/2944329545 shutdown_connections 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 --2- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fac0c13eba0 0x7fac0c140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 --2- 172.21.15.43:0/2944329545 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c0a40c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- 172.21.15.43:0/2944329545 >> 172.21.15.43:0/2944329545 conn(0x7fac0c0a85a0 msgr2=0x7fac0c0a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- 172.21.15.43:0/2944329545 shutdown_connections 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- 172.21.15.43:0/2944329545 wait complete. 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: monmap: 2024-04-19T00:58:35.340 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.341 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 librados: starting msgr at 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 librados: starting objecter 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 Processor -- start 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- start start 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 librados: setting wanted keys 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 librados: calling monclient init 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: init 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient: _add_conns ranks=[8,1,7,4,0,5,3,6,2] 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c13d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): picked mon.noname-e con 0x7fac0c0a1270 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac0c0a1f20 0x7fac0c136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): picked mon.noname-f con 0x7fac0c0a1f20 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac0c13eba0 0x7fac0c137080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): picked mon.noname-i con 0x7fac0c13eba0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac0c0a2480 con 0x7fac0c0a1f20 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fac0c0a2300 con 0x7fac0c13eba0 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fac0c0a4600 con 0x7fac0c0a1270 2024-04-19T00:58:35.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac13524640 10 monclient(hunting): authenticate will time out at 1785.455734s 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c13d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c13d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:32900/0 (socket says 172.21.15.43:32900) 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 1 -- 172.21.15.43:0/4218621355 learned_addr learned my addr 172.21.15.43:0/4218621355 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): get_auth_request con 0x7fac0c0a1270 auth_method 0 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.323+0000 7fac112c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac0c0a1f20 0x7fac0c136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): get_auth_request con 0x7fac0c0a1f20 auth_method 0 2024-04-19T00:58:35.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac11ac1640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac0c13eba0 0x7fac0c137080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 10 monclient(hunting): handle_auth_done global_id 114407 payload 824 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac0c13eba0 msgr2=0x7fac0c137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fac0c13eba0 0x7fac0c137080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac0c0a1f20 msgr2=0x7fac0c136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac0c0a1f20 0x7fac0c136b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 monclient: found mon.noname-e 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0c13cbd0 con 0x7fac0c0a1270 2024-04-19T00:58:35.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac112c0640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c13d5b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c4d0 tx=0x7fac0000c990 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 client.?.objecter ms_handle_connect 0x7fac0c0a1270 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac00017070 con 0x7fac0c0a1270 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: dump: 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.346 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac00007070 con 0x7fac0c0a1270 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 5 monclient: authenticate success, global_id 114407 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 monclient: _renew_subs 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac00007690 con 0x7fac0c0a1270 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac0c073100 con 0x7fac0c0a1270 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabff7fe640 4 set_mon_vals no callback set 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: dump: 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.348 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 client.114407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 monclient: _renew_subs 2024-04-19T00:58:35.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fac0c13b680 con 0x7fac0c0a1270 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 1 librados: init done 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fac00007220 con 0x7fac0c0a1270 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 0x7fac0c09cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fac0c13c510 con 0x7fac0c1375c0 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fabf0038e10 0x7fabf003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(533..533 src has 1..533) ==== 6617+0+0 (secure 0 0 0) 0x7fac000488a0 con 0x7fac0c0a1270 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 client.114407.objecter ms_dispatch 0x7fac0c000ba0 osd_map(533..533 src has 1..533) 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 3 client.114407.objecter handle_osd_map got epochs [533,533] > 0 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 3 client.114407.objecter handle_osd_map decoding full epoch 533 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 0x7fac0c09cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 20 client.114407.objecter dump_active .. 0 homeless 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient: get_auth_request con 0x7fac0c1375c0 auth_method 0 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac11ac1640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fabf0038e10 0x7fabf003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac11ac1640 10 monclient: get_auth_request con 0x7fabf0038e10 auth_method 0 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 10 monclient(hunting): handle_auth_done global_id 19686 payload 824 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac0c1379a0 con 0x7fac0c1375c0 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 client.114407.objecter ms_handle_connect 0x7fac0c1375c0 2024-04-19T00:58:35.351 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac10abf640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 0x7fac0c09cb90 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0a58f0 tx=0x7fac0c0a56e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 client.114407.objecter resend_mon_ops 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 client.114407.objecter ms_handle_connect 0x7fabf0038e10 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fac11ac1640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fabf0038e10 0x7fabf003b2d0 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7fac0c09fbc0 tx=0x7fac0400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac080b9070 con 0x7fac0c1375c0 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: dump: 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.352 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac08003620 con 0x7fac0c1375c0 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 msgr2=0x7fac0c09cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 0x7fac0c09cb90 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0a58f0 tx=0x7fac0c0a56e0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 msgr2=0x7fac0c09cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.354 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.327+0000 7fabfdffb640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c1375c0 0x7fac0c09cb90 secure :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fac0c0a58f0 tx=0x7fac0c0a56e0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac13524640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac13524640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 0x7fac0c09d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac13524640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fac0c06d140 con 0x7fac0c09d0d0 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 0x7fac0c09d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient: get_auth_request con 0x7fac0c09d0d0 auth_method 0 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.356 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 10 monclient(hunting): handle_auth_done global_id 19695 payload 824 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 1 -- 172.21.15.43:0/4218621355 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac0c09da20 con 0x7fac0c09d0d0 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fabfdffb640 10 client.114407.objecter ms_handle_connect 0x7fac0c09d0d0 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fabfdffb640 10 client.114407.objecter resend_mon_ops 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fac10abf640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 0x7fac0c09d4e0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fac08000c10 tx=0x7fac080a6000 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fac080b9040 con 0x7fac0c09d0d0 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fabfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fabfdffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.335+0000 7fabfdffb640 10 monclient: dump: 2024-04-19T00:58:35.357 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.358 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9275 (secure 0 0 0) 0x7fac080b58c0 con 0x7fac0c09d0d0 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 msgr2=0x7fac0c09d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 0x7fac0c09d4e0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fac08000c10 tx=0x7fac080a6000 comp rx=0 tx=0).stop 2024-04-19T00:58:35.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 msgr2=0x7fac0c09d4e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fabfdffb640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fac0c09d0d0 0x7fac0c09d4e0 secure :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fac08000c10 tx=0x7fac080a6000 comp rx=0 tx=0).stop 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 858826, 2024-04-19T00:58:35.360 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:58:35.361 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.362 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.363 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:35.364 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.365 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:35.366 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.367 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.368 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.369 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.370 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.371 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.372 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.373 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.374 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:35.375 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.376 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.377 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.378 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.379 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.380 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.381 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.382 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:35.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:35.384 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 10 librados: watch_flush enter 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 10 librados: watch_flush exit 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 20 client.114407.objecter shutdown clearing up homeless session... 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 10 client.114407.objecter successfully canceled tick 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fabf0038e10 msgr2=0x7fabf003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fabf0038e10 0x7fabf003b2d0 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7fac0c09fbc0 tx=0x7fac0400b040 comp rx=0 tx=0).stop 2024-04-19T00:58:35.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 10 monclient: shutdown 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 msgr2=0x7fac0c13d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 --2- 172.21.15.43:0/4218621355 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fac0c0a1270 0x7fac0c13d5b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac0000c4d0 tx=0x7fac0000c990 comp rx=0 tx=0).stop 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac11ac1640 1 -- 172.21.15.43:0/4218621355 reap_dead start 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac11ac1640 1 -- 172.21.15.43:0/4218621355 reap_dead start 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 shutdown_connections 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 >> 172.21.15.43:0/4218621355 conn(0x7fac0c0a85a0 msgr2=0x7fac0c0a7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 shutdown_connections 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 -- 172.21.15.43:0/4218621355 wait complete. 2024-04-19T00:58:35.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.339+0000 7fac13524640 1 librados: shutdown 2024-04-19T00:58:35.388 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:35.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T00:58:35.388 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.503+0000 7f269aa97640 10 monclient: get_monmap_and_config 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.503+0000 7f269aa97640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.503+0000 7f269aa97640 10 monclient: monmap: 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:35.510195+0000 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:35.510195+0000 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.507 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:35.508 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 Processor -- start 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 -- start start 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient: init 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient: _add_conns ranks=[7,4,6,1,9,2,10,3,5,0,8] 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26940a1f20 0x7f26940a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): picked mon.noname-i con 0x7f26940a1f20 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:35.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26940a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): picked mon.noname-c con 0x7f26940a1270 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 0x7f269413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): picked mon.noname-d con 0x7f26940a4430 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f26940a0920 con 0x7f26940a1270 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f269406bf40 con 0x7f26940a4430 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f269409fff0 con 0x7f26940a1f20 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f269aa97640 10 monclient(hunting): authenticate will time out at 1785.637444s 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 0x7f269413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26940a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 0x7f269413fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42824/0 (socket says 172.21.15.43:42824) 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 -- 172.21.15.43:0/118759400 learned_addr learned my addr 172.21.15.43:0/118759400 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): get_auth_request con 0x7f26940a4430 auth_method 0 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): get_auth_request con 0x7f26940a1270 auth_method 0 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2698833640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26940a1f20 0x7f26940a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2698833640 10 monclient(hunting): get_auth_request con 0x7f26940a1f20 auth_method 0 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2698833640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2698833640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2698833640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient(hunting): handle_auth_done global_id 74706 payload 274 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 -- 172.21.15.43:0/118759400 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26940a1f20 msgr2=0x7f26940a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26940a1f20 0x7f26940a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 -- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 msgr2=0x7f26940a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26940a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 monclient: found mon.noname-d 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2699034640 1 -- 172.21.15.43:0/118759400 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26940a02d0 con 0x7f26940a4430 2024-04-19T00:58:35.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.507+0000 7f2693fff640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26940a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2699034640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 0x7f269413fba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f268c009e80 tx=0x7f268c01c910 comp rx=0 tx=0).ready entity=mon.6 client_cookie=8367ea74290573dd server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 1 -- 172.21.15.43:0/118759400 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f268c0090d0 con 0x7f26940a4430 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: dump: 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.518 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 1 -- 172.21.15.43:0/118759400 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f268c0a6980 con 0x7f26940a4430 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 1 -- 172.21.15.43:0/118759400 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f268c0af480 con 0x7f26940a4430 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 5 monclient: authenticate success, global_id 74706 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 4 set_mon_vals no callback set 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26937fe640 10 monclient: dump: 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.520 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.521 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: shutdown 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 msgr2=0x7f269413fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 0x7f269413fba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f268c009e80 tx=0x7f268c01c910 comp rx=0 tx=0).stop 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/118759400 shutdown_connections 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a4430 0x7f269413fba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26940a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- 172.21.15.43:0/118759400 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f26940a1f20 0x7f26940a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/118759400 >> 172.21.15.43:0/118759400 conn(0x7f26940a85a0 msgr2=0x7f26940a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/118759400 shutdown_connections 2024-04-19T00:58:35.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/118759400 wait complete. 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: monmap: 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.523 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 librados: starting msgr at 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 librados: starting objecter 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 Processor -- start 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- start start 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 librados: setting wanted keys 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 librados: calling monclient init 2024-04-19T00:58:35.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: init 2024-04-19T00:58:35.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: _add_conns ranks=[4,6,1,7,0,8,2,5,3] 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26941d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): picked mon.noname-c con 0x7f26940a1270 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a1f20 0x7f26941d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): picked mon.noname-d con 0x7f26940a1f20 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26940a4430 0x7f26941d1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): picked mon.noname-f con 0x7f26940a4430 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f26940a2480 con 0x7f26940a4430 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f26940a2300 con 0x7f26940a1270 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2694142900 con 0x7f26940a1f20 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient(hunting): authenticate will time out at 1785.642883s 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26941d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a1f20 0x7f26941d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26941d0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:46380/0 (socket says 172.21.15.43:46380) 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 -- 172.21.15.43:0/2029551379 learned_addr learned my addr 172.21.15.43:0/2029551379 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): get_auth_request con 0x7f26940a1270 auth_method 0 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): get_auth_request con 0x7f26940a1f20 auth_method 0 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2699034640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26940a4430 0x7f26941d1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient(hunting): handle_auth_done global_id 134230 payload 824 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a1f20 msgr2=0x7f26941d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f26940a1f20 0x7f26941d0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26940a4430 msgr2=0x7f26941d1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26940a4430 0x7f26941d1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 monclient: found mon.noname-c 2024-04-19T00:58:35.533 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2694072bb0 con 0x7f26940a1270 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2699034640 10 monclient: get_auth_request con 0x7f26940a4430 auth_method 0 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2699034640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26940a4430 0x7f26941d1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2698833640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26941d0a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f268400d800 tx=0x7f268400dcc0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 client.?.objecter ms_handle_connect 0x7f26940a1270 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f268400bb30 con 0x7f26940a1270 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: dump: 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.534 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.535 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2684019070 con 0x7f26940a1270 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f268400bcf0 con 0x7f26940a1270 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 5 monclient: authenticate success, global_id 134230 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: _renew_subs 2024-04-19T00:58:35.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f269409c280 con 0x7f26940a1270 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f2692ffd640 4 set_mon_vals no callback set 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f26917fa640 10 monclient: dump: 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.537 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 client.134230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: _renew_subs 2024-04-19T00:58:35.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f269409d0a0 con 0x7f26940a1270 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.511+0000 7f269aa97640 1 librados: init done 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f269aa97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f269aa97640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 0x7f2694131ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f2694070a70 con 0x7f2694131af0 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f268400b6c0 con 0x7f26940a1270 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2680038e10 0x7f268003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 0x7f2694131ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient: get_auth_request con 0x7f2694131af0 auth_method 0 2024-04-19T00:58:35.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(533..533 src has 1..533) ==== 6617+0+0 (secure 0 0 0) 0x7f268400f040 con 0x7f26940a1270 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 client.134230.objecter ms_dispatch 0x7f2694000ba0 osd_map(533..533 src has 1..533) 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 3 client.134230.objecter handle_osd_map got epochs [533,533] > 0 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 3 client.134230.objecter handle_osd_map decoding full epoch 533 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 20 client.134230.objecter dump_active .. 0 homeless 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2699034640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2680038e10 0x7f268003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2699034640 10 monclient: get_auth_request con 0x7f2680038e10 auth_method 0 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 10 monclient(hunting): handle_auth_done global_id 74624 payload 824 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f269409ef70 con 0x7f2694131af0 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2693fff640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 0x7f2694131ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f26940a1650 tx=0x7f2688011e30 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 client.134230.objecter ms_handle_connect 0x7f2694131af0 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 client.134230.objecter resend_mon_ops 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2688014440 con 0x7f2694131af0 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 monclient: dump: 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.541 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.542 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 client.134230.objecter ms_handle_connect 0x7f2680038e10 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f2699034640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2680038e10 0x7f268003b2d0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f268c0ae000 tx=0x7f268c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f268800fb30 con 0x7f2694131af0 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 msgr2=0x7f2694131ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 0x7f2694131ed0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f26940a1650 tx=0x7f2688011e30 comp rx=0 tx=0).stop 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 msgr2=0x7f2694131ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.515+0000 7f26917fa640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694131af0 0x7f2694131ed0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f26940a1650 tx=0x7f2688011e30 comp rx=0 tx=0).stop 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 0x7f2694132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f269406d140 con 0x7f2694132410 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 0x7f2694132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient: get_auth_request con 0x7f2694132410 auth_method 0 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 10 monclient(hunting): handle_auth_done global_id 74633 payload 824 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 1 -- 172.21.15.43:0/2029551379 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26940008e0 con 0x7f2694132410 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2693fff640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 0x7f2694132d10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f268801c000 tx=0x7f26880046d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 client.134230.objecter ms_handle_connect 0x7f2694132410 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 client.134230.objecter resend_mon_ops 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f268802b8c0 con 0x7f2694132410 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 monclient: dump: 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.545 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.546 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f268802b450 con 0x7f2694132410 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 msgr2=0x7f2694132d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 0x7f2694132d10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f268801c000 tx=0x7f26880046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 msgr2=0x7f2694132d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f26917fa640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f2694132410 0x7f2694132d10 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f268801c000 tx=0x7f26880046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 295143, 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:35.548 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:58:35.549 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.551 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.552 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.553 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.554 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.555 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.556 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.557 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:35.558 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.559 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.560 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.561 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:35.562 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.563 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:35.564 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.565 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.567 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.570 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:35.571 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 10 librados: watch_flush enter 2024-04-19T00:58:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 10 librados: watch_flush exit 2024-04-19T00:58:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 20 client.134230.objecter shutdown clearing up homeless session... 2024-04-19T00:58:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 10 client.134230.objecter successfully canceled tick 2024-04-19T00:58:35.573 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2680038e10 msgr2=0x7f268003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2680038e10 0x7f268003b2d0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f268c0ae000 tx=0x7f268c0a7000 comp rx=0 tx=0).stop 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 10 monclient: shutdown 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 msgr2=0x7f26941d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 --2- 172.21.15.43:0/2029551379 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f26940a1270 0x7f26941d0a00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f268400d800 tx=0x7f268400dcc0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2699034640 1 -- 172.21.15.43:0/2029551379 reap_dead start 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f2699034640 1 -- 172.21.15.43:0/2029551379 reap_dead start 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 shutdown_connections 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 >> 172.21.15.43:0/2029551379 conn(0x7f26940a85a0 msgr2=0x7f26941383b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 shutdown_connections 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 -- 172.21.15.43:0/2029551379 wait complete. 2024-04-19T00:58:35.574 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.523+0000 7f269aa97640 1 librados: shutdown 2024-04-19T00:58:35.575 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:35.575 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T00:58:35.575 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:58:35.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient: get_monmap_and_config 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient: monmap: 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:35.709062+0000 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:35.709062+0000 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.706 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:35.707 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 Processor -- start 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 -- start start 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient: init 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient: _add_conns ranks=[6,8,5,7,3,1,10,2,4,9,0] 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 0x7f0ad0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): picked mon.noname-d con 0x7f0ad0138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 0x7f0ad0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): picked mon.noname-e con 0x7f0ad00a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0137960 0x7f0ad0137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): picked mon.noname-h con 0x7f0ad0137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:35.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0ad00a1020 con 0x7f0ad0137960 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f0ad01398b0 con 0x7f0ad0138a50 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0ad00a0780 con 0x7f0ad00a5bb0 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.703+0000 7f0ad7e2f640 10 monclient(hunting): authenticate will time out at 1785.835963s 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 0x7f0ad0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 0x7f0ad0139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 0x7f0ad0138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:42842/0 (socket says 172.21.15.43:42842) 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 0x7f0ad0139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:32916/0 (socket says 172.21.15.43:32916) 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 -- 172.21.15.43:0/1594209603 learned_addr learned my addr 172.21.15.43:0/1594209603 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): get_auth_request con 0x7f0ad0138a50 auth_method 0 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): get_auth_request con 0x7f0ad00a5bb0 auth_method 0 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad63cc640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0137960 0x7f0ad0137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_done global_id 74724 payload 274 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 -- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 msgr2=0x7f0ad0139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 0x7f0ad0139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 -- 172.21.15.43:0/1594209603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0137960 msgr2=0x7f0ad0137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0137960 0x7f0ad0137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 monclient: found mon.noname-d 2024-04-19T00:58:35.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 -- 172.21.15.43:0/1594209603 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad00a0d40 con 0x7f0ad0138a50 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad63cc640 10 monclient: get_auth_request con 0x7f0ad0137960 auth_method 0 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad63cc640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0137960 0x7f0ad0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad53ca640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 0x7f0ad0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 0x7f0ad0138e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e80 tx=0x7f0ac001c8e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ef1e8c9540c8a97b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 1 -- 172.21.15.43:0/1594209603 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ac00090d0 con 0x7f0ad0138a50 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T00:58:35.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: dump: 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.722 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 1 -- 172.21.15.43:0/1594209603 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac00a5b50 con 0x7f0ad0138a50 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 5 monclient: authenticate success, global_id 74724 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 4 set_mon_vals no callback set 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 1 -- 172.21.15.43:0/1594209603 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ac00a5d40 con 0x7f0ad0138a50 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad4bc9640 10 monclient: dump: 2024-04-19T00:58:35.724 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.725 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 10 monclient: shutdown 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 msgr2=0x7f0ad0138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 0x7f0ad0138e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e80 tx=0x7f0ac001c8e0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/1594209603 shutdown_connections 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0137960 0x7f0ad0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0ad00a5bb0 0x7f0ad0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/1594209603 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f0ad0138a50 0x7f0ad0138e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/1594209603 >> 172.21.15.43:0/1594209603 conn(0x7f0ad00a7cb0 msgr2=0x7f0ad00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/1594209603 shutdown_connections 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/1594209603 wait complete. 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.707+0000 7f0ad7e2f640 10 monclient: monmap: 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.727 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.728 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 librados: starting msgr at 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 librados: starting objecter 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 Processor -- start 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- start start 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 librados: setting wanted keys 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 librados: calling monclient init 2024-04-19T00:58:35.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: init 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: _add_conns ranks=[3,1,5,4,6,2,0,7,8] 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00a5bb0 0x7f0ad009d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): picked mon.noname-g con 0x7f0ad00a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0137960 0x7f0ad009ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): picked mon.noname-f con 0x7f0ad0137960 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0138a50 0x7f0ad01d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): picked mon.noname-h con 0x7f0ad0138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0ad00a2100 con 0x7f0ad0137960 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f0ad00a1f80 con 0x7f0ad00a5bb0 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f0ad0142810 con 0x7f0ad0138a50 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): authenticate will time out at 1785.840830s 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0138a50 0x7f0ad01d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0137960 0x7f0ad009ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00a5bb0 0x7f0ad009d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0137960 0x7f0ad009ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:40100/0 (socket says 172.21.15.43:40100) 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 -- 172.21.15.43:0/4172999680 learned_addr learned my addr 172.21.15.43:0/4172999680 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): get_auth_request con 0x7f0ad0137960 auth_method 0 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): get_auth_request con 0x7f0ad00a5bb0 auth_method 0 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): get_auth_request con 0x7f0ad0138a50 auth_method 0 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad63cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient(hunting): handle_auth_done global_id 114472 payload 824 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0138a50 msgr2=0x7f0ad01d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f0ad0138a50 0x7f0ad01d07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00a5bb0 msgr2=0x7f0ad009d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f0ad00a5bb0 0x7f0ad009d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 monclient: found mon.noname-f 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ad009ee60 con 0x7f0ad0137960 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad53ca640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0137960 0x7f0ad009ded0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0ad009c660 tx=0x7f0ac400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 client.?.objecter ms_handle_connect 0x7f0ad0137960 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ac4017070 con 0x7f0ad0137960 2024-04-19T00:58:35.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: dump: 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.736 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 5 monclient: authenticate success, global_id 114472 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ac400cc30 con 0x7f0ad0137960 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: _renew_subs 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ad009bbe0 con 0x7f0ad0137960 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ac400cde0 con 0x7f0ad0137960 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 client.114472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0acffff640 4 set_mon_vals no callback set 2024-04-19T00:58:35.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ace7fc640 10 monclient: dump: 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.739 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: _renew_subs 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ad009e410 con 0x7f0ad0137960 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 librados: init done 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 0x7f0ad01d0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0ad009c3e0 con 0x7f0ad01359f0 2024-04-19T00:58:35.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 0x7f0ad01d0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient: get_auth_request con 0x7f0ad01359f0 auth_method 0 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.711+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0ac401a070 con 0x7f0ad0137960 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0abc038e10 0x7f0abc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(534..534 src has 1..534) ==== 6630+0+0 (secure 0 0 0) 0x7f0ac400f040 con 0x7f0ad0137960 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 client.114472.objecter ms_dispatch 0x7f0ad0000ba0 osd_map(534..534 src has 1..534) 2024-04-19T00:58:35.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 3 client.114472.objecter handle_osd_map got epochs [534,534] > 0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 3 client.114472.objecter handle_osd_map decoding full epoch 534 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_done global_id 74509 payload 824 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ad5bcb640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ad009bde0 con 0x7f0ad01359f0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 20 client.114472.objecter dump_active .. 0 homeless 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ad63cc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0abc038e10 0x7f0abc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 0x7f0ad01d0ce0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e50 tx=0x7f0ac0019680 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 client.114472.objecter ms_handle_connect 0x7f0ad01359f0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 client.114472.objecter resend_mon_ops 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ad63cc640 10 monclient: get_auth_request con 0x7f0abc038e10 auth_method 0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ac0003e10 con 0x7f0ad01359f0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 monclient: dump: 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.743 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 client.114472.objecter ms_handle_connect 0x7f0abc038e10 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ad63cc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0abc038e10 0x7f0abc03b2d0 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0136d80 tx=0x7f0ac8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ac00039a0 con 0x7f0ad01359f0 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 msgr2=0x7f0ad01d0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 0x7f0ad01d0ce0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e50 tx=0x7f0ac0019680 comp rx=0 tx=0).stop 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 msgr2=0x7f0ad01d0ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.715+0000 7f0ace7fc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01359f0 0x7f0ad01d0ce0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.719+0000 7f0ad7e2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:58:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.719+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 0x7f0ad012eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.719+0000 7f0ad7e2f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.719+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f0ad01d1630 con 0x7f0ad01d1220 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 0x7f0ad012eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient: get_auth_request con 0x7f0ad01d1220 auth_method 0 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 10 monclient(hunting): handle_auth_done global_id 74518 payload 824 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 1 -- 172.21.15.43:0/4172999680 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ad0076190 con 0x7f0ad01d1220 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad5bcb640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 0x7f0ad012eae0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e20 tx=0x7f0ac00a7000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 client.114472.objecter ms_handle_connect 0x7f0ad01d1220 2024-04-19T00:58:35.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 client.114472.objecter resend_mon_ops 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0ac00ad860 con 0x7f0ad01d1220 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 monclient: dump: 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.748 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f0ac00cc470 con 0x7f0ad01d1220 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 msgr2=0x7f0ad012eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 0x7f0ad012eae0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e20 tx=0x7f0ac00a7000 comp rx=0 tx=0).stop 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 msgr2=0x7f0ad012eae0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ace7fc640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f0ad01d1220 0x7f0ad012eae0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009e20 tx=0x7f0ac00a7000 comp rx=0 tx=0).stop 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 94, 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 295343, 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:58:35.750 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:58:35.751 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.752 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.753 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.754 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:58:35.755 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.756 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.757 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.759 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:58:35.760 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.761 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.762 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.763 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.764 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.765 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.766 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.767 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.768 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.769 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:58:35.770 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.771 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:58:35.772 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:58:35.773 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:58:35.774 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:58:35.775 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:58:35.775 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:58:35.775 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:58:35.775 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:58:35.775 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:58:35.775 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:35.776 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 10 librados: watch_flush enter 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 10 librados: watch_flush exit 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 20 client.114472.objecter shutdown clearing up homeless session... 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 10 client.114472.objecter successfully canceled tick 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0abc038e10 msgr2=0x7f0abc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0abc038e10 0x7f0abc03b2d0 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0136d80 tx=0x7f0ac8008040 comp rx=0 tx=0).stop 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 10 monclient: shutdown 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0137960 msgr2=0x7f0ad009ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 --2- 172.21.15.43:0/4172999680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0ad0137960 0x7f0ad009ded0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0ad009c660 tx=0x7f0ac400c520 comp rx=0 tx=0).stop 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad63cc640 1 -- 172.21.15.43:0/4172999680 reap_dead start 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad63cc640 1 -- 172.21.15.43:0/4172999680 reap_dead start 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 shutdown_connections 2024-04-19T00:58:35.777 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 >> 172.21.15.43:0/4172999680 conn(0x7f0ad00a7cb0 msgr2=0x7f0ad006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 shutdown_connections 2024-04-19T00:58:35.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 -- 172.21.15.43:0/4172999680 wait complete. 2024-04-19T00:58:35.778 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.723+0000 7f0ad7e2f640 1 librados: shutdown 2024-04-19T00:58:35.778 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T00:58:35.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['f', 'b', 'e', 'd'] 2024-04-19T00:58:35.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['i'] 2024-04-19T00:58:35.778 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f 2024-04-19T00:58:35.779 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.f store 2024-04-19T00:58:35.779 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f sync_force --yes-i-really-mean-it 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.895+0000 7f3ceb45f640 10 monclient: get_monmap_and_config 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.895+0000 7f3ceb45f640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient: monmap: 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:35.902431+0000 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:35.902431+0000 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.899 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:35.900 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 Processor -- start 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 -- start start 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient: init 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient: _add_conns ranks=[4,2,7,5,9,3,10,0,6,8,1] 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3ce40a4950 0x7f3ce4071700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): picked mon.noname-c con 0x7f3ce40a4950 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 0x7f3ce4136b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): picked mon.noname-b con 0x7f3ce4071c40 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3ce4072060 0x7f3ce41370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): picked mon.noname-i con 0x7f3ce4072060 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce40a4d30 con 0x7f3ce4071c40 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f3ce40a5250 con 0x7f3ce40a4950 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3ce40a5690 con 0x7f3ce4072060 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ceb45f640 10 monclient(hunting): authenticate will time out at 1786.029717s 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 0x7f3ce4136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3ce40a4950 0x7f3ce4071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 0x7f3ce4136b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35884/0 (socket says 172.21.15.43:35884) 2024-04-19T00:58:35.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): get_auth_request con 0x7f3ce40a4950 auth_method 0 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 1 -- 172.21.15.43:0/3438379910 learned_addr learned my addr 172.21.15.43:0/3438379910 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): get_auth_request con 0x7f3ce4071c40 auth_method 0 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce99fc640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3ce4072060 0x7f3ce41370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce99fc640 10 monclient(hunting): get_auth_request con 0x7f3ce4072060 auth_method 0 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce99fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce99fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce99fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_done global_id 134239 payload 274 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 -- 172.21.15.43:0/3438379910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3ce4072060 msgr2=0x7f3ce41370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3ce4072060 0x7f3ce41370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 -- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 msgr2=0x7f3ce4136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 0x7f3ce4136b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 monclient: found mon.noname-c 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:58:35.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce91fb640 1 -- 172.21.15.43:0/3438379910 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce4073180 con 0x7f3ce40a4950 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.899+0000 7f3ce89fa640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 0x7f3ce4136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3ce40a4950 0x7f3ce4071700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009820 tx=0x7f3ce001c380 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d193299ae2d7bf44 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 1 -- 172.21.15.43:0/3438379910 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ce00b0070 con 0x7f3ce40a4950 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: dump: 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.909 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 5 monclient: authenticate success, global_id 134239 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 1 -- 172.21.15.43:0/3438379910 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce00aa810 con 0x7f3ce40a4950 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: get_monmap_and_config success 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 4 set_mon_vals no callback set 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 1 -- 172.21.15.43:0/3438379910 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ce00a9300 con 0x7f3ce40a4950 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdbfff640 10 monclient: dump: 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.911 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.912 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: shutdown 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3ce40a4950 msgr2=0x7f3ce4071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3ce40a4950 0x7f3ce4071700 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009820 tx=0x7f3ce001c380 comp rx=0 tx=0).stop 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- 172.21.15.43:0/3438379910 shutdown_connections 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3ce4071c40 0x7f3ce4136b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3ce4072060 0x7f3ce41370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/3438379910 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3ce40a4950 0x7f3ce4071700 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- 172.21.15.43:0/3438379910 >> 172.21.15.43:0/3438379910 conn(0x7f3ce40a9770 msgr2=0x7f3ce40a5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- 172.21.15.43:0/3438379910 shutdown_connections 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- 172.21.15.43:0/3438379910 wait complete. 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: build_initial_monmap 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: monmap: 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:58:35.914 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 librados: starting msgr at 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 librados: starting objecter 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 Processor -- start 2024-04-19T00:58:35.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- start start 2024-04-19T00:58:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 librados: setting wanted keys 2024-04-19T00:58:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 librados: calling monclient init 2024-04-19T00:58:35.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: init 2024-04-19T00:58:35.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: _add_conns ranks=[8,1,3,6,5,7,0,2,4] 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3ce4071c40 0x7f3ce41ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): picked mon.noname-e con 0x7f3ce4071c40 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce4072060 0x7f3ce4133d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): picked mon.noname-f con 0x7f3ce4072060 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3ce40a4950 0x7f3ce4139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): picked mon.noname-g con 0x7f3ce40a4950 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce409ff80 con 0x7f3ce4072060 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce41379f0 con 0x7f3ce40a4950 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f3ce41408c0 con 0x7f3ce4071c40 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient(hunting): authenticate will time out at 1786.034958s 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3ce4071c40 0x7f3ce41ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3ce4071c40 0x7f3ce41ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:32920/0 (socket says 172.21.15.43:32920) 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 -- 172.21.15.43:0/1363210159 learned_addr learned my addr 172.21.15.43:0/1363210159 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): get_auth_request con 0x7f3ce4071c40 auth_method 0 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce89fa640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce4072060 0x7f3ce4133d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce99fc640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3ce40a4950 0x7f3ce4139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce89fa640 10 monclient(hunting): get_auth_request con 0x7f3ce4072060 auth_method 0 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce89fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce89fa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce89fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce99fc640 10 monclient(hunting): get_auth_request con 0x7f3ce40a4950 auth_method 0 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce99fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce99fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce99fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient(hunting): handle_auth_done global_id 114425 payload 824 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3ce40a4950 msgr2=0x7f3ce4139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3ce40a4950 0x7f3ce4139f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce4072060 msgr2=0x7f3ce4133d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce4072060 0x7f3ce4133d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient: _finish_hunting 0 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 monclient: found mon.noname-e 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce4072490 con 0x7f3ce4071c40 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ce91fb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3ce4071c40 0x7f3ce41ce8c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009710 tx=0x7f3ce0019120 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 client.?.objecter ms_handle_connect 0x7f3ce4071c40 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ce00b0050 con 0x7f3ce4071c40 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: dump: 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.927 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: _finish_auth 0 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: _check_auth_tickets 2024-04-19T00:58:35.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce00bc070 con 0x7f3ce4071c40 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 5 monclient: authenticate success, global_id 114425 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: _renew_subs 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce413bc00 con 0x7f3ce4071c40 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3ce0004250 con 0x7f3ce4071c40 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cd9ffb640 10 monclient: dump: 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.929 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.903+0000 7f3cdaffd640 4 set_mon_vals no callback set 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 10 client.114425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 10 monclient: _renew_subs 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce413c4b0 con 0x7f3ce4071c40 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 1 librados: init done 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 0x7f3ce413a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce4074f70 con 0x7f3ce413a4c0 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3ce0003620 con 0x7f3ce4071c40 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 0x7f3ce413a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient: get_auth_request con 0x7f3ce413a4c0 auth_method 0 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3cc8038e10 0x7f3cc803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(534..534 src has 1..534) ==== 6630+0+0 (secure 0 0 0) 0x7f3ce00e3f10 con 0x7f3ce4071c40 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 client.114425.objecter ms_dispatch 0x7f3ce4000ba0 osd_map(534..534 src has 1..534) 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 3 client.114425.objecter handle_osd_map got epochs [534,534] > 0 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 3 client.114425.objecter handle_osd_map decoding full epoch 534 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 20 client.114425.objecter dump_active .. 0 homeless 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce99fc640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3cc8038e10 0x7f3cc803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce99fc640 10 monclient: get_auth_request con 0x7f3cc8038e10 auth_method 0 2024-04-19T00:58:35.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_done global_id 114490 payload 824 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ce411f620 con 0x7f3ce413a4c0 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce89fa640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 0x7f3ce413a8a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40115d0 tx=0x7f3cd4011a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 client.114425.objecter ms_handle_connect 0x7f3ce413a4c0 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 client.114425.objecter resend_mon_ops 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3cd40044a0 con 0x7f3ce413a4c0 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3ce99fc640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3cc8038e10 0x7f3cc803b2d0 secure :-1 s=READY pgs=3575 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4134a00 tx=0x7f3cdc00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 monclient: dump: 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.933 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 client.114425.objecter ms_handle_connect 0x7f3cc8038e10 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3cd4004030 con 0x7f3ce413a4c0 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 msgr2=0x7f3ce413a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 0x7f3ce413a8a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40115d0 tx=0x7f3cd4011a90 comp rx=0 tx=0).stop 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 msgr2=0x7f3ce413a8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.907+0000 7f3cd9ffb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413a4c0 0x7f3ce413a8a0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40115d0 tx=0x7f3cd4011a90 comp rx=0 tx=0).stop 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ceb45f640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 0x7f3ce409da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ceb45f640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce406d140 con 0x7f3ce413ade0 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 0x7f3ce409da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient: get_auth_request con 0x7f3ce413ade0 auth_method 0 2024-04-19T00:58:35.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 10 monclient(hunting): handle_auth_done global_id 114499 payload 824 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 1 -- 172.21.15.43:0/1363210159 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f3ce40693b0 con 0x7f3ce413ade0 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3ce89fa640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 0x7f3ce409da80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4000c10 tx=0x7f3cd40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3cd9ffb640 10 client.114425.objecter ms_handle_connect 0x7f3ce413ade0 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3cd9ffb640 10 client.114425.objecter resend_mon_ops 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3cd402d080 con 0x7f3ce413ade0 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:58:35.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3cd9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.915+0000 7f3cd9ffb640 10 monclient: dump: 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:58:35.937 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f3cd4005c70 con 0x7f3ce413ade0 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 msgr2=0x7f3ce409da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 0x7f3ce409da80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4000c10 tx=0x7f3cd40046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 msgr2=0x7f3ce409da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:58:35.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3cd9ffb640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3ce413ade0 0x7f3ce409da80 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4000c10 tx=0x7f3cd40046d0 comp rx=0 tx=0).stop 2024-04-19T00:58:35.939 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:58:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T00:58:35.939 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T00:58:35.939 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 10 librados: watch_flush enter 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 10 librados: watch_flush exit 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 20 client.114425.objecter shutdown clearing up homeless session... 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 10 client.114425.objecter successfully canceled tick 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3cc8038e10 msgr2=0x7f3cc803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3cc8038e10 0x7f3cc803b2d0 secure :-1 s=READY pgs=3575 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4134a00 tx=0x7f3cdc00b040 comp rx=0 tx=0).stop 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 10 monclient: shutdown 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3ce4071c40 msgr2=0x7f3ce41ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 --2- 172.21.15.43:0/1363210159 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3ce4071c40 0x7f3ce41ce8c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009710 tx=0x7f3ce0019120 comp rx=0 tx=0).stop 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ce99fc640 1 -- 172.21.15.43:0/1363210159 reap_dead start 2024-04-19T00:58:35.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ce99fc640 1 -- 172.21.15.43:0/1363210159 reap_dead start 2024-04-19T00:58:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 shutdown_connections 2024-04-19T00:58:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 >> 172.21.15.43:0/1363210159 conn(0x7f3ce40a9770 msgr2=0x7f3ce40a0820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:58:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 shutdown_connections 2024-04-19T00:58:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 -- 172.21.15.43:0/1363210159 wait complete. 2024-04-19T00:58:35.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:35.919+0000 7f3ceb45f640 1 librados: shutdown 2024-04-19T00:58:35.942 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-04-19T00:58:35.942 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-04-19T00:58:35.943 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:58:35.964 INFO:tasks.ceph.mon.f:Stopped 2024-04-19T00:58:35.965 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-19T00:58:35.965 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-19T00:58:35.965 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T00:58:35.965 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:58:35.980 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T00:58:35.980 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.e 2024-04-19T00:58:35.981 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-04-19T00:58:35.981 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-04-19T00:58:35.981 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:58:36.036 INFO:tasks.ceph.mon.e:Stopped 2024-04-19T00:58:36.036 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-04-19T00:58:36.036 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T00:58:36.037 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-04-19T00:58:36.037 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T00:58:36.097 INFO:tasks.ceph.mon.d:Stopped 2024-04-19T00:58:36.097 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-04-19T00:58:36.098 INFO:tasks.ceph.mon.i:Sent signal 19 2024-04-19T00:58:36.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T00:58:51.098 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-04-19T00:58:51.099 INFO:tasks.ceph.mon.i:Sent signal 18 2024-04-19T00:58:51.100 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-04-19T00:58:51.100 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T00:58:51.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient: get_monmap_and_config 2024-04-19T00:58:51.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient: build_initial_monmap 2024-04-19T00:58:51.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient: monmap: 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:58:51.235958+0000 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:58:51.235958+0000 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:58:51.233 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:58:51.234 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:58:51.234 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:58:51.234 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:58:51.234 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 Processor -- start 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 -- start start 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient: init 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient: _reopen_session rank -1 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient: _add_conns ranks=[5,4,10,0,3,7,8,1,2,6,9] 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): picked mon.noname-h con 0x7fb84809e430 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): picked mon.noname-c con 0x7fb84809e810 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 0x7fb8480a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:51.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): picked mon.noname-f con 0x7fb84809ec70 addr v2:172.21.15.161:3300/0 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb84806c680 con 0x7fb84809ec70 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fb84809bc10 con 0x7fb84809e810 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb8480a5440 con 0x7fb84809e430 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb850584640 10 monclient(hunting): authenticate will time out at 1801.363129s 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84eb21640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 msgr2=0x7fb8480a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84eb21640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 0x7fb8480a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:37244/0 (socket says 172.21.15.43:37244) 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 learned_addr learned my addr 172.21.15.43:0/1144023059 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb84809e810 auth_method 0 2024-04-19T00:58:51.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb84809e430 auth_method 0 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:51.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:51.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.231+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 msgr2=0x7fb8480a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84eb21640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 0x7fb8480a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb84809e810 auth_method 0 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:51.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb84809e430 auth_method 0 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:51.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:51.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:51.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:51.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:51.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:51.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:51.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.435+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 msgr2=0x7fb8480a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84eb21640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 0x7fb8480a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb84809e810 auth_method 0 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb84809e430 auth_method 0 2024-04-19T00:58:51.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:51.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:51.839+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 msgr2=0x7fb8480a3ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84eb21640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 0x7fb8480a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb84809e810 auth_method 0 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb84809e430 auth_method 0 2024-04-19T00:58:52.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:52.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:52.643+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:58:54.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): tick 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 monclient(hunting): continuing hunt 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 msgr2=0x7fb848136e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb84809e430 0x7fb848136e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 msgr2=0x7fb8480a39b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb84809e810 0x7fb8480a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 msgr2=0x7fb8480a3ef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.161:3300/0 conn(0x7fb84809ec70 0x7fb8480a3ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient: _add_conns ranks=[9,3,1,6,0,7,2,8] 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb834003800 addr v2:172.21.15.43:3300/0 2024-04-19T00:58:54.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-g con 0x7fb834006190 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fb834008cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb848073390 con 0x7fb834008cd0 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fb84806c680 con 0x7fb834003800 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fb84809bc10 con 0x7fb834006190 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb847fff640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v1:172.21.15.161:6789/0 conn(0x7fb834008cd0 legacy=0x7fb83400b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834003800 auth_method 0 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:54.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb834006190 auth_method 0 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:54.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.235+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v1:172.21.15.161:6789/0 conn(0x7fb834008cd0 legacy=0x7fb83400b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834003800 auth_method 0 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:54.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb834006190 auth_method 0 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:54.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.439+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v1:172.21.15.161:6789/0 conn(0x7fb834008cd0 legacy=0x7fb83400b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834003800 auth_method 0 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:54.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb834006190 auth_method 0 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:54.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:54.843+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 >> v1:172.21.15.161:6789/0 conn(0x7fb834008cd0 legacy=0x7fb83400b190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834003800 auth_method 0 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:58:55.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 10 monclient(hunting): get_auth_request con 0x7fb834006190 auth_method 0 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:58:55.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:55.647+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:58:57.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 10 monclient(hunting): tick 2024-04-19T00:58:57.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:58:57.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 monclient(hunting): continuing hunt 2024-04-19T00:58:57.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:58:57.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fb834006190 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 msgr2=0x7fb834005c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> v2:172.21.15.43:3300/0 conn(0x7fb834003800 0x7fb834005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> v1:172.21.15.161:6789/0 conn(0x7fb834008cd0 legacy=0x7fb83400b190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 reap_dead start 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 10 monclient: _add_conns ranks=[2,6,0,8,7] 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 reap_dead start 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 0x7fb834008620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb83400cbc0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 0x7fb83400d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.231+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-d con 0x7fb83400cfa0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb83400b190 addr v1:172.21.15.43:6789/0 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb83400dbd0 con 0x7fb83400b190 2024-04-19T00:58:57.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb84806c680 con 0x7fb83400cbc0 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb84809bc10 con 0x7fb83400cfa0 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb847fff640 10 monclient(hunting): _renew_subs 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 msgr2=0x7fb83400d380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 0x7fb83400d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:57.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:58:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.439+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 msgr2=0x7fb83400d380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:58:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.439+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:58:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.439+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 0x7fb83400d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:57.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:58:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.843+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:58:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.843+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 msgr2=0x7fb83400d380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:58:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.843+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:57.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:57.843+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 0x7fb83400d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:58:58.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:58.647+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 msgr2=0x7fb83400d380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:58:58.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:58.647+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:58:58.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:58.647+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 0x7fb83400d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:58:58.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:58:58.647+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): tick 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 monclient(hunting): continuing hunt 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 msgr2=0x7fb83400d380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb83400cfa0 0x7fb83400d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 msgr2=0x7fb834008620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb83400cbc0 0x7fb834008620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 >> v1:172.21.15.43:6789/0 conn(0x7fb83400b190 legacy=0x7fb83400b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient: _add_conns ranks=[7,8] 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-i con 0x7fb834005c30 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 0x7fb834013490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): picked mon.noname-e con 0x7fb8340110a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb84806c680 con 0x7fb834005c30 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 1 -- 172.21.15.43:0/1144023059 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb84809bc10 con 0x7fb8340110a0 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb847fff640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 msgr2=0x7fb834013490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 0x7fb834013490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834005c30 auth_method 0 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 msgr2=0x7fb83400c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 msgr2=0x7fb83400c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:59:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.235+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:00.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 msgr2=0x7fb834013490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:00.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 0x7fb834013490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834005c30 auth_method 0 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 msgr2=0x7fb83400c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 msgr2=0x7fb83400c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:59:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.439+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:59:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84db1f640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 msgr2=0x7fb834013490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:00.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84db1f640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 0x7fb834013490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834005c30 auth_method 0 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:00.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient(hunting): handle_auth_done global_id 74536 payload 274 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 msgr2=0x7fb834013490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb8340110a0 0x7fb834013490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient: _finish_hunting 0 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 1 monclient: found mon.noname-i 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:59:00.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.843+0000 7fb84e320640 1 -- 172.21.15.43:0/1144023059 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb848073390 con 0x7fb834005c30 2024-04-19T00:59:00.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.847+0000 7fb84e320640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb840009de0 tx=0x7fb84000a870 comp rx=0 tx=0).ready entity=mon.7 client_cookie=17af33f5803d061 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 1 -- 172.21.15.43:0/1144023059 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8400ad070 con 0x7fb834005c30 2024-04-19T00:59:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:59:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: dump: 2024-04-19T00:59:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:00.851 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:00.852 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: _finish_auth 0 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: _check_auth_tickets 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 1 -- 172.21.15.43:0/1144023059 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8400a7ab0 con 0x7fb834005c30 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 1 -- 172.21.15.43:0/1144023059 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb8400a7c60 con 0x7fb834005c30 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84d31e640 10 monclient: dump: 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.853 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 5 monclient: authenticate success, global_id 74536 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 10 monclient: get_monmap_and_config success 2024-04-19T00:59:00.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 4 set_mon_vals no callback set 2024-04-19T00:59:00.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 10 monclient: shutdown 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 1 -- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 msgr2=0x7fb83400c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 1 --2- 172.21.15.43:0/1144023059 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb834005c30 0x7fb83400c070 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb840009de0 tx=0x7fb84000a870 comp rx=0 tx=0).stop 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb84eb21640 1 -- 172.21.15.43:0/1144023059 reap_dead start 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 1 -- 172.21.15.43:0/1144023059 shutdown_connections 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 1 -- 172.21.15.43:0/1144023059 >> 172.21.15.43:0/1144023059 conn(0x7fb8480a85a0 msgr2=0x7fb8480a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 1 -- 172.21.15.43:0/1144023059 shutdown_connections 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.851+0000 7fb850584640 1 -- 172.21.15.43:0/1144023059 wait complete. 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: build_initial_monmap 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: monmap: 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:59:00.856 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:00.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 librados: starting msgr at 2024-04-19T00:59:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 librados: starting objecter 2024-04-19T00:59:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 Processor -- start 2024-04-19T00:59:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 -- start start 2024-04-19T00:59:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 librados: setting wanted keys 2024-04-19T00:59:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 librados: calling monclient init 2024-04-19T00:59:00.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: init 2024-04-19T00:59:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: _add_conns ranks=[0,2,4,6,5,1,3,7,8] 2024-04-19T00:59:00.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb834005c30 0x7fb8481cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): picked mon.noname-a con 0x7fb834005c30 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb848134b70 0x7fb848134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): picked mon.noname-b con 0x7fb848134b70 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 0x7fb848124a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): picked mon.noname-c con 0x7fb848124640 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb848136e30 con 0x7fb834005c30 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb8480a20b0 con 0x7fb848134b70 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fb848136fb0 con 0x7fb848124640 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:00.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient(hunting): authenticate will time out at 1810.986072s 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84db1f640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb848134b70 msgr2=0x7fb848134f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84db1f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb848134b70 0x7fb848134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 0x7fb848124a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb834005c30 0x7fb8481cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 0x7fb848124a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:37318/0 (socket says 172.21.15.43:37318) 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 -- 172.21.15.43:0/1203902054 learned_addr learned my addr 172.21.15.43:0/1203902054 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): get_auth_request con 0x7fb848124640 auth_method 0 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): get_auth_request con 0x7fb834005c30 auth_method 0 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:00.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient(hunting): handle_auth_done global_id 134248 payload 824 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 -- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb848134b70 msgr2=0x7fb848134f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb848134b70 0x7fb848134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 -- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb834005c30 msgr2=0x7fb8481cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb834005c30 0x7fb8481cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient: _finish_hunting 0 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 monclient: found mon.noname-c 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 -- 172.21.15.43:0/1203902054 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8481282d0 con 0x7fb848124640 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84e320640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb834005c30 0x7fb8481cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:59:00.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84eb21640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 0x7fb848124a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb838003850 tx=0x7fb8380094f0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 client.?.objecter ms_handle_connect 0x7fb848124640 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb838003d90 con 0x7fb848124640 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: dump: 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:00.865 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: _finish_auth 0 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: _check_auth_tickets 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb838006070 con 0x7fb848124640 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb838006690 con 0x7fb848124640 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:59:00.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb846ffd640 10 monclient: dump: 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:00.867 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 5 monclient: authenticate success, global_id 134248 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: _renew_subs 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8481cdb50 con 0x7fb848124640 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb84cb1d640 4 set_mon_vals no callback set 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 client.134248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: _renew_subs 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fb84806bc70 con 0x7fb848124640 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.855+0000 7fb850584640 1 librados: init done 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb850584640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb850584640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb850584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8480736e0 con 0x7fb848124640 2024-04-19T00:59:00.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb838010070 con 0x7fb848124640 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb83403d480 0x7fb83403f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(535..535 src has 1..535) ==== 6630+0+0 (secure 0 0 0) 0x7fb838048350 con 0x7fb848124640 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 10 client.134248.objecter ms_dispatch 0x7fb848000ba0 osd_map(535..535 src has 1..535) 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 3 client.134248.objecter handle_osd_map got epochs [535,535] > 0 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 3 client.134248.objecter handle_osd_map decoding full epoch 535 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 20 client.134248.objecter dump_active .. 0 homeless 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb84e320640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb83403d480 0x7fb83403f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb84e320640 10 monclient: get_auth_request con 0x7fb83403d480 auth_method 0 2024-04-19T00:59:00.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb846ffd640 10 client.134248.objecter ms_handle_connect 0x7fb83403d480 2024-04-19T00:59:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.859+0000 7fb84e320640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb83403d480 0x7fb83403f940 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7fb8400b2000 tx=0x7fb8400a9070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.867+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb8380113b0 con 0x7fb848124640 2024-04-19T00:59:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.867+0000 7fb846ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:00.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:00.867+0000 7fb846ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:59:01.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.091+0000 7fb850584640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T00:59:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.091+0000 7fb850584640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T00:59:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.091+0000 7fb850584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.091+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb848071700 con 0x7fb848124640 2024-04-19T00:59:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb846ffd640 1 -- 172.21.15.43:0/1203902054 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3169 (secure 0 0 0) 0x7fb838048080 con 0x7fb848124640 2024-04-19T00:59:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb846ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T00:59:01.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb846ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:59:01.096 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":102,"quorum":[0,3,4,5,7],"quorum_names":["a","g","c","h","i"],"quorum_leader_name":"g","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T00:59:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 10 librados: watch_flush enter 2024-04-19T00:59:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 10 librados: watch_flush exit 2024-04-19T00:59:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 20 client.134248.objecter shutdown clearing up homeless session... 2024-04-19T00:59:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 10 client.134248.objecter successfully canceled tick 2024-04-19T00:59:01.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb83403d480 msgr2=0x7fb83403f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb83403d480 0x7fb83403f940 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7fb8400b2000 tx=0x7fb8400a9070 comp rx=0 tx=0).stop 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 10 monclient: shutdown 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 msgr2=0x7fb848124a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 0x7fb848124a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb838003850 tx=0x7fb8380094f0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 shutdown_connections 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb83403d480 0x7fb83403f940 unknown :-1 s=CLOSED pgs=3576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fb848124640 0x7fb848124a80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb848134b70 0x7fb848134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 --2- 172.21.15.43:0/1203902054 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb834005c30 0x7fb8481cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 >> 172.21.15.43:0/1203902054 conn(0x7fb8480a85a0 msgr2=0x7fb84809d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 shutdown_connections 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 -- 172.21.15.43:0/1203902054 wait complete. 2024-04-19T00:59:01.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.095+0000 7fb850584640 1 librados: shutdown 2024-04-19T00:59:01.115 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-04-19T00:59:01.115 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon a 2024-04-19T00:59:01.116 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T00:59:01.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient: get_monmap_and_config 2024-04-19T00:59:01.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient: build_initial_monmap 2024-04-19T00:59:01.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient: monmap: 2024-04-19T00:59:01.227 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:59:01.230886+0000 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:59:01.230886+0000 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:01.228 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:01.229 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:59:01.229 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:59:01.229 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 Processor -- start 2024-04-19T00:59:01.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 -- start start 2024-04-19T00:59:01.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient: init 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient: _add_conns ranks=[6,9,4,10,8,0,1,7,3,5,2] 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f96a00a6420 0x7f96a00a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): picked mon.noname-d con 0x7f96a00a6420 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f96a00a6800 0x7f96a00a6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): picked mon.noname-a con 0x7f96a00a6800 addr v2:172.21.15.43:3300/0 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 0x7f96a0139000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): picked mon.noname-c con 0x7f96a0138b50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:59:01.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f96a009f340 con 0x7f96a00a6800 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f96a00a20f0 con 0x7f96a0138b50 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f96a009f540 con 0x7f96a00a6420 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f96a4b66640 10 monclient(hunting): authenticate will time out at 1811.357910s 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969e59c640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f96a00a6420 msgr2=0x7f96a00a5bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:59:01.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969e59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f96a00a6420 0x7f96a00a5bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 0x7f96a0139000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f96a00a6800 0x7f96a00a6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 0x7f96a0139000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:45646/0 (socket says 172.21.15.43:45646) 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f96a00a6800 0x7f96a00a6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42262/0 (socket says 172.21.15.43:42262) 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 1 -- 172.21.15.43:0/2661346744 learned_addr learned my addr 172.21.15.43:0/2661346744 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): get_auth_request con 0x7f96a0138b50 auth_method 0 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): get_auth_request con 0x7f96a00a6800 auth_method 0 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:01.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient(hunting): handle_auth_done global_id 65232 payload 274 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 -- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f96a00a6420 msgr2=0x7f96a00a5bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f96a00a6420 0x7f96a00a5bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 -- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 msgr2=0x7f96a0139000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 0x7f96a0139000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient: _finish_hunting 0 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 monclient: found mon.noname-a 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:59:01.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f9695d9b640 1 -- 172.21.15.43:0/2661346744 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a00a07f0 con 0x7f96a00a6800 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.227+0000 7f969ed9d640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 0x7f96a0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f9695d9b640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a00a6800 0x7f96a00a6c20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f96880097c0 tx=0x7f968801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381d206720011147 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 1 -- 172.21.15.43:0/2661346744 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f96880b0070 con 0x7f96a00a6800 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: dump: 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.244 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: _finish_auth 0 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: _check_auth_tickets 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 1 -- 172.21.15.43:0/2661346744 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96880aa810 con 0x7f96a00a6800 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 5 monclient: authenticate success, global_id 65232 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:59:01.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: get_monmap_and_config success 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 4 set_mon_vals no callback set 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 1 -- 172.21.15.43:0/2661346744 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f96880a9300 con 0x7f96a00a6800 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969dd9b640 10 monclient: dump: 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.246 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: shutdown 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a00a6800 msgr2=0x7f96a00a6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a00a6800 0x7f96a00a6c20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f96880097c0 tx=0x7f968801c350 comp rx=0 tx=0).stop 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- 172.21.15.43:0/2661346744 shutdown_connections 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f96a0138b50 0x7f96a0139000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a00a6800 0x7f96a00a6c20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- 172.21.15.43:0/2661346744 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f96a00a6420 0x7f96a00a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- 172.21.15.43:0/2661346744 >> 172.21.15.43:0/2661346744 conn(0x7f96a00ab1b0 msgr2=0x7f96a00ab5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- 172.21.15.43:0/2661346744 shutdown_connections 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- 172.21.15.43:0/2661346744 wait complete. 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: build_initial_monmap 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: monmap: 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:59:01.248 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 librados: starting msgr at 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 librados: starting objecter 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 Processor -- start 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- start start 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 librados: setting wanted keys 2024-04-19T00:59:01.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 librados: calling monclient init 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: init 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient: _add_conns ranks=[8,1,5,4,2,3,0,7,6] 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f96a00a6420 0x7f96a0121c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): picked mon.noname-e con 0x7f96a00a6420 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96a00a6800 0x7f96a013e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): picked mon.noname-f con 0x7f96a00a6800 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f96a0138b50 0x7f96a01d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): picked mon.noname-h con 0x7f96a0138b50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96a0142820 con 0x7f96a00a6800 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f96a01426a0 con 0x7f96a0138b50 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f96a01429a0 con 0x7f96a00a6420 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969e59c640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f96a00a6420 msgr2=0x7f96a0121c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f96a4b66640 10 monclient(hunting): authenticate will time out at 1811.362759s 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969e59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f96a00a6420 0x7f96a0121c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f9695d9b640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96a00a6800 msgr2=0x7f96a013e7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f9695d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96a00a6800 0x7f96a013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f96a0138b50 0x7f96a01d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f96a0138b50 0x7f96a01d0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:49684/0 (socket says 172.21.15.43:49684) 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 -- 172.21.15.43:0/1467988380 learned_addr learned my addr 172.21.15.43:0/1467988380 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): get_auth_request con 0x7f96a0138b50 auth_method 0 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:01.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient(hunting): handle_auth_done global_id 74651 payload 824 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f96a00a6420 msgr2=0x7f96a0121c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f96a00a6420 0x7f96a0121c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96a00a6800 msgr2=0x7f96a013e7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96a00a6800 0x7f96a013e7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient: _finish_hunting 0 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 monclient: found mon.noname-h 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T00:59:01.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.231+0000 7f969ed9d640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a009eef0 con 0x7f96a0138b50 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969ed9d640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f96a0138b50 0x7f96a01d0aa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f969800efc0 tx=0x7f969800c490 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 client.?.objecter ms_handle_connect 0x7f96a0138b50 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 client.?.objecter resend_mon_ops 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f969800cbe0 con 0x7f96a0138b50 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: dump: 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.253 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: _finish_auth 0 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: _check_auth_tickets 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 5 monclient: authenticate success, global_id 74651 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 monclient: _renew_subs 2024-04-19T00:59:01.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a009fd90 con 0x7f96a0138b50 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f969800cd80 con 0x7f96a0138b50 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969d59a640 4 set_mon_vals no callback set 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 client.74651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 monclient: _renew_subs 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a013fa20 con 0x7f96a0138b50 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f96980148f0 con 0x7f96a0138b50 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 1 librados: init done 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: dump: 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.255 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:01.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 0x7f96a01d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f96a006d140 con 0x7f96a013dee0 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 0x7f96a01d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient: get_auth_request con 0x7f96a013dee0 auth_method 0 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 10 monclient(hunting): handle_auth_done global_id 65241 payload 824 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96a00693b0 con 0x7f96a013dee0 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 client.74651.objecter ms_handle_connect 0x7f96a013dee0 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 client.74651.objecter resend_mon_ops 2024-04-19T00:59:01.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f969e59c640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 0x7f96a01d0fe0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9690003990 tx=0x7f969000b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9690016070 con 0x7f96a013dee0 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: dump: 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.258 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f969801b070 con 0x7f96a0138b50 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f96800390b0 0x7f968003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(535..535 src has 1..535) ==== 6630+0+0 (secure 0 0 0) 0x7f9698019020 con 0x7f96a0138b50 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 client.74651.objecter ms_dispatch 0x7f96a0000ba0 osd_map(535..535 src has 1..535) 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 3 client.74651.objecter handle_osd_map got epochs [535,535] > 0 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 3 client.74651.objecter handle_osd_map decoding full epoch 535 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 20 client.74651.objecter dump_active .. 0 homeless 2024-04-19T00:59:01.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9690003bd0 con 0x7f96a013dee0 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 msgr2=0x7f96a01d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 0x7f96a01d0fe0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9690003990 tx=0x7f969000b810 comp rx=0 tx=0).stop 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 msgr2=0x7f96a01d0fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013dee0 0x7f96a01d0fe0 secure :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9690003990 tx=0x7f969000b810 comp rx=0 tx=0).stop 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f9695d9b640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f96800390b0 0x7f968003b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f9695d9b640 10 monclient: get_auth_request con 0x7f96800390b0 auth_method 0 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f96977fe640 10 client.74651.objecter ms_handle_connect 0x7f96800390b0 2024-04-19T00:59:01.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.235+0000 7f9695d9b640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f96800390b0 0x7f968003b570 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f96880ad000 tx=0x7f9688008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 0x7f96a01d1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f96a0068240 con 0x7f96a013ecf0 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 0x7f96a01d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient: get_auth_request con 0x7f96a013ecf0 auth_method 0 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:01.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 10 monclient(hunting): handle_auth_done global_id 65250 payload 824 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 1 -- 172.21.15.43:0/1467988380 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f96a0122ba0 con 0x7f96a013ecf0 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 client.74651.objecter ms_handle_connect 0x7f96a013ecf0 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969e59c640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 0x7f96a01d1520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9690003990 tx=0x7f9690003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 client.74651.objecter resend_mon_ops 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f96900192a0 con 0x7f96a013ecf0 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 monclient: dump: 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.263 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9400 (secure 0 0 0) 0x7f96900192a0 con 0x7f96a013ecf0 2024-04-19T00:59:01.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 msgr2=0x7f96a01d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 0x7f96a01d1520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9690003990 tx=0x7f9690003800 comp rx=0 tx=0).stop 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 msgr2=0x7f96a01d1520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96977fe640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f96a013ecf0 0x7f96a01d1520 secure :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9690003990 tx=0x7f9690003800 comp rx=0 tx=0).stop 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 102, 2024-04-19T00:59:01.265 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 467515, 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:01.266 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:01.267 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:59:01.268 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:59:01.269 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:01.270 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:59:01.272 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.275 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:59:01.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:59:01.280 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:59:01.281 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:59:01.282 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:01.284 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:01.285 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:01.286 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:01.287 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:01.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:01.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:01.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:01.287 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:59:01.287 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 10 librados: watch_flush enter 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 10 librados: watch_flush exit 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 20 client.74651.objecter shutdown clearing up homeless session... 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 10 client.74651.objecter successfully canceled tick 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f96800390b0 msgr2=0x7f968003b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f96800390b0 0x7f968003b570 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f96880ad000 tx=0x7f9688008040 comp rx=0 tx=0).stop 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 10 monclient: shutdown 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f96a0138b50 msgr2=0x7f96a01d0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 --2- 172.21.15.43:0/1467988380 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f96a0138b50 0x7f96a01d0aa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f969800efc0 tx=0x7f969800c490 comp rx=0 tx=0).stop 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969ed9d640 1 -- 172.21.15.43:0/1467988380 reap_dead start 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f969ed9d640 1 -- 172.21.15.43:0/1467988380 reap_dead start 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 shutdown_connections 2024-04-19T00:59:01.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 >> 172.21.15.43:0/1467988380 conn(0x7f96a00ab1b0 msgr2=0x7f96a00aa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:01.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 shutdown_connections 2024-04-19T00:59:01.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 -- 172.21.15.43:0/1467988380 wait complete. 2024-04-19T00:59:01.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.243+0000 7f96a4b66640 1 librados: shutdown 2024-04-19T00:59:01.290 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:59:01.290 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon c 2024-04-19T00:59:01.290 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient: get_monmap_and_config 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient: build_initial_monmap 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient: monmap: 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:59:01.406520+0000 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:59:01.406520+0000 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.403 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:59:01.404 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 Processor -- start 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 -- start start 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient: init 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient: _add_conns ranks=[1,10,7,6,9,5,2,0,3,8,4] 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7080138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70800a5b80 0x7f708013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): picked mon.noname-f con 0x7f70800a5b80 addr v2:172.21.15.161:3300/0 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 0x7f70801407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): picked mon.noname-i con 0x7f70801397c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70800a3dc0 con 0x7f7080138cd0 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f70800a4410 con 0x7f70800a5b80 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f70800a40a0 con 0x7f70801397c0 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7085b30640 10 monclient(hunting): authenticate will time out at 1811.533635s 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707f7fe640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f7080138cd0 legacy=0x7f70801390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7077fff640 1 -- >> v2:172.21.15.161:3300/0 conn(0x7f70800a5b80 msgr2=0x7f708013e270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f7077fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f70800a5b80 0x7f708013e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 0x7f70801407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 0x7f70801407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:41258/0 (socket says 172.21.15.43:41258) 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 -- 172.21.15.43:0/3726822287 learned_addr learned my addr 172.21.15.43:0/3726822287 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): get_auth_request con 0x7f70801397c0 auth_method 0 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:01.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient(hunting): handle_auth_done global_id 74545 payload 274 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 -- 172.21.15.43:0/3726822287 >> v2:172.21.15.161:3300/0 conn(0x7f70800a5b80 msgr2=0x7f708013e270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 --2- 172.21.15.43:0/3726822287 >> v2:172.21.15.161:3300/0 conn(0x7f70800a5b80 0x7f708013e270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 -- 172.21.15.43:0/3726822287 >> v1:172.21.15.161:6789/0 conn(0x7f7080138cd0 legacy=0x7f70801390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 monclient: found mon.noname-i 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 10 monclient: _send_mon_message to mon.noname-i at v2:172.21.15.161:3303/0 2024-04-19T00:59:01.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.403+0000 7f707ffff640 1 -- 172.21.15.43:0/3726822287 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70800a2ae0 con 0x7f70801397c0 2024-04-19T00:59:01.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707ffff640 1 --2- 172.21.15.43:0/3726822287 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 0x7f70801407c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70800a7360 tx=0x7f707801c8f0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=1c16d397cce05067 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:01.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 1 -- 172.21.15.43:0/3726822287 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70800a20a0 con 0x7f70801397c0 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: got monmap 1 from mon.noname-i (according to old e1) 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: dump: 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.412 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: _finish_auth 0 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: _check_auth_tickets 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 1 -- 172.21.15.43:0/3726822287 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70780090d0 con 0x7f70801397c0 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 1 -- 172.21.15.43:0/3726822287 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70780a5bc0 con 0x7f70801397c0 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:59:01.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707effd640 10 monclient: dump: 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:01.414 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 5 monclient: authenticate success, global_id 74545 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: get_monmap_and_config success 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 4 set_mon_vals no callback set 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: shutdown 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- 172.21.15.43:0/3726822287 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 msgr2=0x7f70801407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 --2- 172.21.15.43:0/3726822287 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 0x7f70801407c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70800a7360 tx=0x7f707801c8f0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- 172.21.15.43:0/3726822287 shutdown_connections 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 --2- 172.21.15.43:0/3726822287 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f70801397c0 0x7f70801407c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 --2- 172.21.15.43:0/3726822287 >> v2:172.21.15.161:3300/0 conn(0x7f70800a5b80 0x7f708013e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- 172.21.15.43:0/3726822287 >> 172.21.15.43:0/3726822287 conn(0x7f70800ab2f0 msgr2=0x7f70800ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- 172.21.15.43:0/3726822287 shutdown_connections 2024-04-19T00:59:01.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- 172.21.15.43:0/3726822287 wait complete. 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: build_initial_monmap 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: monmap: 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:01.416 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 librados: starting msgr at 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 librados: starting objecter 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 Processor -- start 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- start start 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 librados: setting wanted keys 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 librados: calling monclient init 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: init 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient: _add_conns ranks=[2,8,6,4,1,7,0,5,3] 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 0x7f708009f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): picked mon.noname-b con 0x7f70800a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:59:01.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 0x7f708009ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): picked mon.noname-e con 0x7f7080138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 0x7f708009bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): picked mon.noname-d con 0x7f70801397c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f70801438b0 con 0x7f70800a5b80 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f7080143730 con 0x7f70801397c0 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7080143a30 con 0x7f7080138cd0 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7085b30640 10 monclient(hunting): authenticate will time out at 1811.539071s 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7077fff640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 msgr2=0x7f708009ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707f7fe640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 msgr2=0x7f708009f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f7077fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 0x7f708009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707ffff640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 msgr2=0x7f708009bfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:59:01.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 0x7f708009f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.407+0000 7f707ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 0x7f708009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.611+0000 7f7077fff640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 msgr2=0x7f708009ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.611+0000 7f707ffff640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 msgr2=0x7f708009bfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:59:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.611+0000 7f7077fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 0x7f708009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:59:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.611+0000 7f707ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 0x7f708009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:59:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.611+0000 7f707f7fe640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 msgr2=0x7f708009f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:01.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:01.611+0000 7f707f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 0x7f708009f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T00:59:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.015+0000 7f7077fff640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 msgr2=0x7f708009ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.015+0000 7f707ffff640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 msgr2=0x7f708009bfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:59:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.015+0000 7f707f7fe640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 msgr2=0x7f708009f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.015+0000 7f7077fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 0x7f708009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:59:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.015+0000 7f707ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 0x7f708009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:59:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.015+0000 7f707f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 0x7f708009f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T00:59:02.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.819+0000 7f7077fff640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 msgr2=0x7f708009ba70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:02.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.819+0000 7f707ffff640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 msgr2=0x7f708009bfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:59:02.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.819+0000 7f707f7fe640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 msgr2=0x7f708009f5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:02.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.819+0000 7f7077fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 0x7f708009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:59:02.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.819+0000 7f707ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 0x7f708009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:59:02.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:02.819+0000 7f707f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 0x7f708009f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T00:59:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): tick 2024-04-19T00:59:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-19T00:59:04.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 monclient(hunting): continuing hunt 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 msgr2=0x7f708009ba70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7080138cd0 0x7f708009ba70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 msgr2=0x7f708009bfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f70801397c0 0x7f708009bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 msgr2=0x7f708009f5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70800a5b80 0x7f708009f5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient: _add_conns ranks=[5,0,7,3,1,4] 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f70640037d0 0x7f7064005be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): picked mon.noname-h con 0x7f70640037d0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 0x7f7064008670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f70640061e0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:59:04.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7064008d10 0x7f706400b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): picked mon.noname-i con 0x7f7064008d10 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f70801438b0 con 0x7f70640061e0 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7080143730 con 0x7f70640037d0 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7080143a30 con 0x7f7064008d10 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7076ffd640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.407+0000 7f7077fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 0x7f7064008670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 0x7f7064008670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42278/0 (socket says 172.21.15.43:42278) 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 -- 172.21.15.43:0/2233433475 learned_addr learned my addr 172.21.15.43:0/2233433475 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): get_auth_request con 0x7f70640061e0 auth_method 0 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7064008d10 0x7f706400b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f70640037d0 0x7f7064005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): get_auth_request con 0x7f7064008d10 auth_method 0 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): get_auth_request con 0x7f70640037d0 auth_method 0 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient(hunting): handle_auth_done global_id 65259 payload 824 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7064008d10 msgr2=0x7f706400b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7064008d10 0x7f706400b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f70640037d0 msgr2=0x7f7064005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f70640037d0 0x7f7064005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient: _finish_hunting 0 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 monclient: found mon.noname-a 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:04.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708009eea0 con 0x7f70640061e0 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7064008d10 0x7f706400b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707ffff640 1 -- 172.21.15.43:0/2233433475 reap_dead start 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7077fff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 0x7f7064008670 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f707000b9e0 tx=0x7f707000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 client.?.objecter ms_handle_connect 0x7f70640061e0 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f707000ce10 con 0x7f70640061e0 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: dump: 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.418 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: _finish_auth 0 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: _check_auth_tickets 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:04.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7070010070 con 0x7f70640061e0 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70700158e0 con 0x7f70640061e0 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707cff9640 10 monclient: dump: 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.420 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f707dffb640 4 set_mon_vals no callback set 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 5 monclient: authenticate success, global_id 65259 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 monclient: _renew_subs 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708009ceb0 con 0x7f70640061e0 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 client.65259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 monclient: _renew_subs 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708009e3b0 con 0x7f70640061e0 2024-04-19T00:59:04.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 1 librados: init done 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 0x7f708009ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.411+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f70800a3040 con 0x7f708009f560 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 0x7f708009ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient: get_auth_request con 0x7f708009f560 auth_method 0 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7070015470 con 0x7f70640061e0 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 10 monclient(hunting): handle_auth_done global_id 134266 payload 824 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7080068240 con 0x7f708009f560 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7060038d80 0x7f706003b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(538..538 src has 1..538) ==== 6625+0+0 (secure 0 0 0) 0x7f70700492f0 con 0x7f70640061e0 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 client.65259.objecter ms_dispatch 0x7f7080000ba0 osd_map(538..538 src has 1..538) 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 3 client.65259.objecter handle_osd_map got epochs [538,538] > 0 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 3 client.65259.objecter handle_osd_map decoding full epoch 538 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 20 client.65259.objecter dump_active .. 0 homeless 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 client.65259.objecter ms_handle_connect 0x7f708009f560 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 client.65259.objecter resend_mon_ops 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707f7fe640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 0x7f708009ba70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7068003a60 tx=0x7f706800be70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707ffff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7060038d80 0x7f706003b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707ffff640 10 monclient: get_auth_request con 0x7f7060038d80 auth_method 0 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f706801a070 con 0x7f708009f560 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 monclient: dump: 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.428 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7068003cf0 con 0x7f708009f560 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 msgr2=0x7f708009ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 0x7f708009ba70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7068003a60 tx=0x7f706800be70 comp rx=0 tx=0).stop 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 msgr2=0x7f708009ba70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 0x7f708009ba70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707ffff640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7060038d80 0x7f706003b240 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7f70780ae000 tx=0x7f70780a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.415+0000 7f707cff9640 10 client.65259.objecter ms_handle_connect 0x7f7060038d80 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f7085b30640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 0x7f708009c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f7085b30640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f708006d140 con 0x7f708009bfb0 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 0x7f708009c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient: get_auth_request con 0x7f708009bfb0 auth_method 0 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 10 monclient(hunting): handle_auth_done global_id 134275 payload 824 2024-04-19T00:59:04.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 1 -- 172.21.15.43:0/2233433475 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f708009c8d0 con 0x7f708009bfb0 2024-04-19T00:59:04.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707f7fe640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 0x7f708009c390 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7068000c10 tx=0x7f7068005b70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707cff9640 10 client.65259.objecter ms_handle_connect 0x7f708009bfb0 2024-04-19T00:59:04.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707cff9640 10 client.65259.objecter resend_mon_ops 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f706801a070 con 0x7f708009bfb0 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.423+0000 7f707cff9640 10 monclient: dump: 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.432 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9399 (secure 0 0 0) 0x7f7068005df0 con 0x7f708009bfb0 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 msgr2=0x7f708009c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 0x7f708009c390 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7068000c10 tx=0x7f7068005b70 comp rx=0 tx=0).stop 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 msgr2=0x7f708009c390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f707cff9640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 0x7f708009c390 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7068000c10 tx=0x7f7068005b70 comp rx=0 tx=0).stop 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 102, 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 49127, 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:59:04.434 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:04.435 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:04.436 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:59:04.437 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:04.438 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.440 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:59:04.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:59:04.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:59:04.447 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.450 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:59:04.451 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.452 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:59:04.453 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.454 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.455 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.456 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:59:04.456 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:59:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 10 librados: watch_flush enter 2024-04-19T00:59:04.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 10 librados: watch_flush exit 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 20 client.65259.objecter shutdown clearing up homeless session... 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 10 client.65259.objecter successfully canceled tick 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7060038d80 msgr2=0x7f706003b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7060038d80 0x7f706003b240 secure :-1 s=READY pgs=3579 cs=0 l=1 rev1=1 crypto rx=0x7f70780ae000 tx=0x7f70780a7070 comp rx=0 tx=0).stop 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 10 monclient: shutdown 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 msgr2=0x7f7064008670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 0x7f7064008670 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f707000b9e0 tx=0x7f707000bea0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 shutdown_connections 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7060038d80 0x7f706003b240 unknown :-1 s=CLOSED pgs=3579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70640061e0 0x7f7064008670 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009bfb0 0x7f708009c390 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 --2- 172.21.15.43:0/2233433475 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f708009f560 0x7f708009ba70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 >> 172.21.15.43:0/2233433475 conn(0x7f70800ab2f0 msgr2=0x7f7080131e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:04.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 shutdown_connections 2024-04-19T00:59:04.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 -- 172.21.15.43:0/2233433475 wait complete. 2024-04-19T00:59:04.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.427+0000 7f7085b30640 1 librados: shutdown 2024-04-19T00:59:04.459 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:59:04.459 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-04-19T00:59:04.459 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T00:59:04.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient: get_monmap_and_config 2024-04-19T00:59:04.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient: build_initial_monmap 2024-04-19T00:59:04.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient: monmap: 2024-04-19T00:59:04.588 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:59:04.591323+0000 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:59:04.591323+0000 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:04.589 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:04.590 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:59:04.591 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:59:04.591 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 Processor -- start 2024-04-19T00:59:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 -- start start 2024-04-19T00:59:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient: init 2024-04-19T00:59:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:04.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient: _add_conns ranks=[9,0,4,6,2,8,1,5,3,7,10] 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08c4138cd0 0x7f08c41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): picked mon.noname-a con 0x7f08c4138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08c40a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08c41395f0 0x7f08c4135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): picked mon.noname-c con 0x7f08c41395f0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08c40a3dc0 con 0x7f08c40a5b80 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f08c40a4410 con 0x7f08c4138cd0 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f08c40a40a0 con 0x7f08c41395f0 2024-04-19T00:59:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08cab78640 10 monclient(hunting): authenticate will time out at 1814.718441s 2024-04-19T00:59:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f08c4138cd0 0x7f08c41390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c3fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f08c40a5b80 0x7f08c41358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45864/0 (socket says 172.21.15.43:45864) 2024-04-19T00:59:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c3fff640 1 -- 172.21.15.43:0/747333563 learned_addr learned my addr 172.21.15.43:0/747333563 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T00:59:04.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): get_auth_request con 0x7f08c4138cd0 auth_method 0 2024-04-19T00:59:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08c41395f0 0x7f08c4135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): get_auth_request con 0x7f08c41395f0 auth_method 0 2024-04-19T00:59:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c9115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient(hunting): handle_auth_done global_id 65268 payload 274 2024-04-19T00:59:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 1 -- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08c41395f0 msgr2=0x7f08c4135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:59:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 1 --2- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08c41395f0 0x7f08c4135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 1 -- 172.21.15.43:0/747333563 >> v1:172.21.15.43:6789/0 conn(0x7f08c40a5b80 legacy=0x7f08c41358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient: _finish_hunting 0 2024-04-19T00:59:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 1 monclient: found mon.noname-a 2024-04-19T00:59:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c37fe640 1 -- 172.21.15.43:0/747333563 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348927386 0 0) 0x7f08c40a3dc0 con 0x7f08c40a5b80 2024-04-19T00:59:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c8914640 1 -- 172.21.15.43:0/747333563 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c40a2ae0 con 0x7f08c4138cd0 2024-04-19T00:59:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.587+0000 7f08c37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T00:59:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 --2- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4138cd0 0x7f08c41390b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f08b40097c0 tx=0x7f08b401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac6e425c43e2493 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 1 -- 172.21.15.43:0/747333563 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08b40b2070 con 0x7f08c4138cd0 2024-04-19T00:59:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:59:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: dump: 2024-04-19T00:59:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.610 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.611 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: _finish_auth 0 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: _check_auth_tickets 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 5 monclient: authenticate success, global_id 65268 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 1 -- 172.21.15.43:0/747333563 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b40ab7f0 con 0x7f08c4138cd0 2024-04-19T00:59:04.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: get_monmap_and_config success 2024-04-19T00:59:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 4 set_mon_vals no callback set 2024-04-19T00:59:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 1 -- 172.21.15.43:0/747333563 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08b40abe10 con 0x7f08c4138cd0 2024-04-19T00:59:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:04.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c37fe640 10 monclient: dump: 2024-04-19T00:59:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.614 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.615 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.616 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: shutdown 2024-04-19T00:59:04.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4138cd0 msgr2=0x7f08c41390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 --2- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4138cd0 0x7f08c41390b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f08b40097c0 tx=0x7f08b401c360 comp rx=0 tx=0).stop 2024-04-19T00:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- 172.21.15.43:0/747333563 shutdown_connections 2024-04-19T00:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 --2- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f08c41395f0 0x7f08c4135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 --2- 172.21.15.43:0/747333563 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c4138cd0 0x7f08c41390b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- 172.21.15.43:0/747333563 >> 172.21.15.43:0/747333563 conn(0x7f08c40ab2f0 msgr2=0x7f08c40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- 172.21.15.43:0/747333563 shutdown_connections 2024-04-19T00:59:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- 172.21.15.43:0/747333563 wait complete. 2024-04-19T00:59:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: build_initial_monmap 2024-04-19T00:59:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: monmap: 2024-04-19T00:59:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:59:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 librados: starting msgr at 2024-04-19T00:59:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 librados: starting objecter 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 Processor -- start 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- start start 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 librados: setting wanted keys 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 librados: calling monclient init 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: init 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient: _add_conns ranks=[0,2,7,5,6,1,8,3,4] 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 0x7f08c409cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): picked mon.noname-a con 0x7f08c40a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T00:59:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f08c4138cd0 0x7f08c409d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): picked mon.noname-b con 0x7f08c4138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:59:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f08c41395f0 0x7f08c409d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): picked mon.noname-i con 0x7f08c41395f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T00:59:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f08c4136500 con 0x7f08c40a5b80 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f08c40a20a0 con 0x7f08c4138cd0 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f08c4136680 con 0x7f08c41395f0 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08cab78640 10 monclient(hunting): authenticate will time out at 1814.723533s 2024-04-19T00:59:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c3fff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f08c4138cd0 msgr2=0x7f08c409d000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f08c4138cd0 0x7f08c409d000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 0x7f08c409cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 0x7f08c409cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42292/0 (socket says 172.21.15.43:42292) 2024-04-19T00:59:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 -- 172.21.15.43:0/1205705249 learned_addr learned my addr 172.21.15.43:0/1205705249 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): get_auth_request con 0x7f08c40a5b80 auth_method 0 2024-04-19T00:59:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient(hunting): handle_auth_done global_id 65286 payload 824 2024-04-19T00:59:04.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c9115640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f08c41395f0 0x7f08c409d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f08c41395f0 msgr2=0x7f08c409d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f08c41395f0 0x7f08c409d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f08c4138cd0 msgr2=0x7f08c409d000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:04.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f08c4138cd0 0x7f08c409d000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 10 monclient: _finish_hunting 0 2024-04-19T00:59:04.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 1 monclient: found mon.noname-a 2024-04-19T00:59:04.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.591+0000 7f08c8914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:04.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c8914640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c8914640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c409ff20 con 0x7f08c40a5b80 2024-04-19T00:59:04.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c8914640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 0x7f08c409cac0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08b40b0040 tx=0x7f08b4019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 client.?.objecter ms_handle_connect 0x7f08c40a5b80 2024-04-19T00:59:04.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:59:04.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08b40ba020 con 0x7f08c40a5b80 2024-04-19T00:59:04.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:59:04.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: dump: 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.629 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.630 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: _finish_auth 0 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: _check_auth_tickets 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08b40044b0 con 0x7f08c40a5b80 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 5 monclient: authenticate success, global_id 65286 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08b40b2040 con 0x7f08c40a5b80 2024-04-19T00:59:04.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 monclient: _renew_subs 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c40a04e0 con 0x7f08c40a5b80 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c2ffd640 4 set_mon_vals no callback set 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 client.65286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: dump: 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.632 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.633 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.634 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.634 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.634 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.634 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.634 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.634 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 monclient: _renew_subs 2024-04-19T00:59:04.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T00:59:04.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c40a3040 con 0x7f08c40a5b80 2024-04-19T00:59:04.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 1 librados: init done 2024-04-19T00:59:04.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:04.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 0x7f08c412ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f08c406d140 con 0x7f08c412e840 2024-04-19T00:59:04.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f08b40bd020 con 0x7f08c40a5b80 2024-04-19T00:59:04.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f08b0038e10 0x7f08b003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(538..538 src has 1..538) ==== 6625+0+0 (secure 0 0 0) 0x7f08b40e3040 con 0x7f08c40a5b80 2024-04-19T00:59:04.636 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 client.65286.objecter ms_dispatch 0x7f08c4000ba0 osd_map(538..538 src has 1..538) 2024-04-19T00:59:04.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 3 client.65286.objecter handle_osd_map got epochs [538,538] > 0 2024-04-19T00:59:04.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 3 client.65286.objecter handle_osd_map decoding full epoch 538 2024-04-19T00:59:04.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 0x7f08c412ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient: get_auth_request con 0x7f08c412e840 auth_method 0 2024-04-19T00:59:04.637 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 20 client.65286.objecter dump_active .. 0 homeless 2024-04-19T00:59:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c9115640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f08b0038e10 0x7f08b003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.638 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c9115640 10 monclient: get_auth_request con 0x7f08b0038e10 auth_method 0 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 10 monclient(hunting): handle_auth_done global_id 19722 payload 824 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08c40693b0 con 0x7f08c412e840 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c3fff640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 0x7f08c412ec20 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f08b800c3a0 tx=0x7f08b800c860 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 client.65286.objecter ms_handle_connect 0x7f08c412e840 2024-04-19T00:59:04.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 client.65286.objecter resend_mon_ops 2024-04-19T00:59:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 client.65286.objecter ms_handle_connect 0x7f08b0038e10 2024-04-19T00:59:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c9115640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f08b0038e10 0x7f08b003b2d0 secure :-1 s=READY pgs=3580 cs=0 l=1 rev1=1 crypto rx=0x7f08c4136210 tx=0x7f08bc00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08b8017070 con 0x7f08c412e840 2024-04-19T00:59:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:59:04.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: dump: 2024-04-19T00:59:04.641 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.641 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.641 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.641 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.641 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.641 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.642 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.642 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.642 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.642 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.642 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.642 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f08b8005af0 con 0x7f08c412e840 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 msgr2=0x7f08c412ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 0x7f08c412ec20 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f08b800c3a0 tx=0x7f08b800c860 comp rx=0 tx=0).stop 2024-04-19T00:59:04.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 msgr2=0x7f08c412ec20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:04.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.595+0000 7f08c17fa640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412e840 0x7f08c412ec20 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08cab78640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:59:04.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08cab78640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 0x7f08c4132f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08cab78640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f08c4068240 con 0x7f08c412f160 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 0x7f08c4132f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient: get_auth_request con 0x7f08c412f160 auth_method 0 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 10 monclient(hunting): handle_auth_done global_id 19731 payload 824 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 1 -- 172.21.15.43:0/1205705249 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f08c409e860 con 0x7f08c412f160 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c3fff640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 0x7f08c4132f50 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f08b8000fc0 tx=0x7f08b8000bd0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c17fa640 10 client.65286.objecter ms_handle_connect 0x7f08c412f160 2024-04-19T00:59:04.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c17fa640 10 client.65286.objecter resend_mon_ops 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f08b8017040 con 0x7f08c412f160 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c17fa640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.603+0000 7f08c17fa640 10 monclient: dump: 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.647 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9402 (secure 0 0 0) 0x7f08b8012400 con 0x7f08c412f160 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 msgr2=0x7f08c4132f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 0x7f08c4132f50 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f08b8000fc0 tx=0x7f08b8000bd0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 msgr2=0x7f08c4132f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:04.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c17fa640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f08c412f160 0x7f08c4132f50 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 102, 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 888093, 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:59:04.649 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:04.650 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:04.651 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:04.652 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:04.653 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:04.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:04.654 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:04.654 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:04.654 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:04.654 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:04.655 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:04.655 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:04.655 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.655 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:59:04.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.655 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:59:04.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.656 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:59:04.656 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:59:04.656 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.657 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:59:04.657 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.657 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:59:04.658 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.658 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.658 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:59:04.659 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:59:04.659 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.659 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.659 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:59:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:59:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:59:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:59:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.662 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:59:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:59:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.663 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:59:04.666 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.666 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.666 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:59:04.666 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:59:04.667 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.667 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.667 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.667 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:59:04.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.672 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.673 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.674 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.675 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.676 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:59:04.677 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.678 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.679 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:04.680 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:04.681 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:04.682 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:59:04.682 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 10 librados: watch_flush enter 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 10 librados: watch_flush exit 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 20 client.65286.objecter shutdown clearing up homeless session... 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 10 client.65286.objecter successfully canceled tick 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f08b0038e10 msgr2=0x7f08b003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f08b0038e10 0x7f08b003b2d0 secure :-1 s=READY pgs=3580 cs=0 l=1 rev1=1 crypto rx=0x7f08c4136210 tx=0x7f08bc00b040 comp rx=0 tx=0).stop 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 10 monclient: shutdown 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:04.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 msgr2=0x7f08c409cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 0x7f08c409cac0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08b40b0040 tx=0x7f08b4019100 comp rx=0 tx=0).stop 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08c9115640 1 -- 172.21.15.43:0/1205705249 reap_dead start 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 shutdown_connections 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 --2- 172.21.15.43:0/1205705249 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f08c40a5b80 0x7f08c409cac0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 >> 172.21.15.43:0/1205705249 conn(0x7f08c40ab2f0 msgr2=0x7f08c4131c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 shutdown_connections 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 -- 172.21.15.43:0/1205705249 wait complete. 2024-04-19T00:59:04.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.607+0000 7f08cab78640 1 librados: shutdown 2024-04-19T00:59:04.685 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:59:04.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-04-19T00:59:04.686 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T00:59:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient: get_monmap_and_config 2024-04-19T00:59:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient: build_initial_monmap 2024-04-19T00:59:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient: monmap: 2024-04-19T00:59:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.821 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:59:04.824005+0000 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:59:04.824005+0000 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:04.822 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:59:04.823 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 Processor -- start 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 -- start start 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient: init 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient: _add_conns ranks=[2,1,4,7,5,0,3,6,8,9,10] 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff530138a50 0x7ff530138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient(hunting): picked mon.noname-b con 0x7ff530138a50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T00:59:04.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff5300a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 0x7ff530138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient(hunting): picked mon.noname-c con 0x7ff530139370 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5300a0d40 con 0x7ff5300a5bb0 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff5300a1020 con 0x7ff530138a50 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff5300a0780 con 0x7ff530139370 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff536943640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.819+0000 7ff52ffff640 1 -- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff530138a50 msgr2=0x7ff530138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3301/0 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 10 monclient(hunting): authenticate will time out at 1814.952016s 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff530138a50 0x7ff530138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52f7fe640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7ff5300a5bb0 legacy=0x7ff530137960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 0x7ff530138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 0x7ff530138070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:45678/0 (socket says 172.21.15.43:45678) 2024-04-19T00:59:04.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 -- 172.21.15.43:0/3963125620 learned_addr learned my addr 172.21.15.43:0/3963125620 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): get_auth_request con 0x7ff530139370 auth_method 0 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient(hunting): handle_auth_done global_id 134293 payload 274 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 -- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff530138a50 msgr2=0x7ff530138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 --2- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff530138a50 0x7ff530138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 -- 172.21.15.43:0/3963125620 >> v1:172.21.15.161:6789/0 conn(0x7ff5300a5bb0 legacy=0x7ff530137960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient: _finish_hunting 0 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 monclient: found mon.noname-c 2024-04-19T00:59:04.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 -- 172.21.15.43:0/3963125620 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5300a0a60 con 0x7ff530139370 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff534ee0640 1 --2- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 0x7ff530138070 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff528009e80 tx=0x7ff52801c8d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=7eb2167740fb6f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 1 -- 172.21.15.43:0/3963125620 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff5280090d0 con 0x7ff530139370 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: dump: 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.837 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: _finish_auth 0 2024-04-19T00:59:04.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: _check_auth_tickets 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 1 -- 172.21.15.43:0/3963125620 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5280a5a40 con 0x7ff530139370 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 1 -- 172.21.15.43:0/3963125620 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff5280a5c30 con 0x7ff530139370 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 5 monclient: authenticate success, global_id 134293 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 10 monclient: get_monmap_and_config success 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 4 set_mon_vals no callback set 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff52effd640 10 monclient: dump: 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.839 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 10 monclient: shutdown 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 -- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 msgr2=0x7ff530138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 --2- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 0x7ff530138070 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff528009e80 tx=0x7ff52801c8d0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 -- 172.21.15.43:0/3963125620 shutdown_connections 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 --2- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff530139370 0x7ff530138070 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 --2- 172.21.15.43:0/3963125620 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff530138a50 0x7ff530138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 -- 172.21.15.43:0/3963125620 >> 172.21.15.43:0/3963125620 conn(0x7ff5300a7cb0 msgr2=0x7ff5300a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 -- 172.21.15.43:0/3963125620 shutdown_connections 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.823+0000 7ff536943640 1 -- 172.21.15.43:0/3963125620 wait complete. 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: build_initial_monmap 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: monmap: 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:59:04.841 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 librados: starting msgr at 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 librados: starting objecter 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 Processor -- start 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- start start 2024-04-19T00:59:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 librados: setting wanted keys 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 librados: calling monclient init 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: init 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: _add_conns ranks=[4,1,6,3,0,2,7,5,8] 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff5300a5bb0 0x7ff530130b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): picked mon.noname-c con 0x7ff5300a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff530138a50 0x7ff5301310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): picked mon.noname-f con 0x7ff530138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff530139370 0x7ff530135090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): picked mon.noname-d con 0x7ff530139370 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff530138730 con 0x7ff530138a50 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff5301385b0 con 0x7ff5300a5bb0 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ff530143820 con 0x7ff530139370 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): authenticate will time out at 1814.957227s 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff534ee0640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff530139370 msgr2=0x7ff530135090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff534ee0640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff530139370 0x7ff530135090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff5300a5bb0 0x7ff530130b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff5300a5bb0 0x7ff530130b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:45688/0 (socket says 172.21.15.43:45688) 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 -- 172.21.15.43:0/1485164234 learned_addr learned my addr 172.21.15.43:0/1485164234 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): get_auth_request con 0x7ff5300a5bb0 auth_method 0 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff530138a50 msgr2=0x7ff5301310c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:59:04.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff530138a50 0x7ff5301310c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient(hunting): handle_auth_done global_id 134302 payload 824 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff530139370 msgr2=0x7ff530135090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff530139370 0x7ff530135090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff530138a50 msgr2=0x7ff5301310c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff530138a50 0x7ff5301310c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient: _finish_hunting 0 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 monclient: found mon.noname-c 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:04.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53009ee60 con 0x7ff5300a5bb0 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52ffff640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff5300a5bb0 0x7ff530130b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff52000b580 tx=0x7ff52000ba40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 client.?.objecter ms_handle_connect 0x7ff5300a5bb0 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 client.?.objecter resend_mon_ops 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff520016070 con 0x7ff5300a5bb0 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: dump: 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.846 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: _finish_auth 0 2024-04-19T00:59:04.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: _check_auth_tickets 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff520003bd0 con 0x7ff5300a5bb0 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff520003d80 con 0x7ff5300a5bb0 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 monclient: dump: 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:04.848 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52dffb640 4 set_mon_vals no callback set 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 5 monclient: authenticate success, global_id 134302 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: _renew_subs 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff530136b90 con 0x7ff5300a5bb0 2024-04-19T00:59:04.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 client.134302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: _renew_subs 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff530139ee0 con 0x7ff5300a5bb0 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 librados: init done 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ff53006d140 con 0x7ff5301376f0 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff5200050d0 con 0x7ff5300a5bb0 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff51c038e10 0x7ff51c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(539..539 src has 1..539) ==== 6625+0+0 (secure 0 0 0) 0x7ff520048800 con 0x7ff5300a5bb0 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 10 client.134302.objecter ms_dispatch 0x7ff530000ba0 osd_map(539..539 src has 1..539) 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 3 client.134302.objecter handle_osd_map got epochs [539,539] > 0 2024-04-19T00:59:04.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 3 client.134302.objecter handle_osd_map decoding full epoch 539 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52cff9640 20 client.134302.objecter dump_active .. 0 homeless 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient: get_auth_request con 0x7ff5301376f0 auth_method 0 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff534ee0640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff51c038e10 0x7ff51c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff534ee0640 10 monclient: get_auth_request con 0x7ff51c038e10 auth_method 0 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.827+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_done global_id 74669 payload 824 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52f7fe640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5300008e0 con 0x7ff5301376f0 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52f7fe640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 msgr2=0x7ff530135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T00:59:04.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52f7fe640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 msgr2=0x7ff530135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T00:59:04.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff53006e8c0 tx=0x7ff52400b830 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T00:59:04.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:04.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff52cff9640 10 client.134302.objecter ms_handle_connect 0x7ff51c038e10 2024-04-19T00:59:04.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:04.831+0000 7ff534ee0640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff51c038e10 0x7ff51c03b2d0 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7ff5280ae000 tx=0x7ff5280a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: get_auth_request con 0x7ff5301376f0 auth_method 0 2024-04-19T00:59:05.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: _init_auth method 2 2024-04-19T00:59:05.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: _init_auth already have auth, reseting 2024-04-19T00:59:05.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: handle_auth_reply_more payload 9 2024-04-19T00:59:05.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: handle_auth_reply_more payload_len 9 2024-04-19T00:59:05.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-04-19T00:59:05.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 10 monclient: handle_auth_done global_id 74669 payload 293 2024-04-19T00:59:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff52400bfd0 tx=0x7ff524009190 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 client.134302.objecter ms_handle_connect 0x7ff5301376f0 2024-04-19T00:59:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 client.134302.objecter resend_mon_ops 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff524003ba0 con 0x7ff5301376f0 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 monclient: dump: 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff524003b20 con 0x7ff5301376f0 2024-04-19T00:59:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:59:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 msgr2=0x7ff530135d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff52400bfd0 tx=0x7ff524009190 comp rx=0 tx=0).stop 2024-04-19T00:59:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 msgr2=0x7ff530135d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:05.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.035+0000 7ff52cff9640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff5301376f0 0x7ff530135d90 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff52400bfd0 tx=0x7ff524009190 comp rx=0 tx=0).stop 2024-04-19T00:59:05.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff536943640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:59:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff536943640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 0x7ff53002c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff536943640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ff53009d180 con 0x7ff53002c450 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 0x7ff53002c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient: get_auth_request con 0x7ff53002c450 auth_method 0 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.043+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52f7fe640 10 monclient(hunting): handle_auth_done global_id 74678 payload 824 2024-04-19T00:59:05.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52f7fe640 1 -- 172.21.15.43:0/1485164234 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff530140380 con 0x7ff53002c450 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52f7fe640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 0x7ff53002c860 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff52400bfd0 tx=0x7ff5240059d0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 client.134302.objecter ms_handle_connect 0x7ff53002c450 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 client.134302.objecter resend_mon_ops 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff524005d80 con 0x7ff53002c450 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 monclient: dump: 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.048 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.049 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9230 (secure 0 0 0) 0x7ff524005d80 con 0x7ff53002c450 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 msgr2=0x7ff53002c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 0x7ff53002c860 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff52400bfd0 tx=0x7ff5240059d0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 msgr2=0x7ff53002c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:05.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff52cff9640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ff53002c450 0x7ff53002c860 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 102, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 324666, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.051 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:05.052 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.053 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:59:05.054 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.055 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:59:05.056 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.057 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.058 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.059 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.060 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.061 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:59:05.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.063 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.064 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:59:05.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.067 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:05.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:59:05.070 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:59:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 10 librados: watch_flush enter 2024-04-19T00:59:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 10 librados: watch_flush exit 2024-04-19T00:59:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 20 client.134302.objecter shutdown clearing up homeless session... 2024-04-19T00:59:05.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 10 client.134302.objecter successfully canceled tick 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff51c038e10 msgr2=0x7ff51c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff51c038e10 0x7ff51c03b2d0 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7ff5280ae000 tx=0x7ff5280a7070 comp rx=0 tx=0).stop 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 10 monclient: shutdown 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff5300a5bb0 msgr2=0x7ff530130b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 --2- 172.21.15.43:0/1485164234 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff5300a5bb0 0x7ff530130b80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff52000b580 tx=0x7ff52000ba40 comp rx=0 tx=0).stop 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff534ee0640 1 -- 172.21.15.43:0/1485164234 reap_dead start 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff534ee0640 1 -- 172.21.15.43:0/1485164234 reap_dead start 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 shutdown_connections 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 >> 172.21.15.43:0/1485164234 conn(0x7ff5300a7cb0 msgr2=0x7ff53009e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 shutdown_connections 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 -- 172.21.15.43:0/1485164234 wait complete. 2024-04-19T00:59:05.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.047+0000 7ff536943640 1 librados: shutdown 2024-04-19T00:59:05.075 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:59:05.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon i 2024-04-19T00:59:05.075 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.203+0000 7fdb42ecf640 10 monclient: get_monmap_and_config 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.203+0000 7fdb42ecf640 10 monclient: build_initial_monmap 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.203+0000 7fdb42ecf640 10 monclient: monmap: 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:59:05.210319+0000 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:59:05.210319+0000 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T00:59:05.207 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T00:59:05.208 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 Processor -- start 2024-04-19T00:59:05.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 -- start start 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient: init 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient: _add_conns ranks=[8,5,9,4,6,3,2,10,0,1,7] 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c138cd0 0x7fdb3c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): picked mon.noname-e con 0x7fdb3c138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): picked mon.noname-h con 0x7fdb3c0a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdb3c1358b0 0x7fdb3c135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): picked mon.noname-a con 0x7fdb3c1358b0 addr v2:172.21.15.43:3300/0 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fdb3c0a4410 con 0x7fdb3c1358b0 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdb3c139b30 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.211 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdb3c0a3dc0 con 0x7fdb3c138cd0 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb42ecf640 10 monclient(hunting): authenticate will time out at 1815.337466s 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb40c6b640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c138cd0 msgr2=0x7fdb3c1390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb40c6b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c138cd0 0x7fdb3c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdb3c1358b0 0x7fdb3c135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdb3c1358b0 0x7fdb3c135d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:42304/0 (socket says 172.21.15.43:42304) 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 -- 172.21.15.43:0/3928662491 learned_addr learned my addr 172.21.15.43:0/3928662491 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): get_auth_request con 0x7fdb3c1358b0 auth_method 0 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:05.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb3bfff640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb3bfff640 10 monclient(hunting): get_auth_request con 0x7fdb3c0a5b80 auth_method 0 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient(hunting): handle_auth_done global_id 65295 payload 274 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 -- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c138cd0 msgr2=0x7fdb3c1390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c138cd0 0x7fdb3c1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 -- 172.21.15.43:0/3928662491 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb3c0a5b80 msgr2=0x7fdb3c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient: _finish_hunting 0 2024-04-19T00:59:05.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 monclient: found mon.noname-a 2024-04-19T00:59:05.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:05.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T00:59:05.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.207+0000 7fdb4146c640 1 -- 172.21.15.43:0/3928662491 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c0a40a0 con 0x7fdb3c1358b0 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb4146c640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb3c1358b0 0x7fdb3c135d60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009e50 tx=0x7fdb3401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017feefee64d389 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 1 -- 172.21.15.43:0/3928662491 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb340090d0 con 0x7fdb3c1358b0 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: dump: 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.220 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: _finish_auth 0 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: _check_auth_tickets 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 1 -- 172.21.15.43:0/3928662491 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb340a5a70 con 0x7fdb3c1358b0 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 5 monclient: authenticate success, global_id 65295 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 1 -- 172.21.15.43:0/3928662491 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb340a5c60 con 0x7fdb3c1358b0 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3b7fe640 10 monclient: dump: 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.222 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: get_monmap_and_config success 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 4 set_mon_vals no callback set 2024-04-19T00:59:05.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: shutdown 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb3c1358b0 msgr2=0x7fdb3c135d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb3c1358b0 0x7fdb3c135d60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009e50 tx=0x7fdb3401c8b0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/3928662491 shutdown_connections 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdb3c1358b0 0x7fdb3c135d60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/3928662491 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c138cd0 0x7fdb3c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/3928662491 >> 172.21.15.43:0/3928662491 conn(0x7fdb3c0ab2f0 msgr2=0x7fdb3c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/3928662491 shutdown_connections 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/3928662491 wait complete. 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: build_initial_monmap 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: monmap: 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T00:59:05.224 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 librados: starting msgr at 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 librados: starting objecter 2024-04-19T00:59:05.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 Processor -- start 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- start start 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 librados: setting wanted keys 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 librados: calling monclient init 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: init 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: _reopen_session rank -1 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: _add_conns ranks=[4,8,1,5,0,3,7,6,2] 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1d0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3c0a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c1358b0 0x7fdb3c13eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): picked mon.noname-e con 0x7fdb3c1358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb3c138cd0 0x7fdb3c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): picked mon.noname-f con 0x7fdb3c138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T00:59:05.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c0a2010 con 0x7fdb3c138cd0 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdb3c0a1e90 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdb3c142aa0 con 0x7fdb3c1358b0 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): _renew_subs 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient(hunting): authenticate will time out at 1815.342685s 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3bfff640 1 -- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c1358b0 msgr2=0x7fdb3c13eef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3304/0 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c1358b0 0x7fdb3c13eef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1d0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb4146c640 1 -- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb3c138cd0 msgr2=0x7fdb3c13f430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3300/0 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1d0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:45696/0 (socket says 172.21.15.43:45696) 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 -- 172.21.15.43:0/2235059891 learned_addr learned my addr 172.21.15.43:0/2235059891 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T00:59:05.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): get_auth_request con 0x7fdb3c0a5b80 auth_method 0 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb4146c640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb3c138cd0 0x7fdb3c13f430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient(hunting): handle_auth_done global_id 134311 payload 824 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c1358b0 msgr2=0x7fdb3c13eef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb3c1358b0 0x7fdb3c13eef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb3c138cd0 msgr2=0x7fdb3c13f430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb3c138cd0 0x7fdb3c13f430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient: _finish_hunting 0 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 monclient: found mon.noname-c 2024-04-19T00:59:05.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c13b6e0 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb40c6b640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1d0e00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00b550 tx=0x7fdb2c00ba10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 client.?.objecter ms_handle_connect 0x7fdb3c0a5b80 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 client.?.objecter resend_mon_ops 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb2c0092d0 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: dump: 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.229 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: _finish_auth 0 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: _check_auth_tickets 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2c015070 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 5 monclient: authenticate success, global_id 134311 2024-04-19T00:59:05.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: _renew_subs 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3c0a3590 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: handle_config config(0 keys) 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb2c003c30 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 client.134311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb3a7fc640 4 set_mon_vals no callback set 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb397fa640 10 monclient: dump: 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.231 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: _renew_subs 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3c13b400 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.211+0000 7fdb42ecf640 1 librados: init done 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb42ecf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 0x7fdb3c133a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdb3c13fc20 con 0x7fdb3c133650 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdb2c0050f0 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb28038e10 0x7fdb2803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(539..539 src has 1..539) ==== 6625+0+0 (secure 0 0 0) 0x7fdb2c049360 con 0x7fdb3c0a5b80 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 client.134311.objecter ms_dispatch 0x7fdb3c000ba0 osd_map(539..539 src has 1..539) 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 3 client.134311.objecter handle_osd_map got epochs [539,539] > 0 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 3 client.134311.objecter handle_osd_map decoding full epoch 539 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 20 client.134311.objecter dump_active .. 0 homeless 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 0x7fdb3c133a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient: get_auth_request con 0x7fdb3c133650 auth_method 0 2024-04-19T00:59:05.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb4146c640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb28038e10 0x7fdb2803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb4146c640 10 monclient: get_auth_request con 0x7fdb28038e10 auth_method 0 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_done global_id 74563 payload 824 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb3c13f970 con 0x7fdb3c133650 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 client.134311.objecter ms_handle_connect 0x7fdb28038e10 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb4146c640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb28038e10 0x7fdb2803b2d0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7fdb340ae000 tx=0x7fdb340a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 client.134311.objecter ms_handle_connect 0x7fdb3c133650 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 client.134311.objecter resend_mon_ops 2024-04-19T00:59:05.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb3bfff640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 0x7fdb3c133a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c135100 tx=0x7fdb3000d8b0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb30012a60 con 0x7fdb3c133650 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 monclient: dump: 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.235 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb30018070 con 0x7fdb3c133650 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 msgr2=0x7fdb3c133a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 0x7fdb3c133a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c135100 tx=0x7fdb3000d8b0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 msgr2=0x7fdb3c133a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:05.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.215+0000 7fdb397fa640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133650 0x7fdb3c133a30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb42ecf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 0x7fdb3c130190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb42ecf640 10 monclient(hunting): start opening mon connection 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdb3c06d140 con 0x7fdb3c133f70 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 0x7fdb3c130190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient: get_auth_request con 0x7fdb3c133f70 auth_method 0 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T00:59:05.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 10 monclient(hunting): handle_auth_done global_id 74572 payload 824 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 1 -- 172.21.15.43:0/2235059891 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdb3c1d1a60 con 0x7fdb3c133f70 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb3bfff640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 0x7fdb3c130190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c135100 tx=0x7fdb30004740 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 client.134311.objecter ms_handle_connect 0x7fdb3c133f70 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 client.134311.objecter resend_mon_ops 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb30012e10 con 0x7fdb3c133f70 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 monclient: dump: 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T00:59:05.239 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T00:59:05.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9400 (secure 0 0 0) 0x7fdb30012e10 con 0x7fdb3c133f70 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 msgr2=0x7fdb3c130190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 0x7fdb3c130190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c135100 tx=0x7fdb30004740 comp rx=0 tx=0).stop 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 msgr2=0x7fdb3c130190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.223+0000 7fdb397fa640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdb3c133f70 0x7fdb3c130190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 102, 2024-04-19T00:59:05.241 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 324842, 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: 7 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:05.242 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:05.243 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T00:59:05.244 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T00:59:05.245 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T00:59:05.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T00:59:05.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.250 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.255 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T00:59:05.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.258 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T00:59:05.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:05.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T00:59:05.261 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T00:59:05.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T00:59:05.262 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T00:59:05.262 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T00:59:05.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 10 librados: watch_flush enter 2024-04-19T00:59:05.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 10 librados: watch_flush exit 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 20 client.134311.objecter shutdown clearing up homeless session... 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 10 client.134311.objecter successfully canceled tick 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb28038e10 msgr2=0x7fdb2803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb28038e10 0x7fdb2803b2d0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7fdb340ae000 tx=0x7fdb340a7070 comp rx=0 tx=0).stop 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 10 monclient: shutdown 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb3c0a5b80 msgr2=0x7fdb3c1d0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 --2- 172.21.15.43:0/2235059891 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdb3c0a5b80 0x7fdb3c1d0e00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c00b550 tx=0x7fdb2c00ba10 comp rx=0 tx=0).stop 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb4146c640 1 -- 172.21.15.43:0/2235059891 reap_dead start 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb4146c640 1 -- 172.21.15.43:0/2235059891 reap_dead start 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 shutdown_connections 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 >> 172.21.15.43:0/2235059891 conn(0x7fdb3c0ab2f0 msgr2=0x7fdb3c1322d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 shutdown_connections 2024-04-19T00:59:05.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 -- 172.21.15.43:0/2235059891 wait complete. 2024-04-19T00:59:05.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T00:59:05.227+0000 7fdb42ecf640 1 librados: shutdown 2024-04-19T00:59:05.265 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T00:59:05.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T01:00:07.882 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8958 ms) 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6002 ms) 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5085 ms) 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T01:00:07.883 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (55238 ms) 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13369 ms) 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11988 ms) 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 00000012 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12087 ms) 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: require_osd_release = squid 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6105 ms) 2024-04-19T01:00:07.884 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (705418 ms total) 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1032 ms) 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10994 ms) 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17966 ms) 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (6 ms) 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (29998 ms total) 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-19T01:00:07.885 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6942 ms) 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (19742 ms) 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: waiting for scrub... 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: done waiting 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (50336 ms) 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9950 ms) 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8101 ms) 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-19T01:00:07.886 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7871 ms) 2024-04-19T01:00:07.887 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-19T01:00:07.887 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10074 ms) 2024-04-19T01:00:07.887 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-19T01:00:35.265 INFO:tasks.mon_thrash.mon_thrasher:killing mon.f 2024-04-19T01:00:35.266 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.f 2024-04-19T01:00:35.266 INFO:tasks.ceph.mon.f:Restarting daemon 2024-04-19T01:00:35.266 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i f 2024-04-19T01:00:35.269 INFO:tasks.ceph.mon.f:Started 2024-04-19T01:00:35.270 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-19T01:00:35.270 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-19T01:00:35.270 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-19T01:00:35.270 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-19T01:00:35.272 INFO:tasks.ceph.mon.b:Started 2024-04-19T01:00:35.273 INFO:tasks.mon_thrash.mon_thrasher:killing mon.e 2024-04-19T01:00:35.273 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.e 2024-04-19T01:00:35.273 INFO:tasks.ceph.mon.e:Restarting daemon 2024-04-19T01:00:35.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i e 2024-04-19T01:00:35.275 INFO:tasks.ceph.mon.e:Started 2024-04-19T01:00:35.275 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T01:00:35.275 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-04-19T01:00:35.275 INFO:tasks.ceph.mon.d:Restarting daemon 2024-04-19T01:00:35.276 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-04-19T01:00:35.278 INFO:tasks.ceph.mon.d:Started 2024-04-19T01:00:35.278 INFO:tasks.mon_thrash:Sending STOP to mon i 2024-04-19T01:00:35.278 INFO:tasks.ceph.mon.i:Sent signal 19 2024-04-19T01:00:35.278 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T01:00:35.395 INFO:tasks.ceph.mon.f.smithi161.stderr:2024-04-19T01:00:35.397+0000 7f0caea58c80 -1 mon.f@-1(???).log v0 log_external_backlog rewinding external_log_to from 562 -> 0 (sync_force? mon rebuild?) 2024-04-19T01:00:50.280 INFO:tasks.mon_thrash:Sending CONT to mon i 2024-04-19T01:00:50.281 INFO:tasks.ceph.mon.i:Sent signal 18 2024-04-19T01:00:50.281 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T01:00:50.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:00:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient: get_monmap_and_config 2024-04-19T01:00:50.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient: build_initial_monmap 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient: monmap: 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:50.419711+0000 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:50.419711+0000 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:50.417 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:50.418 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:50.418 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 Processor -- start 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 -- start start 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient: init 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient: _add_conns ranks=[9,5,0,8,3,7,2,10,6,1,4] 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): picked mon.noname-a con 0x7fc400138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): picked mon.noname-h con 0x7fc4000a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4001358b0 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:50.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4000a3dc0 con 0x7fc4001358b0 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fc4000a4410 con 0x7fc400138cd0 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc400139b30 con 0x7fc4000a5b80 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc40795b640 10 monclient(hunting): authenticate will time out at 1920.547187s 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:36282/0 (socket says 172.21.15.43:36282) 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 learned_addr learned my addr 172.21.15.43:0/3222194482 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc400138cd0 auth_method 0 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:50.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 10 monclient(hunting): get_auth_request con 0x7fc4000a5b80 auth_method 0 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:50.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.417+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:00:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc400138cd0 auth_method 0 2024-04-19T01:00:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:50.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 10 monclient(hunting): get_auth_request con 0x7fc4000a5b80 auth_method 0 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:50.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:50.621+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc400138cd0 auth_method 0 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 10 monclient(hunting): get_auth_request con 0x7fc4000a5b80 auth_method 0 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:51.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:00:51.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.025+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:00:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc400138cd0 auth_method 0 2024-04-19T01:00:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:51.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 10 monclient(hunting): get_auth_request con 0x7fc4000a5b80 auth_method 0 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:00:51.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:51.829+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:00:53.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): tick 2024-04-19T01:00:53.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 monclient(hunting): continuing hunt 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 msgr2=0x7fc4001395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc4000a5b80 0x7fc4001395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 msgr2=0x7fc4001390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.43:3300/0 conn(0x7fc400138cd0 0x7fc4001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 -- 172.21.15.43:0/3222194482 >> v1:172.21.15.43:6789/0 conn(0x7fc4001358b0 legacy=0x7fc400135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient: _add_conns ranks=[10,1,4,8,2,3,7,6] 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): picked mon.noname-f con 0x7fc3e4003800 addr v2:172.21.15.161:3300/0 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fc3e4006260 addr v1:172.21.15.161:6789/0 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e4009f70 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 -- 172.21.15.43:0/3222194482 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4000a40a0 con 0x7fc3e4006260 2024-04-19T01:00:53.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 -- 172.21.15.43:0/3222194482 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc4000a4410 con 0x7fc3e4003800 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 1 -- 172.21.15.43:0/3222194482 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc400139b30 con 0x7fc3e4009f70 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc3f6ffd640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.417+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc3e4003800 auth_method 0 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 10 monclient(hunting): get_auth_request con 0x7fc3e4009f70 auth_method 0 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:53.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:53.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.421+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:00:53.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 10 monclient(hunting): get_auth_request con 0x7fc3e4009f70 auth_method 0 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc3e4003800 auth_method 0 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:53.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:00:53.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:00:53.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:53.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:53.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:53.625+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 10 monclient(hunting): get_auth_request con 0x7fc3e4009f70 auth_method 0 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:54.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc3e4003800 auth_method 0 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:54.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.029+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:00:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 10 monclient(hunting): get_auth_request con 0x7fc3e4009f70 auth_method 0 2024-04-19T01:00:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:54.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc3e4003800 auth_method 0 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:54.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:54.833+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:00:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.065+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942127296 0 0) 0x7fc4000a40a0 con 0x7fc3e4006260 2024-04-19T01:00:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.065+0000 7fc3f7fff640 10 monclient(hunting): _negotiate 2024-04-19T01:00:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.065+0000 7fc3f7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.065+0000 7fc3f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.065+0000 7fc3f7fff640 10 monclient(hunting): my global_id is 144100 2024-04-19T01:00:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.065+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3ec003530 con 0x7fc3e4006260 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3977426058 0 0) 0x7fc3ec003530 con 0x7fc3e4006260 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 msgr2=0x7fc3e400c430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4009f70 0x7fc3e400c430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 msgr2=0x7fc3e4005c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 --2- 172.21.15.43:0/3222194482 >> v2:172.21.15.161:3300/0 conn(0x7fc3e4003800 0x7fc3e4005c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 monclient: found mon.noname-f-legacy 2024-04-19T01:00:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 1 -- 172.21.15.43:0/3222194482 reap_dead start 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.161:6789/0 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e400dcd0 con 0x7fc3e4006260 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fc3fc00d7f0 con 0x7fc3e4006260 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: dump: 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] but i'm connected to v1:172.21.15.161:6789/0, reconnecting 2024-04-19T01:00:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 >> v1:172.21.15.161:6789/0 conn(0x7fc3e4006260 legacy=0x7fc3e4008720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient: _add_conns ranks=[7,6,5,4,2,8,1,3,0] 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc3ec004210 0x7fc3ec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): picked mon.i con 0x7fc3ec004210 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec004e20 0x7fc3ec0052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): picked mon.d con 0x7fc3ec004e20 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec009c00 0x7fc3ec00c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): picked mon.h con 0x7fc3ec009c00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc4000a4410 con 0x7fc3ec009c00 2024-04-19T01:00:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fc400139b30 con 0x7fc3ec004e20 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc3ec00c9a0 con 0x7fc3ec004210 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc3f7fff640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec004e20 0x7fc3ec0052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): get_auth_request con 0x7fc3ec004e20 auth_method 0 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec009c00 0x7fc3ec00c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc3ec004210 0x7fc3ec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc3ec004210 auth_method 0 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): get_auth_request con 0x7fc3ec009c00 auth_method 0 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc405ef8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient(hunting): handle_auth_done global_id 144100 payload 293 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec004e20 msgr2=0x7fc3ec0052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec004e20 0x7fc3ec0052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec009c00 msgr2=0x7fc3ec00c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec009c00 0x7fc3ec00c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 monclient: found mon.i 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient: _send_mon_message to mon.i at v2:172.21.15.161:3303/0 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 1 -- 172.21.15.43:0/3222194482 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec003530 con 0x7fc3ec004210 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc4056f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc40795b640 5 monclient: authenticate success, global_id 144100 2024-04-19T01:00:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc40795b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.073+0000 7fc404ef6640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec004e20 0x7fc3ec0052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc4056f7640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc3ec004210 0x7fc3ec004810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4003740 tx=0x7fc3f001dcc0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=4e97b16357a0887a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3f00b3070 con 0x7fc3ec004210 2024-04-19T01:00:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:00:55.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: dump: 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.083 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 <== mon.7 v2:172.21.15.161:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3f00a6cf0 con 0x7fc3ec004210 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 1 -- 172.21.15.43:0/3222194482 <== mon.7 v2:172.21.15.161:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3f000f040 con 0x7fc3ec004210 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 4 set_mon_vals no callback set 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc3f7fff640 10 monclient: dump: 2024-04-19T01:00:55.084 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.085 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 10 monclient: shutdown 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 -- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc3ec004210 msgr2=0x7fc3ec004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc3ec004210 0x7fc3ec004810 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4003740 tx=0x7fc3f001dcc0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 -- 172.21.15.43:0/3222194482 shutdown_connections 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec009c00 0x7fc3ec00c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec004e20 0x7fc3ec0052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 --2- 172.21.15.43:0/3222194482 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc3ec004210 0x7fc3ec004810 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 -- 172.21.15.43:0/3222194482 >> 172.21.15.43:0/3222194482 conn(0x7fc4000ab2f0 msgr2=0x7fc4000ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 -- 172.21.15.43:0/3222194482 shutdown_connections 2024-04-19T01:00:55.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.081+0000 7fc40795b640 1 -- 172.21.15.43:0/3222194482 wait complete. 2024-04-19T01:00:55.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient: monmap: 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.092 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 librados: starting msgr at 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 librados: starting objecter 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 Processor -- start 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 -- start start 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 librados: setting wanted keys 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 librados: calling monclient init 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient: init 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient: _add_conns ranks=[4,8,5,2,7,1,3,0,6] 2024-04-19T01:00:55.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 0x7fc4001d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e4006260 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 0x7fc4000a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): picked mon.noname-e con 0x7fc3ec004210 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec004e20 0x7fc4000a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): picked mon.noname-h con 0x7fc3ec004e20 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc400135c90 con 0x7fc3e4006260 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc4000a20a0 con 0x7fc3ec004e20 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fc400135e10 con 0x7fc3ec004210 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient(hunting): authenticate will time out at 1925.214489s 2024-04-19T01:00:55.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 0x7fc4000a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 0x7fc4001d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 0x7fc4001d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52740/0 (socket says 172.21.15.43:52740) 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 0x7fc4000a0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:38996/0 (socket says 172.21.15.43:38996) 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 1 -- 172.21.15.43:0/982325471 learned_addr learned my addr 172.21.15.43:0/982325471 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): get_auth_request con 0x7fc3e4006260 auth_method 0 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): get_auth_request con 0x7fc3ec004210 auth_method 0 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec004e20 0x7fc4000a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): get_auth_request con 0x7fc3ec004e20 auth_method 0 2024-04-19T01:00:55.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc405ef8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient(hunting): handle_auth_done global_id 144107 payload 824 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 -- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec004e20 msgr2=0x7fc4000a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec004e20 0x7fc4000a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 -- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 msgr2=0x7fc4001d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 0x7fc4001d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 monclient: found mon.noname-e 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 -- 172.21.15.43:0/982325471 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40009d4d0 con 0x7fc3ec004210 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc4056f7640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 0x7fc4001d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc404ef6640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 0x7fc4000a0f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0038d0 tx=0x7fc3fc003d90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 client.?.objecter ms_handle_connect 0x7fc3ec004210 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3fc01b070 con 0x7fc3ec004210 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:00:55.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 monclient: dump: 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.098 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc017440 con 0x7fc3ec004210 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 5 monclient: authenticate success, global_id 144107 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc40795b640 10 monclient: _renew_subs 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3fc006b80 con 0x7fc3ec004210 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.085+0000 7fc3f77fe640 4 set_mon_vals no callback set 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40009f7c0 con 0x7fc3ec004210 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 10 monclient: dump: 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.100 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 client.144107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 monclient: _renew_subs 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4001d0df0 con 0x7fc3ec004210 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 1 librados: init done 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc3fc015af0 con 0x7fc3ec004210 2024-04-19T01:00:55.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3e4039f50 0x7fc3e403c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(610..610 src has 501..610) ==== 6709+0+0 (secure 0 0 0) 0x7fc3fc00f040 con 0x7fc3ec004210 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 10 client.144107.objecter ms_dispatch 0x7fc400000ba0 osd_map(610..610 src has 501..610) 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 3 client.144107.objecter handle_osd_map got epochs [610,610] > 0 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 3 client.144107.objecter handle_osd_map decoding full epoch 610 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4000693b0 con 0x7fc3ec004210 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 20 client.144107.objecter dump_active .. 0 homeless 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc4056f7640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3e4039f50 0x7fc3e403c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc4056f7640 10 monclient: get_auth_request con 0x7fc3e4039f50 auth_method 0 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc4056f7640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3e4039f50 0x7fc3e403c410 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7fc3f001c090 tx=0x7fc3f0004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.089+0000 7fc3f5ffb640 10 client.144107.objecter ms_handle_connect 0x7fc3e4039f50 2024-04-19T01:00:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.097+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc3fc0163d0 con 0x7fc3ec004210 2024-04-19T01:00:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.097+0000 7fc3f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.097+0000 7fc3f5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc40795b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:00:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc40795b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:00:55.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc40795b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:55.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc400069650 con 0x7fc3ec004210 2024-04-19T01:00:55.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc3f5ffb640 1 -- 172.21.15.43:0/982325471 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fc3fc01fdf0 con 0x7fc3ec004210 2024-04-19T01:00:55.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc3f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:00:55.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.345+0000 7fc3f5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:55.346 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:00:55.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 10 librados: watch_flush enter 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 10 librados: watch_flush exit 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 20 client.144107.objecter shutdown clearing up homeless session... 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 10 client.144107.objecter successfully canceled tick 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3e4039f50 msgr2=0x7fc3e403c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3e4039f50 0x7fc3e403c410 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7fc3f001c090 tx=0x7fc3f0004740 comp rx=0 tx=0).stop 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 10 monclient: shutdown 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 msgr2=0x7fc4000a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 0x7fc4000a0f20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0038d0 tx=0x7fc3fc003d90 comp rx=0 tx=0).stop 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 shutdown_connections 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec004e20 0x7fc4000a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3e4039f50 0x7fc3e403c410 unknown :-1 s=CLOSED pgs=3591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.349 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec004210 0x7fc4000a0f20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 --2- 172.21.15.43:0/982325471 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3e4006260 0x7fc4001d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 >> 172.21.15.43:0/982325471 conn(0x7fc4000ab2f0 msgr2=0x7fc4000a8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 shutdown_connections 2024-04-19T01:00:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 -- 172.21.15.43:0/982325471 wait complete. 2024-04-19T01:00:55.350 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.349+0000 7fc40795b640 1 librados: shutdown 2024-04-19T01:00:55.366 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T01:00:55.366 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T01:00:55.366 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:00:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient: get_monmap_and_config 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient: monmap: 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:55.487038+0000 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:55.487038+0000 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:55.485 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 Processor -- start 2024-04-19T01:00:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 -- start start 2024-04-19T01:00:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient: init 2024-04-19T01:00:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient: _add_conns ranks=[10,7,6,3,8,0,5,2,1,4,9] 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8194072b50 0x7f8194139090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): picked mon.noname-f con 0x7f8194072b50 addr v2:172.21.15.161:3300/0 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8194072f30 0x7f81940a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): picked mon.noname-i con 0x7f8194072f30 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 0x7f81940a5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): picked mon.noname-d con 0x7f81940a4f00 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f819406deb0 con 0x7f8194072b50 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f819406dc70 con 0x7f81940a4f00 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f819406e0b0 con 0x7f8194072f30 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f819affb640 10 monclient(hunting): authenticate will time out at 1925.614928s 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 0x7f81940a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 0x7f81940a5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53774/0 (socket says 172.21.15.43:53774) 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 -- 172.21.15.43:0/1942410579 learned_addr learned my addr 172.21.15.43:0/1942410579 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): get_auth_request con 0x7f81940a4f00 auth_method 0 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8198d97640 1 --2- 172.21.15.43:0/1942410579 >> v2:172.21.15.161:3300/0 conn(0x7f8194072b50 0x7f8194139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8193fff640 1 --2- 172.21.15.43:0/1942410579 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8194072f30 0x7f81940a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8198d97640 10 monclient(hunting): get_auth_request con 0x7f8194072b50 auth_method 0 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8198d97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8198d97640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8198d97640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8193fff640 10 monclient(hunting): get_auth_request con 0x7f8194072f30 auth_method 0 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8193fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient(hunting): handle_auth_done global_id 144105 payload 274 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 -- 172.21.15.43:0/1942410579 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8194072f30 msgr2=0x7f81940a49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:55.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 --2- 172.21.15.43:0/1942410579 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8194072f30 0x7f81940a49c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 -- 172.21.15.43:0/1942410579 >> v2:172.21.15.161:3300/0 conn(0x7f8194072b50 msgr2=0x7f8194139090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 --2- 172.21.15.43:0/1942410579 >> v2:172.21.15.161:3300/0 conn(0x7f8194072b50 0x7f8194139090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 monclient: found mon.noname-d 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:00:55.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.485+0000 7f8199598640 1 -- 172.21.15.43:0/1942410579 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819406f3d0 con 0x7f81940a4f00 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f8199598640 1 --2- 172.21.15.43:0/1942410579 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 0x7f81940a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f818c0098a0 tx=0x7f818c01c400 comp rx=0 tx=0).ready entity=mon.6 client_cookie=90123068f40a18ad server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 1 -- 172.21.15.43:0/1942410579 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f818c0b0070 con 0x7f81940a4f00 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: dump: 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.501 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 5 monclient: authenticate success, global_id 144105 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 1 -- 172.21.15.43:0/1942410579 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f818c0aa810 con 0x7f81940a4f00 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 4 set_mon_vals no callback set 2024-04-19T01:00:55.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 1 -- 172.21.15.43:0/1942410579 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f818c0a9300 con 0x7f81940a4f00 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f81937fe640 10 monclient: dump: 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.503 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 10 monclient: shutdown 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 -- 172.21.15.43:0/1942410579 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 msgr2=0x7f81940a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 --2- 172.21.15.43:0/1942410579 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 0x7f81940a5380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f818c0098a0 tx=0x7f818c01c400 comp rx=0 tx=0).stop 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 -- 172.21.15.43:0/1942410579 shutdown_connections 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 --2- 172.21.15.43:0/1942410579 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f81940a4f00 0x7f81940a5380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 --2- 172.21.15.43:0/1942410579 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f8194072f30 0x7f81940a49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 --2- 172.21.15.43:0/1942410579 >> v2:172.21.15.161:3300/0 conn(0x7f8194072b50 0x7f8194139090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 -- 172.21.15.43:0/1942410579 >> 172.21.15.43:0/1942410579 conn(0x7f81940a6270 msgr2=0x7f819406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 -- 172.21.15.43:0/1942410579 shutdown_connections 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 -- 172.21.15.43:0/1942410579 wait complete. 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 10 monclient: monmap: 2024-04-19T01:00:55.505 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.506 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 librados: starting msgr at 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 librados: starting objecter 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.489+0000 7f819affb640 1 Processor -- start 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- start start 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 librados: setting wanted keys 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 librados: calling monclient init 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: init 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: _add_conns ranks=[0,4,1,8,7,3,5,6,2] 2024-04-19T01:00:55.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8194072b50 0x7f81941d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): picked mon.noname-a con 0x7f8194072b50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8194072f30 0x7f81941d0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): picked mon.noname-c con 0x7f8194072f30 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81940a4f00 0x7f81941d1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): picked mon.noname-f con 0x7f81940a4f00 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81941426d0 con 0x7f8194072b50 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8194142850 con 0x7f81940a4f00 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f81941429d0 con 0x7f8194072f30 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): authenticate will time out at 1925.620533s 2024-04-19T01:00:55.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8194072b50 0x7f81941d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8194072f30 0x7f81941d0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8194072b50 0x7f81941d09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55544/0 (socket says 172.21.15.43:55544) 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8194072f30 0x7f81941d0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52748/0 (socket says 172.21.15.43:52748) 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 -- 172.21.15.43:0/2294967453 learned_addr learned my addr 172.21.15.43:0/2294967453 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): get_auth_request con 0x7f8194072b50 auth_method 0 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): get_auth_request con 0x7f8194072f30 auth_method 0 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81940a4f00 0x7f81941d1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 10 monclient(hunting): get_auth_request con 0x7f81940a4f00 auth_method 0 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient(hunting): handle_auth_done global_id 65304 payload 824 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8194072f30 msgr2=0x7f81941d0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8194072f30 0x7f81941d0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81940a4f00 msgr2=0x7f81941d1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f81940a4f00 0x7f81941d1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 monclient: found mon.noname-a 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940a1e70 con 0x7f8194072b50 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f8194072f30 0x7f81941d0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8198d97640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8194072b50 0x7f81941d09d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8184007ef0 tx=0x7f818400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 client.?.objecter ms_handle_connect 0x7f8194072b50 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8184014890 con 0x7f8194072b50 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:55.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: dump: 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.512 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8184019070 con 0x7f8194072b50 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 5 monclient: authenticate success, global_id 65304 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: _renew_subs 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81940a2150 con 0x7f8194072b50 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8184013380 con 0x7f8194072b50 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:55.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: dump: 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.514 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 client.65304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: _renew_subs 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8192ffd640 4 set_mon_vals no callback set 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81940a04d0 con 0x7f8194072b50 2024-04-19T01:00:55.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 librados: init done 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 0x7f819412cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f81940a0780 con 0x7f819412c6d0 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 0x7f819412cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient: get_auth_request con 0x7f819412c6d0 auth_method 0 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8184014420 con 0x7f8194072b50 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 10 monclient(hunting): handle_auth_done global_id 65313 payload 824 2024-04-19T01:00:55.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8180038e10 0x7f818003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8194134af0 con 0x7f819412c6d0 2024-04-19T01:00:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(610..610 src has 501..610) ==== 6709+0+0 (secure 0 0 0) 0x7f8184049ee0 con 0x7f8194072b50 2024-04-19T01:00:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 client.65304.objecter ms_dispatch 0x7f8194000ba0 osd_map(610..610 src has 501..610) 2024-04-19T01:00:55.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 3 client.65304.objecter handle_osd_map got epochs [610,610] > 0 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 3 client.65304.objecter handle_osd_map decoding full epoch 610 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 20 client.65304.objecter dump_active .. 0 homeless 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 client.65304.objecter ms_handle_connect 0x7f819412c6d0 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 client.65304.objecter resend_mon_ops 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8193fff640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 0x7f819412cab0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f819409fee0 tx=0x7f8188007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8188018020 con 0x7f819412c6d0 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: dump: 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.518 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8180038e10 0x7f818003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f8199598640 10 monclient: get_auth_request con 0x7f8180038e10 auth_method 0 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f818800ec10 con 0x7f819412c6d0 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 msgr2=0x7f819412cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 0x7f819412cab0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f819409fee0 tx=0x7f8188007f70 comp rx=0 tx=0).stop 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 msgr2=0x7f819412cab0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.493+0000 7f81917fa640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412c6d0 0x7f819412cab0 secure :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f819409fee0 tx=0x7f8188007f70 comp rx=0 tx=0).stop 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.497+0000 7f81917fa640 10 client.65304.objecter ms_handle_connect 0x7f8180038e10 2024-04-19T01:00:55.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.497+0000 7f8199598640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8180038e10 0x7f818003b2d0 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ad000 tx=0x7f818c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:55.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 0x7f8194130e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8194073520 con 0x7f819412cff0 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 0x7f8194130e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient: get_auth_request con 0x7f819412cff0 auth_method 0 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 10 monclient(hunting): handle_auth_done global_id 65322 payload 824 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 1 -- 172.21.15.43:0/2294967453 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f819406ebc0 con 0x7f819412cff0 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 client.65304.objecter ms_handle_connect 0x7f819412cff0 2024-04-19T01:00:55.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8193fff640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 0x7f8194130e20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8188019000 tx=0x7f81880046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 client.65304.objecter resend_mon_ops 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8188018040 con 0x7f819412cff0 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 monclient: dump: 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.524 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.525 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.525 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.525 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.525 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7f81880039d0 con 0x7f819412cff0 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 msgr2=0x7f8194130e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 0x7f8194130e20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8188019000 tx=0x7f81880046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 msgr2=0x7f8194130e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:55.526 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f81917fa640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f819412cff0 0x7f8194130e20 secure :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8188019000 tx=0x7f81880046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 581773, 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:55.527 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:55.528 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:55.529 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:55.530 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:55.531 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:55.532 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:55.532 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:55.532 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:55.532 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.533 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:55.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.537 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:55.538 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.539 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.540 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.541 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:55.542 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:55.543 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.544 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.545 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.546 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:55.547 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.548 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.549 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.550 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.551 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:00:55.551 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.551 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.551 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.551 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:55.551 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 10 librados: watch_flush enter 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 10 librados: watch_flush exit 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 20 client.65304.objecter shutdown clearing up homeless session... 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 10 client.65304.objecter successfully canceled tick 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8180038e10 msgr2=0x7f818003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8180038e10 0x7f818003b2d0 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ad000 tx=0x7f818c0a6000 comp rx=0 tx=0).stop 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 10 monclient: shutdown 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8194072b50 msgr2=0x7f81941d09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 --2- 172.21.15.43:0/2294967453 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8194072b50 0x7f81941d09d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8184007ef0 tx=0x7f818400e4b0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8199598640 1 -- 172.21.15.43:0/2294967453 reap_dead start 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f8199598640 1 -- 172.21.15.43:0/2294967453 reap_dead start 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 shutdown_connections 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 >> 172.21.15.43:0/2294967453 conn(0x7f81940a6270 msgr2=0x7f81940718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 shutdown_connections 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 -- 172.21.15.43:0/2294967453 wait complete. 2024-04-19T01:00:55.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.501+0000 7f819affb640 1 librados: shutdown 2024-04-19T01:00:55.554 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:55.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T01:00:55.555 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:00:55.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient: get_monmap_and_config 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient: monmap: 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:55.678997+0000 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:55.678997+0000 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.676 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:55.677 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 Processor -- start 2024-04-19T01:00:55.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 -- start start 2024-04-19T01:00:55.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient: init 2024-04-19T01:00:55.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient: _add_conns ranks=[0,7,2,8,1,6,4,3,9,10,5] 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6408138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f64080a5b80 0x7f640813e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): picked mon.noname-i con 0x7f64080a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64081397c0 0x7f64081407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): picked mon.noname-b con 0x7f64081397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64080a3dc0 con 0x7f6408138cd0 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f64080a4410 con 0x7f64081397c0 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f64080a40a0 con 0x7f64080a5b80 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f64100a5640 10 monclient(hunting): authenticate will time out at 1925.806143s 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64081397c0 0x7f64081407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640de41640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6408138cd0 0x7f64081390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43046/0 (socket says 172.21.15.43:43046) 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640de41640 1 -- 172.21.15.43:0/2554580933 learned_addr learned my addr 172.21.15.43:0/2554580933 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): get_auth_request con 0x7f64081397c0 auth_method 0 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 1 --2- 172.21.15.43:0/2554580933 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f64080a5b80 0x7f640813e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): get_auth_request con 0x7f64080a5b80 auth_method 0 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640d640640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient(hunting): handle_auth_done global_id 144101 payload 274 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640ce3f640 1 -- 172.21.15.43:0/2554580933 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 465210732 0 0) 0x7f64080a3dc0 con 0x7f6408138cd0 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 1 -- 172.21.15.43:0/2554580933 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f64080a5b80 msgr2=0x7f640813e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 1 --2- 172.21.15.43:0/2554580933 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f64080a5b80 0x7f640813e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 1 -- 172.21.15.43:0/2554580933 >> v1:172.21.15.43:6789/0 conn(0x7f6408138cd0 legacy=0x7f64081390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 1 monclient: found mon.noname-b 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640e642640 1 -- 172.21.15.43:0/2554580933 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64080a2ae0 con 0x7f64081397c0 2024-04-19T01:00:55.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.677+0000 7f640ce3f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640e642640 1 --2- 172.21.15.43:0/2554580933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64081397c0 0x7f64081407c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6400009dc0 tx=0x7f640001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4cfc0473398242 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 1 -- 172.21.15.43:0/2554580933 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64000b2070 con 0x7f64081397c0 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: dump: 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.690 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 5 monclient: authenticate success, global_id 144101 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 1 -- 172.21.15.43:0/2554580933 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64000a5a50 con 0x7f64081397c0 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 4 set_mon_vals no callback set 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 1 -- 172.21.15.43:0/2554580933 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64000a5c40 con 0x7f64081397c0 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f640ce3f640 10 monclient: dump: 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.692 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: shutdown 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- 172.21.15.43:0/2554580933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64081397c0 msgr2=0x7f64081407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 --2- 172.21.15.43:0/2554580933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64081397c0 0x7f64081407c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6400009dc0 tx=0x7f640001c850 comp rx=0 tx=0).stop 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- 172.21.15.43:0/2554580933 shutdown_connections 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 --2- 172.21.15.43:0/2554580933 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64081397c0 0x7f64081407c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 --2- 172.21.15.43:0/2554580933 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f64080a5b80 0x7f640813e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- 172.21.15.43:0/2554580933 >> 172.21.15.43:0/2554580933 conn(0x7f64080ab2f0 msgr2=0x7f64080ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- 172.21.15.43:0/2554580933 shutdown_connections 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- 172.21.15.43:0/2554580933 wait complete. 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: monmap: 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.694 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 librados: starting msgr at 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 librados: starting objecter 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 Processor -- start 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- start start 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 librados: setting wanted keys 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 librados: calling monclient init 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: init 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient: _add_conns ranks=[1,3,5,2,0,6,4,7,8] 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64080a5b80 0x7f64080a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): picked mon.noname-f con 0x7f64080a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6408138cd0 0x7f6408135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): picked mon.noname-g con 0x7f6408138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f64081397c0 0x7f64081361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): picked mon.noname-h con 0x7f64081397c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6408143730 con 0x7f64080a5b80 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f64080a20a0 con 0x7f6408138cd0 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f64081438b0 con 0x7f64081397c0 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.681+0000 7f64100a5640 10 monclient(hunting): authenticate will time out at 1925.811632s 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f64081397c0 0x7f64081361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64080a5b80 0x7f64080a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6408138cd0 0x7f6408135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f64081397c0 0x7f64081361b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:56648/0 (socket says 172.21.15.43:56648) 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 1 -- 172.21.15.43:0/1949724057 learned_addr learned my addr 172.21.15.43:0/1949724057 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:55.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): get_auth_request con 0x7f64081397c0 auth_method 0 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): get_auth_request con 0x7f64080a5b80 auth_method 0 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): get_auth_request con 0x7f6408138cd0 auth_method 0 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient(hunting): handle_auth_done global_id 19740 payload 824 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f64081397c0 msgr2=0x7f64081361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f64081397c0 0x7f64081361b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64080a5b80 msgr2=0x7f64080a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64080a5b80 0x7f64080a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 monclient: found mon.noname-g 2024-04-19T01:00:55.699 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64080a3590 con 0x7f6408138cd0 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f64081397c0 0x7f64081361b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640d640640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6408138cd0 0x7f6408135c70 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f640400e840 tx=0x7f640400ed00 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 client.?.objecter ms_handle_connect 0x7f6408138cd0 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6404017070 con 0x7f6408138cd0 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: dump: 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.700 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 5 monclient: authenticate success, global_id 19740 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 monclient: _renew_subs 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f640400c9a0 con 0x7f6408138cd0 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64080a3040 con 0x7f6408138cd0 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6404013470 con 0x7f6408138cd0 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fffff640 4 set_mon_vals no callback set 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 client.19740.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: dump: 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.702 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 monclient: _renew_subs 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64080a07e0 con 0x7f6408138cd0 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 1 librados: init done 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 0x7f640809ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f640806d140 con 0x7f640809c8f0 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 0x7f640809ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient: get_auth_request con 0x7f640809c8f0 auth_method 0 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 10 monclient(hunting): handle_auth_done global_id 144110 payload 824 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64080693b0 con 0x7f640809c8f0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640de41640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 0x7f640809ccd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000f70 tx=0x7f63f80019c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 client.19740.objecter ms_handle_connect 0x7f640809c8f0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 client.19740.objecter resend_mon_ops 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f640401a070 con 0x7f6408138cd0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f63ec038e40 0x7f63ec03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(610..610 src has 501..610) ==== 6709+0+0 (secure 0 0 0) 0x7f640400f040 con 0x7f6408138cd0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 client.19740.objecter ms_dispatch 0x7f6408000ba0 osd_map(610..610 src has 501..610) 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 3 client.19740.objecter handle_osd_map got epochs [610,610] > 0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 3 client.19740.objecter handle_osd_map decoding full epoch 610 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 20 client.19740.objecter dump_active .. 0 homeless 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63f8017020 con 0x7f640809c8f0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: dump: 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f63f8013600 con 0x7f640809c8f0 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 msgr2=0x7f640809ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 0x7f640809ccd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000f70 tx=0x7f63f80019c0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 msgr2=0x7f640809ccd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f63fe7fc640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809c8f0 0x7f640809ccd0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63f8000f70 tx=0x7f63f80019c0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f63ec038e40 0x7f63ec03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.685+0000 7f640e642640 10 monclient: get_auth_request con 0x7f63ec038e40 auth_method 0 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.689+0000 7f640e642640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f63ec038e40 0x7f63ec03b300 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7f640001c890 tx=0x7f64000a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.689+0000 7f63fe7fc640 10 client.19740.objecter ms_handle_connect 0x7f63ec038e40 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 0x7f640813bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6408068240 con 0x7f640809d210 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 0x7f640813bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient: get_auth_request con 0x7f640809d210 auth_method 0 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 10 monclient(hunting): handle_auth_done global_id 144119 payload 824 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 1 -- 172.21.15.43:0/1949724057 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64081366f0 con 0x7f640809d210 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 client.19740.objecter ms_handle_connect 0x7f640809d210 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 client.19740.objecter resend_mon_ops 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640de41640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 0x7f640813bcf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63f8001ec0 tx=0x7f63f8000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63f8017050 con 0x7f640809d210 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:55.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 monclient: dump: 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.711 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f63f8006250 con 0x7f640809d210 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 msgr2=0x7f640813bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 0x7f640813bcf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63f8001ec0 tx=0x7f63f8000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 msgr2=0x7f640813bcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f63fe7fc640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f640809d210 0x7f640813bcf0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f63f8001ec0 tx=0x7f63f8000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:55.712 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 20323, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.713 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.714 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.715 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:55.716 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.717 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.718 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:55.719 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.720 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.722 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.723 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.725 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:55.727 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:55.728 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:55.730 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:55.732 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:55.733 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:55.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 10 librados: watch_flush enter 2024-04-19T01:00:55.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 10 librados: watch_flush exit 2024-04-19T01:00:55.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 20 client.19740.objecter shutdown clearing up homeless session... 2024-04-19T01:00:55.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 10 client.19740.objecter successfully canceled tick 2024-04-19T01:00:55.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f63ec038e40 msgr2=0x7f63ec03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f63ec038e40 0x7f63ec03b300 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7f640001c890 tx=0x7f64000a7070 comp rx=0 tx=0).stop 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 10 monclient: shutdown 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6408138cd0 msgr2=0x7f6408135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 --2- 172.21.15.43:0/1949724057 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6408138cd0 0x7f6408135c70 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f640400e840 tx=0x7f640400ed00 comp rx=0 tx=0).stop 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640e642640 1 -- 172.21.15.43:0/1949724057 reap_dead start 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f640e642640 1 -- 172.21.15.43:0/1949724057 reap_dead start 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 shutdown_connections 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 >> 172.21.15.43:0/1949724057 conn(0x7f64080ab2f0 msgr2=0x7f64081377e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 shutdown_connections 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 -- 172.21.15.43:0/1949724057 wait complete. 2024-04-19T01:00:55.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.693+0000 7f64100a5640 1 librados: shutdown 2024-04-19T01:00:55.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:55.737 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T01:00:55.737 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.861+0000 7fdd6c220640 10 monclient: get_monmap_and_config 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.861+0000 7fdd6c220640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.861+0000 7fdd6c220640 10 monclient: monmap: 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:55.865257+0000 2024-04-19T01:00:55.862 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:55.865257+0000 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:55.863 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.861+0000 7fdd6c220640 1 Processor -- start 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.861+0000 7fdd6c220640 1 -- start start 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.861+0000 7fdd6c220640 10 monclient: init 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient: _add_conns ranks=[9,1,10,5,4,0,6,3,8,2,7] 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd64138cd0 0x7fdd641390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): picked mon.noname-a con 0x7fdd64138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:55.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdd640a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd641395f0 0x7fdd64135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): picked mon.noname-f con 0x7fdd641395f0 addr v2:172.21.15.161:3300/0 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd640a3dc0 con 0x7fdd640a5b80 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fdd640a4410 con 0x7fdd64138cd0 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdd640a40a0 con 0x7fdd641395f0 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient(hunting): authenticate will time out at 1925.992316s 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd64138cd0 0x7fdd641390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fdd64138cd0 0x7fdd641390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55566/0 (socket says 172.21.15.43:55566) 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 -- 172.21.15.43:0/2990283599 learned_addr learned my addr 172.21.15.43:0/2990283599 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): get_auth_request con 0x7fdd64138cd0 auth_method 0 2024-04-19T01:00:55.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6a7bd640 1 --2- 172.21.15.43:0/2990283599 >> v2:172.21.15.161:3300/0 conn(0x7fdd641395f0 0x7fdd64135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6a7bd640 10 monclient(hunting): get_auth_request con 0x7fdd641395f0 auth_method 0 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6a7bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6a7bd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6a7bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_done global_id 65340 payload 274 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 -- 172.21.15.43:0/2990283599 >> v2:172.21.15.161:3300/0 conn(0x7fdd641395f0 msgr2=0x7fdd64135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 --2- 172.21.15.43:0/2990283599 >> v2:172.21.15.161:3300/0 conn(0x7fdd641395f0 0x7fdd64135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 -- 172.21.15.43:0/2990283599 >> v1:172.21.15.161:6789/0 conn(0x7fdd640a5b80 legacy=0x7fdd641358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 monclient: found mon.noname-a 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:55.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 -- 172.21.15.43:0/2990283599 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd640a2ae0 con 0x7fdd64138cd0 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd69fbc640 1 --2- 172.21.15.43:0/2990283599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd64138cd0 0x7fdd641390b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd640a79a0 tx=0x7fdd5401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc38721beac8985 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 1 -- 172.21.15.43:0/2990283599 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd540ab870 con 0x7fdd64138cd0 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: dump: 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.877 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.878 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 5 monclient: authenticate success, global_id 65340 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 1 -- 172.21.15.43:0/2990283599 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd540b0070 con 0x7fdd64138cd0 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 4 set_mon_vals no callback set 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 1 -- 172.21.15.43:0/2990283599 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd540aa360 con 0x7fdd64138cd0 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd68fba640 10 monclient: dump: 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.879 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 10 monclient: shutdown 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 -- 172.21.15.43:0/2990283599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd64138cd0 msgr2=0x7fdd641390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 --2- 172.21.15.43:0/2990283599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd64138cd0 0x7fdd641390b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdd640a79a0 tx=0x7fdd5401c3e0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 -- 172.21.15.43:0/2990283599 shutdown_connections 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 --2- 172.21.15.43:0/2990283599 >> v2:172.21.15.161:3300/0 conn(0x7fdd641395f0 0x7fdd64135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 --2- 172.21.15.43:0/2990283599 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fdd64138cd0 0x7fdd641390b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.865+0000 7fdd6c220640 1 -- 172.21.15.43:0/2990283599 >> 172.21.15.43:0/2990283599 conn(0x7fdd640ab2f0 msgr2=0x7fdd640ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- 172.21.15.43:0/2990283599 shutdown_connections 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- 172.21.15.43:0/2990283599 wait complete. 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: build_initial_monmap 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: monmap: 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:55.881 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 librados: starting msgr at 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 librados: starting objecter 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 Processor -- start 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- start start 2024-04-19T01:00:55.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 librados: setting wanted keys 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 librados: calling monclient init 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: init 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: _add_conns ranks=[2,5,1,0,3,7,4,8,6] 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd640a5b80 0x7fdd641d0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): picked mon.noname-b con 0x7fdd640a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdd64138cd0 0x7fdd641d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): picked mon.noname-h con 0x7fdd64138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd641395f0 0x7fdd641d13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): picked mon.noname-f con 0x7fdd641395f0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd64136500 con 0x7fdd641395f0 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdd640a20a0 con 0x7fdd640a5b80 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdd64136680 con 0x7fdd64138cd0 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): authenticate will time out at 1925.997818s 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd640a5b80 0x7fdd641d0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd640a5b80 0x7fdd641d0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46182/0 (socket says 172.21.15.43:46182) 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 -- 172.21.15.43:0/1644345191 learned_addr learned my addr 172.21.15.43:0/1644345191 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): get_auth_request con 0x7fdd640a5b80 auth_method 0 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6a7bd640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd641395f0 0x7fdd641d13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdd64138cd0 0x7fdd641d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6a7bd640 10 monclient(hunting): get_auth_request con 0x7fdd641395f0 auth_method 0 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6a7bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6a7bd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6a7bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient(hunting): handle_auth_done global_id 144128 payload 824 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdd64138cd0 msgr2=0x7fdd641d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdd64138cd0 0x7fdd641d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd641395f0 msgr2=0x7fdd641d13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd641395f0 0x7fdd641d13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient: _finish_hunting 0 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 monclient: found mon.noname-b 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:55.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd640a1590 con 0x7fdd640a5b80 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient: get_auth_request con 0x7fdd64138cd0 auth_method 0 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdd64138cd0 0x7fdd641d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd69fbc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd640a5b80 0x7fdd641d0920 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd540ad040 tx=0x7fdd54019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 client.?.objecter ms_handle_connect 0x7fdd640a5b80 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd540ab400 con 0x7fdd640a5b80 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: dump: 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.886 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:55.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd540ab5e0 con 0x7fdd640a5b80 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd540abc00 con 0x7fdd640a5b80 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 10 monclient: dump: 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.888 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5b7fe640 4 set_mon_vals no callback set 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 5 monclient: authenticate success, global_id 144128 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: _renew_subs 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd6409fd40 con 0x7fdd640a5b80 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 client.144128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: _renew_subs 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:55.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd640a0d40 con 0x7fdd640a5b80 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 librados: init done 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 0x7fdd6412ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdd6409cac0 con 0x7fdd6412e840 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdd540aa070 con 0x7fdd640a5b80 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 0x7fdd6412ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient: get_auth_request con 0x7fdd6412e840 auth_method 0 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.869+0000 7fdd697bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd697bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdd48038e10 0x7fdd4803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd697bb640 10 monclient(hunting): handle_auth_done global_id 134338 payload 824 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd697bb640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd640a3590 con 0x7fdd6412e840 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(610..610 src has 501..610) ==== 6709+0+0 (secure 0 0 0) 0x7fdd540e3960 con 0x7fdd640a5b80 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 client.144128.objecter ms_dispatch 0x7fdd64000ba0 osd_map(610..610 src has 501..610) 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 3 client.144128.objecter handle_osd_map got epochs [610,610] > 0 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 3 client.144128.objecter handle_osd_map decoding full epoch 610 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 20 client.144128.objecter dump_active .. 0 homeless 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 client.144128.objecter ms_handle_connect 0x7fdd6412e840 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 client.144128.objecter resend_mon_ops 2024-04-19T01:00:55.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd697bb640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 0x7fdd6412ec20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdd60001ec0 tx=0x7fdd6000d950 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd600074a0 con 0x7fdd6412e840 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 monclient: dump: 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.892 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd6a7bd640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdd48038e10 0x7fdd4803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd6a7bd640 10 monclient: get_auth_request con 0x7fdd48038e10 auth_method 0 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdd60005280 con 0x7fdd6412e840 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 msgr2=0x7fdd6412ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 0x7fdd6412ec20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdd60001ec0 tx=0x7fdd6000d950 comp rx=0 tx=0).stop 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 msgr2=0x7fdd6412ec20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412e840 0x7fdd6412ec20 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdd60001ec0 tx=0x7fdd6000d950 comp rx=0 tx=0).stop 2024-04-19T01:00:55.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd6a7bd640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdd48038e10 0x7fdd4803b2d0 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7fdd640aacf0 tx=0x7fdd5c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.873+0000 7fdd5a7fc640 10 client.144128.objecter ms_handle_connect 0x7fdd48038e10 2024-04-19T01:00:55.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd6c220640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:55.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd6c220640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 0x7fdd64132ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:55.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd6c220640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:55.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fdd6406d140 con 0x7fdd6412f160 2024-04-19T01:00:55.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 0x7fdd64132ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient: get_auth_request con 0x7fdd6412f160 auth_method 0 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 10 monclient(hunting): handle_auth_done global_id 134347 payload 824 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 1 -- 172.21.15.43:0/1644345191 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd640693b0 con 0x7fdd6412f160 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd697bb640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 0x7fdd64132ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000fc0 tx=0x7fdd60000ee0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd5a7fc640 10 client.144128.objecter ms_handle_connect 0x7fdd6412f160 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd5a7fc640 10 client.144128.objecter resend_mon_ops 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdd60007570 con 0x7fdd6412f160 2024-04-19T01:00:55.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.877+0000 7fdd5a7fc640 10 monclient: dump: 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:55.897 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9614 (secure 0 0 0) 0x7fdd60007570 con 0x7fdd6412f160 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 msgr2=0x7fdd64132ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 0x7fdd64132ec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000fc0 tx=0x7fdd60000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 msgr2=0x7fdd64132ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd5a7fc640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fdd6412f160 0x7fdd64132ec0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000fc0 tx=0x7fdd60000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:55.898 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 160581, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:55.899 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.900 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.901 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:55.902 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:55.903 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.905 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:55.907 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:55.908 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.910 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:55.912 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.914 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.917 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:55.918 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.919 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:55.920 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 10 librados: watch_flush enter 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 10 librados: watch_flush exit 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 20 client.144128.objecter shutdown clearing up homeless session... 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 10 client.144128.objecter successfully canceled tick 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdd48038e10 msgr2=0x7fdd4803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdd48038e10 0x7fdd4803b2d0 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7fdd640aacf0 tx=0x7fdd5c00b040 comp rx=0 tx=0).stop 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 10 monclient: shutdown 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd640a5b80 msgr2=0x7fdd641d0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 --2- 172.21.15.43:0/1644345191 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdd640a5b80 0x7fdd641d0920 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdd540ad040 tx=0x7fdd54019180 comp rx=0 tx=0).stop 2024-04-19T01:00:55.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6a7bd640 1 -- 172.21.15.43:0/1644345191 reap_dead start 2024-04-19T01:00:55.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6a7bd640 1 -- 172.21.15.43:0/1644345191 reap_dead start 2024-04-19T01:00:55.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 shutdown_connections 2024-04-19T01:00:55.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 >> 172.21.15.43:0/1644345191 conn(0x7fdd640ab2f0 msgr2=0x7fdd64131b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:55.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 shutdown_connections 2024-04-19T01:00:55.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 -- 172.21.15.43:0/1644345191 wait complete. 2024-04-19T01:00:55.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:55.881+0000 7fdd6c220640 1 librados: shutdown 2024-04-19T01:00:55.923 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:55.923 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T01:00:55.923 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T01:00:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient: get_monmap_and_config 2024-04-19T01:00:56.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient: monmap: 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:56.050826+0000 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:56.050826+0000 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.048 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:56.049 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 Processor -- start 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 -- start start 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient: init 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient: _add_conns ranks=[0,7,9,10,6,1,8,3,2,5,4] 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7f4138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb7f40a5b80 0x7fb7f413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): picked mon.noname-i con 0x7fb7f40a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fb7f41397c0 0x7fb7f41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f41397c0 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f40a3dc0 con 0x7fb7f4138cd0 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f40a4410 con 0x7fb7f41397c0 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fb7f40a40a0 con 0x7fb7f40a5b80 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7fa24d640 10 monclient(hunting): authenticate will time out at 1926.178010s 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f37fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fb7f4138cd0 0x7fb7f41390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43060/0 (socket says 172.21.15.43:43060) 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f37fe640 1 -- 172.21.15.43:0/1239568146 learned_addr learned my addr 172.21.15.43:0/1239568146 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 1 --2- 172.21.15.43:0/1239568146 >> v2:172.21.15.43:3300/0 conn(0x7fb7f41397c0 0x7fb7f41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): get_auth_request con 0x7fb7f41397c0 auth_method 0 2024-04-19T01:00:56.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f2ffd640 1 --2- 172.21.15.43:0/1239568146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb7f40a5b80 0x7fb7f413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f2ffd640 10 monclient(hunting): get_auth_request con 0x7fb7f40a5b80 auth_method 0 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_done global_id 65349 payload 274 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 1 -- 172.21.15.43:0/1239568146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb7f40a5b80 msgr2=0x7fb7f413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 1 --2- 172.21.15.43:0/1239568146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb7f40a5b80 0x7fb7f413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 1 -- 172.21.15.43:0/1239568146 >> v1:172.21.15.43:6789/0 conn(0x7fb7f4138cd0 legacy=0x7fb7f41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 1 monclient: found mon.noname-a 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.049+0000 7fb7f3fff640 1 -- 172.21.15.43:0/1239568146 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f40a2ae0 con 0x7fb7f41397c0 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 1 --2- 172.21.15.43:0/1239568146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7f41397c0 0x7fb7f41407c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0097c0 tx=0x7fb7ec01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95bb9bda493084f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 1 -- 172.21.15.43:0/1239568146 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb7ec0b0070 con 0x7fb7f41397c0 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: dump: 2024-04-19T01:00:56.056 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.057 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 1 -- 172.21.15.43:0/1239568146 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0aa810 con 0x7fb7f41397c0 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 5 monclient: authenticate success, global_id 65349 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:56.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 1 -- 172.21.15.43:0/1239568146 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb7ec0aa3a0 con 0x7fb7f41397c0 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 4 set_mon_vals no callback set 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f27fc640 10 monclient: dump: 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.059 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.060 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: shutdown 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- 172.21.15.43:0/1239568146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7f41397c0 msgr2=0x7fb7f41407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/1239568146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7f41397c0 0x7fb7f41407c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0097c0 tx=0x7fb7ec01c360 comp rx=0 tx=0).stop 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- 172.21.15.43:0/1239568146 shutdown_connections 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/1239568146 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb7f41397c0 0x7fb7f41407c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/1239568146 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fb7f40a5b80 0x7fb7f413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- 172.21.15.43:0/1239568146 >> 172.21.15.43:0/1239568146 conn(0x7fb7f40ab2f0 msgr2=0x7fb7f40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- 172.21.15.43:0/1239568146 shutdown_connections 2024-04-19T01:00:56.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- 172.21.15.43:0/1239568146 wait complete. 2024-04-19T01:00:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: monmap: 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.070 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 librados: starting msgr at 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 librados: starting objecter 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 Processor -- start 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- start start 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 librados: setting wanted keys 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 librados: calling monclient init 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: init 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient: _add_conns ranks=[8,6,2,7,5,3,4,0,1] 2024-04-19T01:00:56.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb7f40a5b80 0x7fb7f40a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): picked mon.noname-e con 0x7fb7f40a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f4138cd0 0x7fb7f4135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): picked mon.noname-d con 0x7fb7f4138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7f41397c0 0x7fb7f41362d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f41397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f4143730 con 0x7fb7f41397c0 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb7f40a20a0 con 0x7fb7f4138cd0 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb7f41438b0 con 0x7fb7f40a5b80 2024-04-19T01:00:56.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7fa24d640 10 monclient(hunting): authenticate will time out at 1926.183636s 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7f41397c0 0x7fb7f41362d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f4138cd0 0x7fb7f4135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f37fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb7f40a5b80 0x7fb7f40a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7f41397c0 0x7fb7f41362d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46188/0 (socket says 172.21.15.43:46188) 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 1 -- 172.21.15.43:0/63074708 learned_addr learned my addr 172.21.15.43:0/63074708 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): get_auth_request con 0x7fb7f41397c0 auth_method 0 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): get_auth_request con 0x7fb7f4138cd0 auth_method 0 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f37fe640 10 monclient(hunting): get_auth_request con 0x7fb7f40a5b80 auth_method 0 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.053+0000 7fb7f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 10 monclient(hunting): handle_auth_done global_id 144137 payload 824 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb7f40a5b80 msgr2=0x7fb7f40a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb7f40a5b80 0x7fb7f40a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f4138cd0 msgr2=0x7fb7f4135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f4138cd0 0x7fb7f4135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 monclient: found mon.noname-b 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f40a3590 con 0x7fb7f41397c0 2024-04-19T01:00:56.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f3fff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7f41397c0 0x7fb7f41362d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec019f90 tx=0x7fb7ec019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 client.?.objecter ms_handle_connect 0x7fb7f41397c0 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb7ec0b0040 con 0x7fb7f41397c0 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: dump: 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.078 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0044b0 con 0x7fb7f41397c0 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 5 monclient: authenticate success, global_id 144137 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 monclient: _renew_subs 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f40a3040 con 0x7fb7f41397c0 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb7ec0aacc0 con 0x7fb7f41397c0 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 client.144137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: dump: 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.080 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.081 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 monclient: _renew_subs 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f1ffb640 4 set_mon_vals no callback set 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f409ff30 con 0x7fb7f41397c0 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 1 librados: init done 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 0x7fb7f409ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb7f406d140 con 0x7fb7f40a0100 2024-04-19T01:00:56.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb7ec004040 con 0x7fb7f41397c0 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 0x7fb7f409ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient: get_auth_request con 0x7fb7f40a0100 auth_method 0 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb7e0038e10 0x7fb7e003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(610..610 src has 501..610) ==== 6709+0+0 (secure 0 0 0) 0x7fb7ec0b9070 con 0x7fb7f41397c0 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 client.144137.objecter ms_dispatch 0x7fb7f4000ba0 osd_map(610..610 src has 501..610) 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 3 client.144137.objecter handle_osd_map got epochs [610,610] > 0 2024-04-19T01:00:56.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 3 client.144137.objecter handle_osd_map decoding full epoch 610 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_done global_id 144123 payload 824 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7f40693b0 con 0x7fb7f40a0100 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 20 client.144137.objecter dump_active .. 0 homeless 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f37fe640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 0x7fb7f409ccf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400f770 tx=0x7fb7e400fc30 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 client.144137.objecter ms_handle_connect 0x7fb7f40a0100 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 client.144137.objecter resend_mon_ops 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb7e4005cd0 con 0x7fb7f40a0100 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f2ffd640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb7e0038e10 0x7fb7e003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: dump: 2024-04-19T01:00:56.084 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.085 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f2ffd640 10 monclient: get_auth_request con 0x7fb7e0038e10 auth_method 0 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb7e4018070 con 0x7fb7f40a0100 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 msgr2=0x7fb7f409ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 0x7fb7f409ccf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400f770 tx=0x7fb7e400fc30 comp rx=0 tx=0).stop 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 msgr2=0x7fb7f409ccf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f40a0100 0x7fb7f409ccf0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400f770 tx=0x7fb7e400fc30 comp rx=0 tx=0).stop 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7dbfff640 10 client.144137.objecter ms_handle_connect 0x7fb7e0038e10 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.057+0000 7fb7f2ffd640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb7e0038e10 0x7fb7e003b2d0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7fb7f409dd10 tx=0x7fb7e80098d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 0x7fb7f402c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb7f4068240 con 0x7fb7f402c450 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 0x7fb7f402c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient: get_auth_request con 0x7fb7f402c450 auth_method 0 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 10 monclient(hunting): handle_auth_done global_id 144132 payload 824 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 1 -- 172.21.15.43:0/63074708 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7f413d4c0 con 0x7fb7f402c450 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f37fe640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 0x7fb7f402c860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400fc70 tx=0x7fb7e4000ee0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 client.144137.objecter ms_handle_connect 0x7fb7f402c450 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 client.144137.objecter resend_mon_ops 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb7e4007540 con 0x7fb7f402c450 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 monclient: dump: 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.088 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.089 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fb7e40070d0 con 0x7fb7f402c450 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 msgr2=0x7fb7f402c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 0x7fb7f402c860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400fc70 tx=0x7fb7e4000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 msgr2=0x7fb7f402c860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7dbfff640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb7f402c450 0x7fb7f402c860 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400fc70 tx=0x7fb7e4000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 20695, 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:56.091 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.092 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:56.093 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.094 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:56.095 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.096 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.098 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.102 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.103 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.104 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.106 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:56.107 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.110 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.111 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:56.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.116 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:56.116 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 10 librados: watch_flush enter 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 10 librados: watch_flush exit 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 20 client.144137.objecter shutdown clearing up homeless session... 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 10 client.144137.objecter successfully canceled tick 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb7e0038e10 msgr2=0x7fb7e003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb7e0038e10 0x7fb7e003b2d0 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7fb7f409dd10 tx=0x7fb7e80098d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 10 monclient: shutdown 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7f41397c0 msgr2=0x7fb7f41362d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 --2- 172.21.15.43:0/63074708 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb7f41397c0 0x7fb7f41362d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec019f90 tx=0x7fb7ec019100 comp rx=0 tx=0).stop 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f3fff640 1 -- 172.21.15.43:0/63074708 reap_dead start 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7f3fff640 1 -- 172.21.15.43:0/63074708 reap_dead start 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 shutdown_connections 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 >> 172.21.15.43:0/63074708 conn(0x7fb7f40ab2f0 msgr2=0x7fb7f41376a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 shutdown_connections 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 -- 172.21.15.43:0/63074708 wait complete. 2024-04-19T01:00:56.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.065+0000 7fb7fa24d640 1 librados: shutdown 2024-04-19T01:00:56.119 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:56.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T01:00:56.120 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient: get_monmap_and_config 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient: monmap: 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:56.246595+0000 2024-04-19T01:00:56.243 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:56.246595+0000 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:56.244 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 Processor -- start 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 -- start start 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient: init 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient: _add_conns ranks=[4,7,3,9,5,2,6,8,10,1,0] 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 0x7f17f40727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): picked mon.noname-c con 0x7f17f40a3370 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f17f40a3750 0x7f17f40a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): picked mon.noname-i con 0x7f17f40a3750 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17f4135120 0x7f17f41355a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): picked mon.noname-g con 0x7f17f4135120 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f17f4071820 con 0x7f17f4135120 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f17f4070220 con 0x7f17f40a3370 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f17f4071540 con 0x7f17f40a3750 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17fa99d640 10 monclient(hunting): authenticate will time out at 1926.373878s 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 0x7f17f40727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 0x7f17f40727c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52772/0 (socket says 172.21.15.43:52772) 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 -- 172.21.15.43:0/1431436901 learned_addr learned my addr 172.21.15.43:0/1431436901 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): get_auth_request con 0x7f17f40a3370 auth_method 0 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f8f3a640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17f4135120 0x7f17f41355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f37fe640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f17f40a3750 0x7f17f40a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f8f3a640 10 monclient(hunting): get_auth_request con 0x7f17f4135120 auth_method 0 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f8f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f8f3a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f8f3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f37fe640 10 monclient(hunting): get_auth_request con 0x7f17f40a3750 auth_method 0 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient(hunting): handle_auth_done global_id 134356 payload 274 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 -- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f17f40a3750 msgr2=0x7f17f40a3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f17f40a3750 0x7f17f40a3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 -- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17f4135120 msgr2=0x7f17f41355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17f4135120 0x7f17f41355a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 monclient: found mon.noname-c 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f3fff640 1 -- 172.21.15.43:0/1431436901 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f4072d00 con 0x7f17f40a3370 2024-04-19T01:00:56.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.245+0000 7f17f8f3a640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17f4135120 0x7f17f41355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:00:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f3fff640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 0x7f17f40727c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009eb0 tx=0x7f17ec01c980 comp rx=0 tx=0).ready entity=mon.4 client_cookie=215dac8e14613092 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 1 -- 172.21.15.43:0/1431436901 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17ec0090d0 con 0x7f17f40a3370 2024-04-19T01:00:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:00:56.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: dump: 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.258 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 1 -- 172.21.15.43:0/1431436901 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17ec0a5a60 con 0x7f17f40a3370 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 5 monclient: authenticate success, global_id 134356 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 4 set_mon_vals no callback set 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 1 -- 172.21.15.43:0/1431436901 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17ec0a5c50 con 0x7f17f40a3370 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17f2ffd640 10 monclient: dump: 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.260 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 10 monclient: shutdown 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 -- 172.21.15.43:0/1431436901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 msgr2=0x7f17f40727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 0x7f17f40727c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009eb0 tx=0x7f17ec01c980 comp rx=0 tx=0).stop 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 -- 172.21.15.43:0/1431436901 shutdown_connections 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f17f4135120 0x7f17f41355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f17f40a3750 0x7f17f40a3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 --2- 172.21.15.43:0/1431436901 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3370 0x7f17f40727c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 -- 172.21.15.43:0/1431436901 >> 172.21.15.43:0/1431436901 conn(0x7f17f40a4460 msgr2=0x7f17f40a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 -- 172.21.15.43:0/1431436901 shutdown_connections 2024-04-19T01:00:56.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 -- 172.21.15.43:0/1431436901 wait complete. 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 10 monclient: monmap: 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:56.262 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 librados: starting msgr at 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 librados: starting objecter 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.249+0000 7f17fa99d640 1 Processor -- start 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- start start 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 librados: setting wanted keys 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 librados: calling monclient init 2024-04-19T01:00:56.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: init 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: _add_conns ranks=[1,4,6,2,8,0,5,3,7] 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17f40a3370 0x7f17f409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): picked mon.noname-f con 0x7f17f40a3370 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 0x7f17f40a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): picked mon.noname-c con 0x7f17f40a3750 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17f4135120 0x7f17f409f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): picked mon.noname-d con 0x7f17f4135120 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f17f413c700 con 0x7f17f40a3370 2024-04-19T01:00:56.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f17f413c580 con 0x7f17f40a3750 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f17f413c880 con 0x7f17f4135120 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): authenticate will time out at 1926.380764s 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 0x7f17f40a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 0x7f17f40a0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52776/0 (socket says 172.21.15.43:52776) 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 -- 172.21.15.43:0/169658392 learned_addr learned my addr 172.21.15.43:0/169658392 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): get_auth_request con 0x7f17f40a3750 auth_method 0 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17f4135120 0x7f17f409f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): get_auth_request con 0x7f17f4135120 auth_method 0 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17f40a3370 0x7f17f409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): get_auth_request con 0x7f17f40a3370 auth_method 0 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient(hunting): handle_auth_done global_id 134365 payload 824 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17f4135120 msgr2=0x7f17f409f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f17f4135120 0x7f17f409f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17f40a3370 msgr2=0x7f17f409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f17f40a3370 0x7f17f409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 monclient: found mon.noname-c 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f40a03e0 con 0x7f17f40a3750 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f37fe640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 0x7f17f40a0f40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17e800d7d0 tx=0x7f17e800dc90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 client.?.objecter ms_handle_connect 0x7f17f40a3750 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:56.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e80044a0 con 0x7f17f40a3750 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: dump: 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.268 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e800b660 con 0x7f17f40a3750 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 5 monclient: authenticate success, global_id 134365 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17e800bc80 con 0x7f17f40a3750 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: dump: 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.270 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f27fc640 4 set_mon_vals no callback set 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: _renew_subs 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f411e7f0 con 0x7f17f40a3750 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 client.134365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: _renew_subs 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f41347d0 con 0x7f17f40a3750 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 librados: init done 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 0x7f17f41cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f17f406d140 con 0x7f17f409f8e0 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f17e800b810 con 0x7f17f40a3750 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 0x7f17f41cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient: get_auth_request con 0x7f17f409f8e0 auth_method 0 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17e0038e10 0x7f17e003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(610..610 src has 501..610) ==== 6709+0+0 (secure 0 0 0) 0x7f17e800f040 con 0x7f17f40a3750 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 client.134365.objecter ms_dispatch 0x7f17f4000ba0 osd_map(610..610 src has 501..610) 2024-04-19T01:00:56.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 3 client.134365.objecter handle_osd_map got epochs [610,610] > 0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 3 client.134365.objecter handle_osd_map decoding full epoch 610 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 20 client.134365.objecter dump_active .. 0 homeless 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 10 monclient(hunting): handle_auth_done global_id 144125 payload 824 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17f4070750 con 0x7f17f409f8e0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17e0038e10 0x7f17e003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 client.134365.objecter ms_handle_connect 0x7f17f409f8e0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 client.134365.objecter resend_mon_ops 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f3fff640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 0x7f17f41cc6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009e80 tx=0x7f17ec019720 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f8f3a640 10 monclient: get_auth_request con 0x7f17e0038e10 auth_method 0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17ec0bd030 con 0x7f17f409f8e0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.253+0000 7f17f0ff9640 10 monclient: dump: 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.273 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17ec004040 con 0x7f17f409f8e0 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 msgr2=0x7f17f41cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 0x7f17f41cc6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009e80 tx=0x7f17ec019720 comp rx=0 tx=0).stop 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 msgr2=0x7f17f41cc6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f409f8e0 0x7f17f41cc6b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f0ff9640 10 client.134365.objecter ms_handle_connect 0x7f17e0038e10 2024-04-19T01:00:56.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.257+0000 7f17f8f3a640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17e0038e10 0x7f17e003b2d0 secure :-1 s=READY pgs=3596 cs=0 l=1 rev1=1 crypto rx=0x7f17f406c9c0 tx=0x7f17e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 0x7f17f41cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f17f41cd540 con 0x7f17f41ccbf0 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 0x7f17f41cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient: get_auth_request con 0x7f17f41ccbf0 auth_method 0 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 10 monclient(hunting): handle_auth_done global_id 144134 payload 824 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 1 -- 172.21.15.43:0/169658392 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17f4073010 con 0x7f17f41ccbf0 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 client.134365.objecter ms_handle_connect 0x7f17f41ccbf0 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 client.134365.objecter resend_mon_ops 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f3fff640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 0x7f17f41cd000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17ec01c9c0 tx=0x7f17ec0a7000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17ec0bd070 con 0x7f17f41ccbf0 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 monclient: dump: 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.278 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f17ec0a5b20 con 0x7f17f41ccbf0 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:56.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 msgr2=0x7f17f41cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 0x7f17f41cd000 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17ec01c9c0 tx=0x7f17ec0a7000 comp rx=0 tx=0).stop 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 msgr2=0x7f17f41cd000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17f0ff9640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17f41ccbf0 0x7f17f41cd000 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f17ec01c9c0 tx=0x7f17ec0a7000 comp rx=0 tx=0).stop 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 20891, 2024-04-19T01:00:56.280 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:56.281 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:56.282 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:56.283 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.284 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.285 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:56.286 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.288 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:56.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.296 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:56.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.299 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.300 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:56.301 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 10 librados: watch_flush enter 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 10 librados: watch_flush exit 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 20 client.134365.objecter shutdown clearing up homeless session... 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.261+0000 7f17fa99d640 10 client.134365.objecter successfully canceled tick 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17e0038e10 msgr2=0x7f17e003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f17e0038e10 0x7f17e003b2d0 secure :-1 s=READY pgs=3596 cs=0 l=1 rev1=1 crypto rx=0x7f17f406c9c0 tx=0x7f17e4008040 comp rx=0 tx=0).stop 2024-04-19T01:00:56.303 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 10 monclient: shutdown 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 msgr2=0x7f17f40a0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 0x7f17f40a0f40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17e800d7d0 tx=0x7f17e800dc90 comp rx=0 tx=0).stop 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17f8f3a640 1 -- 172.21.15.43:0/169658392 reap_dead start 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 shutdown_connections 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 --2- 172.21.15.43:0/169658392 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f17f40a3750 0x7f17f40a0f40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 >> 172.21.15.43:0/169658392 conn(0x7f17f40a4460 msgr2=0x7f17f40a5760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 shutdown_connections 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 -- 172.21.15.43:0/169658392 wait complete. 2024-04-19T01:00:56.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.265+0000 7f17fa99d640 1 librados: shutdown 2024-04-19T01:00:56.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:56.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T01:00:56.305 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:00:56.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient: get_monmap_and_config 2024-04-19T01:00:56.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient: monmap: 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:56.434731+0000 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:56.434731+0000 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.432 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.433 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.433 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.433 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.433 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:56.433 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:56.433 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 Processor -- start 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 -- start start 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient: init 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient: _add_conns ranks=[5,9,10,3,8,1,0,7,4,2,6] 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d98138a50 0x7f1d98138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): picked mon.noname-h con 0x7f1d98138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1d980a5bb0 0x7f1d98139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): picked mon.noname-a con 0x7f1d980a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1d98137960 0x7f1d98137e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): picked mon.noname-f con 0x7f1d98137960 addr v2:172.21.15.161:3300/0 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f1d980a1020 con 0x7f1d980a5bb0 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1d981398b0 con 0x7f1d98137960 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f1d980a0780 con 0x7f1d98138a50 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9d20d640 10 monclient(hunting): authenticate will time out at 1926.561806s 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1d980a5bb0 0x7f1d98139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f1d980a5bb0 0x7f1d98139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55594/0 (socket says 172.21.15.43:55594) 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 -- 172.21.15.43:0/1252388925 learned_addr learned my addr 172.21.15.43:0/1252388925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): get_auth_request con 0x7f1d980a5bb0 auth_method 0 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d96d9d640 1 --2- 172.21.15.43:0/1252388925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d98138a50 0x7f1d98138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9759e640 1 --2- 172.21.15.43:0/1252388925 >> v2:172.21.15.161:3300/0 conn(0x7f1d98137960 0x7f1d98137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d96d9d640 10 monclient(hunting): get_auth_request con 0x7f1d98138a50 auth_method 0 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient(hunting): handle_auth_done global_id 65367 payload 274 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 -- 172.21.15.43:0/1252388925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d98138a50 msgr2=0x7f1d98138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 --2- 172.21.15.43:0/1252388925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d98138a50 0x7f1d98138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 -- 172.21.15.43:0/1252388925 >> v2:172.21.15.161:3300/0 conn(0x7f1d98137960 msgr2=0x7f1d98137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 --2- 172.21.15.43:0/1252388925 >> v2:172.21.15.161:3300/0 conn(0x7f1d98137960 0x7f1d98137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 monclient: found mon.noname-a 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9659c640 1 -- 172.21.15.43:0/1252388925 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d980a0d40 con 0x7f1d980a5bb0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9759e640 10 monclient: get_auth_request con 0x7f1d98137960 auth_method 0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.433+0000 7f1d9759e640 1 --2- 172.21.15.43:0/1252388925 >> v2:172.21.15.161:3300/0 conn(0x7f1d98137960 0x7f1d98137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 1 --2- 172.21.15.43:0/1252388925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d980a5bb0 0x7f1d98139370 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1d980a7160 tx=0x7f1d8801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1efa95daaf1ec3bc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 1 -- 172.21.15.43:0/1252388925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d880090d0 con 0x7f1d980a5bb0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: dump: 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.447 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 1 -- 172.21.15.43:0/1252388925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d880a5b90 con 0x7f1d980a5bb0 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 1 -- 172.21.15.43:0/1252388925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d880a5d80 con 0x7f1d980a5bb0 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 5 monclient: authenticate success, global_id 65367 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 4 set_mon_vals no callback set 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:56.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d95d9b640 10 monclient: dump: 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.449 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: shutdown 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1252388925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d980a5bb0 msgr2=0x7f1d98139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1252388925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d980a5bb0 0x7f1d98139370 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1d980a7160 tx=0x7f1d8801c880 comp rx=0 tx=0).stop 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1252388925 shutdown_connections 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1252388925 >> v2:172.21.15.161:3300/0 conn(0x7f1d98137960 0x7f1d98137e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1252388925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f1d980a5bb0 0x7f1d98139370 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1252388925 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d98138a50 0x7f1d98138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1252388925 >> 172.21.15.43:0/1252388925 conn(0x7f1d980a7cb0 msgr2=0x7f1d980a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1252388925 shutdown_connections 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1252388925 wait complete. 2024-04-19T01:00:56.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: monmap: 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.451 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 librados: starting msgr at 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 librados: starting objecter 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 Processor -- start 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- start start 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 librados: setting wanted keys 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 librados: calling monclient init 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: init 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: _add_conns ranks=[7,2,6,4,3,8,0,5,1] 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d980a5bb0 0x7f1d981d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): picked mon.noname-i con 0x7f1d980a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:56.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d98137960 0x7f1d98136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): picked mon.noname-b con 0x7f1d98137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 0x7f1d9813ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): picked mon.noname-d con 0x7f1d98138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d980a2100 con 0x7f1d98137960 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1d980a1f80 con 0x7f1d98138a50 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1d98142810 con 0x7f1d980a5bb0 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient(hunting): authenticate will time out at 1926.566750s 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d98137960 0x7f1d98136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 0x7f1d9813ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d98137960 0x7f1d98136ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46196/0 (socket says 172.21.15.43:46196) 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 0x7f1d9813ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53824/0 (socket says 172.21.15.43:53824) 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 1 -- 172.21.15.43:0/1453643837 learned_addr learned my addr 172.21.15.43:0/1453643837 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): get_auth_request con 0x7f1d98137960 auth_method 0 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): get_auth_request con 0x7f1d98138a50 auth_method 0 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d980a5bb0 0x7f1d981d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): get_auth_request con 0x7f1d980a5bb0 auth_method 0 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient(hunting): handle_auth_done global_id 144150 payload 824 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d980a5bb0 msgr2=0x7f1d981d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d980a5bb0 0x7f1d981d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d98137960 msgr2=0x7f1d98136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d98137960 0x7f1d98136ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 monclient: found mon.noname-d 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d981221a0 con 0x7f1d98138a50 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9659c640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d98137960 0x7f1d98136ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9759e640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 0x7f1d9813ca80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000c5e0 tx=0x7f1d9000caa0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 client.?.objecter ms_handle_connect 0x7f1d98138a50 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d90015070 con 0x7f1d98138a50 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: dump: 2024-04-19T01:00:56.456 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.457 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d90011a70 con 0x7f1d98138a50 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 5 monclient: authenticate success, global_id 144150 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: _renew_subs 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d98122480 con 0x7f1d98138a50 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d90011c60 con 0x7f1d98138a50 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9559a640 4 set_mon_vals no callback set 2024-04-19T01:00:56.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 client.144150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d8f7fe640 10 monclient: dump: 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.459 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: _renew_subs 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.437+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9809ee60 con 0x7f1d98138a50 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9d20d640 1 librados: init done 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9d20d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 0x7f1d9813d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d98121be0 con 0x7f1d9813cfc0 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1d90018070 con 0x7f1d98138a50 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d78038e40 0x7f1d7803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(611..611 src has 501..611) ==== 6709+0+0 (secure 0 0 0) 0x7f1d90049480 con 0x7f1d98138a50 2024-04-19T01:00:56.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 client.144150.objecter ms_dispatch 0x7f1d98000ba0 osd_map(611..611 src has 501..611) 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 3 client.144150.objecter handle_osd_map got epochs [611,611] > 0 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 3 client.144150.objecter handle_osd_map decoding full epoch 611 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 0x7f1d9813d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 20 client.144150.objecter dump_active .. 0 homeless 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient: get_auth_request con 0x7f1d9813cfc0 auth_method 0 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9659c640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d78038e40 0x7f1d7803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9659c640 10 monclient: get_auth_request con 0x7f1d78038e40 auth_method 0 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_done global_id 144163 payload 824 2024-04-19T01:00:56.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d98121d90 con 0x7f1d9813cfc0 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 client.144150.objecter ms_handle_connect 0x7f1d9813cfc0 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d96d9d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 0x7f1d9813d3a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d80010820 tx=0x7f1d80010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 client.144150.objecter resend_mon_ops 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 client.144150.objecter ms_handle_connect 0x7f1d78038e40 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d9659c640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d78038e40 0x7f1d7803b300 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 crypto rx=0x7f1d88000f90 tx=0x7f1d880a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d80018020 con 0x7f1d9813cfc0 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 monclient: dump: 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.462 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d80004240 con 0x7f1d9813cfc0 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:56.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 msgr2=0x7f1d9813d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 0x7f1d9813d3a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d80010820 tx=0x7f1d80010ce0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 msgr2=0x7f1d9813d3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.441+0000 7f1d8f7fe640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d9813cfc0 0x7f1d9813d3a0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1d80010820 tx=0x7f1d80010ce0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d9d20d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 0x7f1d980a3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d9d20d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9806d140 con 0x7f1d980a3060 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 0x7f1d980a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient: get_auth_request con 0x7f1d980a3060 auth_method 0 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 10 monclient(hunting): handle_auth_done global_id 144172 payload 824 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 1 -- 172.21.15.43:0/1453643837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d9813d8e0 con 0x7f1d980a3060 2024-04-19T01:00:56.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d96d9d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 0x7f1d980a3470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d80019000 tx=0x7f1d800046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d8f7fe640 10 client.144150.objecter ms_handle_connect 0x7f1d980a3060 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d8f7fe640 10 client.144150.objecter resend_mon_ops 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d80018040 con 0x7f1d980a3060 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d8f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.449+0000 7f1d8f7fe640 10 monclient: dump: 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.467 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f1d8000ebc0 con 0x7f1d980a3060 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 msgr2=0x7f1d980a3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 0x7f1d980a3470 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d80019000 tx=0x7f1d800046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 msgr2=0x7f1d980a3470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d8f7fe640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d980a3060 0x7f1d980a3470 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 21079, 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:56.469 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:56.470 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:56.471 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.472 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:56.473 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.474 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:56.477 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:56.478 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.479 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.484 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.485 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.486 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.487 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.488 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.490 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.491 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.492 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:56.493 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:56.494 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 10 librados: watch_flush enter 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 10 librados: watch_flush exit 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 20 client.144150.objecter shutdown clearing up homeless session... 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 10 client.144150.objecter successfully canceled tick 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d78038e40 msgr2=0x7f1d7803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d78038e40 0x7f1d7803b300 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 crypto rx=0x7f1d88000f90 tx=0x7f1d880a7070 comp rx=0 tx=0).stop 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 10 monclient: shutdown 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 msgr2=0x7f1d9813ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9759e640 1 -- 172.21.15.43:0/1453643837 reap_dead start 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 0x7f1d9813ca80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000c5e0 tx=0x7f1d9000caa0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 shutdown_connections 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 --2- 172.21.15.43:0/1453643837 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1d98138a50 0x7f1d9813ca80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 >> 172.21.15.43:0/1453643837 conn(0x7f1d980a7cb0 msgr2=0x7f1d980a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 shutdown_connections 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 -- 172.21.15.43:0/1453643837 wait complete. 2024-04-19T01:00:56.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.453+0000 7f1d9d20d640 1 librados: shutdown 2024-04-19T01:00:56.497 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:56.497 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T01:00:56.498 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.617+0000 7f4cf7559640 10 monclient: get_monmap_and_config 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.617+0000 7f4cf7559640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.617+0000 7f4cf7559640 10 monclient: monmap: 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:56.621228+0000 2024-04-19T01:00:56.618 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:56.621228+0000 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.619 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:56.620 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:56.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.617+0000 7f4cf7559640 1 Processor -- start 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.617+0000 7f4cf7559640 1 -- start start 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.617+0000 7f4cf7559640 10 monclient: init 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient: _add_conns ranks=[6,2,8,10,4,1,7,5,0,9,3] 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4cf0138a50 0x7f4cf0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): picked mon.noname-d con 0x7f4cf0138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cf00a5bb0 0x7f4cf0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): picked mon.noname-b con 0x7f4cf00a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:56.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 0x7f4cf0137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): picked mon.noname-e con 0x7f4cf0137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf00a1020 con 0x7f4cf00a5bb0 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f4cf01398b0 con 0x7f4cf0138a50 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4cf00a0780 con 0x7f4cf0137960 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient(hunting): authenticate will time out at 1926.748333s 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 0x7f4cf0137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 0x7f4cf0137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39012/0 (socket says 172.21.15.43:39012) 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 -- 172.21.15.43:0/629015111 learned_addr learned my addr 172.21.15.43:0/629015111 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): get_auth_request con 0x7f4cf0137960 auth_method 0 2024-04-19T01:00:56.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4cf0138a50 0x7f4cf0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cf00a5bb0 0x7f4cf0139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): get_auth_request con 0x7f4cf0138a50 auth_method 0 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): get_auth_request con 0x7f4cf00a5bb0 auth_method 0 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf4af4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_done global_id 144143 payload 274 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 -- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4cf0138a50 msgr2=0x7f4cf0138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4cf0138a50 0x7f4cf0138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 -- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cf00a5bb0 msgr2=0x7f4cf0139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cf00a5bb0 0x7f4cf0139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 monclient: found mon.noname-e 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 -- 172.21.15.43:0/629015111 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf00a0d40 con 0x7f4cf0137960 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf5af6640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 0x7f4cf0137dc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009e80 tx=0x7f4ce801c8e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=49cd86570128ac73 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 1 -- 172.21.15.43:0/629015111 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ce80090d0 con 0x7f4cf0137960 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: dump: 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.628 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 1 -- 172.21.15.43:0/629015111 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce80a5b50 con 0x7f4cf0137960 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 5 monclient: authenticate success, global_id 144143 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 1 -- 172.21.15.43:0/629015111 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ce80a5d40 con 0x7f4cf0137960 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 4 set_mon_vals no callback set 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4ce7fff640 10 monclient: dump: 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.630 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 10 monclient: shutdown 2024-04-19T01:00:56.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 -- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 msgr2=0x7f4cf0137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.621+0000 7f4cf7559640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 0x7f4cf0137dc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009e80 tx=0x7f4ce801c8e0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/629015111 shutdown_connections 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0137960 0x7f4cf0137dc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4cf00a5bb0 0x7f4cf0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- 172.21.15.43:0/629015111 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4cf0138a50 0x7f4cf0138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/629015111 >> 172.21.15.43:0/629015111 conn(0x7f4cf00a7cb0 msgr2=0x7f4cf00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/629015111 shutdown_connections 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/629015111 wait complete. 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: monmap: 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:56.632 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 librados: starting msgr at 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 librados: starting objecter 2024-04-19T01:00:56.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 Processor -- start 2024-04-19T01:00:56.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- start start 2024-04-19T01:00:56.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 librados: setting wanted keys 2024-04-19T01:00:56.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 librados: calling monclient init 2024-04-19T01:00:56.634 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: init 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: _add_conns ranks=[7,3,8,5,6,4,2,0,1] 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4cf00a5bb0 0x7f4cf00a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): picked mon.noname-i con 0x7f4cf00a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0137960 0x7f4cf0135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): picked mon.noname-g con 0x7f4cf0137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0138a50 0x7f4cf01360c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): picked mon.noname-e con 0x7f4cf0138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf00a2100 con 0x7f4cf0137960 2024-04-19T01:00:56.640 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f4cf00a1f80 con 0x7f4cf00a5bb0 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4cf0142810 con 0x7f4cf0138a50 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): authenticate will time out at 1926.753668s 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0138a50 0x7f4cf01360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0138a50 0x7f4cf01360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39022/0 (socket says 172.21.15.43:39022) 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 -- 172.21.15.43:0/1792446573 learned_addr learned my addr 172.21.15.43:0/1792446573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): get_auth_request con 0x7f4cf0138a50 auth_method 0 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.641 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient(hunting): handle_auth_done global_id 144152 payload 824 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4cf00a5bb0 msgr2=0x7f4cf00a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4cf00a5bb0 0x7f4cf00a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0137960 msgr2=0x7f4cf0135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf4af4640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0137960 0x7f4cf0135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0137960 0x7f4cf0135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 monclient: found mon.noname-e 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf009fc90 con 0x7f4cf0138a50 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf5af6640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0138a50 0x7f4cf01360c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009e80 tx=0x7f4ce8019680 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 client.?.objecter ms_handle_connect 0x7f4cf0138a50 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ce80bc020 con 0x7f4cf0138a50 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: dump: 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.643 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 5 monclient: authenticate success, global_id 144152 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ce8003e10 con 0x7f4cf0138a50 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ce80090d0 con 0x7f4cf0138a50 2024-04-19T01:00:56.644 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce77fe640 4 set_mon_vals no callback set 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4ce5ffb640 10 monclient: dump: 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.645 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: _renew_subs 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf009ee60 con 0x7f4cf0138a50 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 client.144152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: _renew_subs 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf00a50d0 con 0x7f4cf0138a50 2024-04-19T01:00:56.646 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 librados: init done 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 0x7f4cf01d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.625+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf006d140 con 0x7f4cf009c400 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4ce80bf020 con 0x7f4cf0138a50 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cd4038e10 0x7f4cd403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(611..611 src has 501..611) ==== 6709+0+0 (secure 0 0 0) 0x7f4ce80e4480 con 0x7f4cf0138a50 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 client.144152.objecter ms_dispatch 0x7f4cf0000ba0 osd_map(611..611 src has 501..611) 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 3 client.144152.objecter handle_osd_map got epochs [611,611] > 0 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 3 client.144152.objecter handle_osd_map decoding full epoch 611 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 0x7f4cf01d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 20 client.144152.objecter dump_active .. 0 homeless 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient: get_auth_request con 0x7f4cf009c400 auth_method 0 2024-04-19T01:00:56.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf4af4640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cd4038e10 0x7f4cd403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf4af4640 10 monclient: get_auth_request con 0x7f4cd4038e10 auth_method 0 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_done global_id 19758 payload 824 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cf009ba70 con 0x7f4cf009c400 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 client.144152.objecter ms_handle_connect 0x7f4cf009c400 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 client.144152.objecter resend_mon_ops 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf52f5640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 0x7f4cf01d08a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e7a0 tx=0x7f4ce000ec60 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 client.144152.objecter ms_handle_connect 0x7f4cd4038e10 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4cf4af4640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cd4038e10 0x7f4cd403b2d0 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00a55b0 tx=0x7f4cec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ce0017070 con 0x7f4cf009c400 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 monclient: dump: 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.649 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ce000c720 con 0x7f4cf009c400 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 msgr2=0x7f4cf01d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 0x7f4cf01d08a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e7a0 tx=0x7f4ce000ec60 comp rx=0 tx=0).stop 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 msgr2=0x7f4cf01d08a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.629+0000 7f4ce5ffb640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf009c400 0x7f4cf01d08a0 secure :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f4ce000e7a0 tx=0x7f4ce000ec60 comp rx=0 tx=0).stop 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:56.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 0x7f4cf01d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4cf0076190 con 0x7f4cf0136600 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 0x7f4cf01d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient: get_auth_request con 0x7f4cf0136600 auth_method 0 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 10 monclient(hunting): handle_auth_done global_id 19767 payload 824 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 1 -- 172.21.15.43:0/1792446573 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4cf0068240 con 0x7f4cf0136600 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf52f5640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 0x7f4cf01d0de0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0000c10 tx=0x7f4ce00046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 client.144152.objecter ms_handle_connect 0x7f4cf0136600 2024-04-19T01:00:56.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 client.144152.objecter resend_mon_ops 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4ce0017070 con 0x7f4cf0136600 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 monclient: dump: 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.652 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9277 (secure 0 0 0) 0x7f4ce0004300 con 0x7f4cf0136600 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 msgr2=0x7f4cf01d0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 0x7f4cf01d0de0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0000c10 tx=0x7f4ce00046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 msgr2=0x7f4cf01d0de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4ce5ffb640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4cf0136600 0x7f4cf01d0de0 secure :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0000c10 tx=0x7f4ce00046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 1000118, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:56.654 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.655 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.656 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:56.657 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.658 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:56.659 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:56.661 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.662 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:56.663 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.664 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.665 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:56.666 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.667 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:56.668 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.669 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.670 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.671 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.672 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.673 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.674 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:56.675 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 10 librados: watch_flush enter 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 10 librados: watch_flush exit 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 20 client.144152.objecter shutdown clearing up homeless session... 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 10 client.144152.objecter successfully canceled tick 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cd4038e10 msgr2=0x7f4cd403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4cd4038e10 0x7f4cd403b2d0 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f4cf00a55b0 tx=0x7f4cec008040 comp rx=0 tx=0).stop 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 10 monclient: shutdown 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0138a50 msgr2=0x7f4cf01360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 --2- 172.21.15.43:0/1792446573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4cf0138a50 0x7f4cf01360c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009e80 tx=0x7f4ce8019680 comp rx=0 tx=0).stop 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf5af6640 1 -- 172.21.15.43:0/1792446573 reap_dead start 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf5af6640 1 -- 172.21.15.43:0/1792446573 reap_dead start 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 shutdown_connections 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 >> 172.21.15.43:0/1792446573 conn(0x7f4cf00a7cb0 msgr2=0x7f4cf006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 shutdown_connections 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 -- 172.21.15.43:0/1792446573 wait complete. 2024-04-19T01:00:56.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.637+0000 7f4cf7559640 1 librados: shutdown 2024-04-19T01:00:56.678 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:56.679 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T01:00:56.679 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.797+0000 7f6a8b050640 10 monclient: get_monmap_and_config 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.797+0000 7f6a8b050640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.797+0000 7f6a8b050640 10 monclient: monmap: 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:56.802322+0000 2024-04-19T01:00:56.799 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:56.802322+0000 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:56.800 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:56.801 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 1 Processor -- start 2024-04-19T01:00:56.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 1 -- start start 2024-04-19T01:00:56.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient: init 2024-04-19T01:00:56.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient: _add_conns ranks=[0,1,9,5,4,3,2,7,8,6,10] 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6a84138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6a840a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6a841397c0 0x7f6a841419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient(hunting): picked mon.noname-a con 0x7f6a841397c0 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a840a3dc0 con 0x7f6a84138cd0 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a840a40a0 con 0x7f6a840a5b80 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f6a840a4410 con 0x7f6a841397c0 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a8b050640 10 monclient(hunting): authenticate will time out at 1926.929493s 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6a841397c0 0x7f6a841419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f6a841397c0 0x7f6a841419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55602/0 (socket says 172.21.15.43:55602) 2024-04-19T01:00:56.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 -- 172.21.15.43:0/2279698734 learned_addr learned my addr 172.21.15.43:0/2279698734 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): get_auth_request con 0x7f6a841397c0 auth_method 0 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient(hunting): handle_auth_done global_id 65376 payload 274 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 -- 172.21.15.43:0/2279698734 >> v1:172.21.15.161:6789/0 conn(0x7f6a840a5b80 legacy=0x7f6a8413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 -- 172.21.15.43:0/2279698734 >> v1:172.21.15.43:6789/0 conn(0x7f6a84138cd0 legacy=0x7f6a841390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 monclient: found mon.noname-a 2024-04-19T01:00:56.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:56.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.801+0000 7f6a895ed640 1 -- 172.21.15.43:0/2279698734 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a840a2ae0 con 0x7f6a841397c0 2024-04-19T01:00:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a895ed640 1 --2- 172.21.15.43:0/2279698734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a841397c0 0x7f6a841419a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6a840a4590 tx=0x7f6a7c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf35eebf5ec5d32e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 1 -- 172.21.15.43:0/2279698734 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a840a20a0 con 0x7f6a841397c0 2024-04-19T01:00:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:56.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: dump: 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 1 -- 172.21.15.43:0/2279698734 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7c0090d0 con 0x7f6a841397c0 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 1 -- 172.21.15.43:0/2279698734 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a7c0a5c20 con 0x7f6a841397c0 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 5 monclient: authenticate success, global_id 65376 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 4 set_mon_vals no callback set 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a837fe640 10 monclient: dump: 2024-04-19T01:00:56.814 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.815 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: shutdown 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- 172.21.15.43:0/2279698734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a841397c0 msgr2=0x7f6a841419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 --2- 172.21.15.43:0/2279698734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a841397c0 0x7f6a841419a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f6a840a4590 tx=0x7f6a7c01c8d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- 172.21.15.43:0/2279698734 shutdown_connections 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 --2- 172.21.15.43:0/2279698734 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f6a841397c0 0x7f6a841419a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- 172.21.15.43:0/2279698734 >> 172.21.15.43:0/2279698734 conn(0x7f6a840ab2f0 msgr2=0x7f6a840ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- 172.21.15.43:0/2279698734 shutdown_connections 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- 172.21.15.43:0/2279698734 wait complete. 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: monmap: 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:56.816 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 librados: starting msgr at 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 librados: starting objecter 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 Processor -- start 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- start start 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 librados: setting wanted keys 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 librados: calling monclient init 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: init 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient: _add_conns ranks=[3,5,8,2,4,0,7,1,6] 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a840a5b80 0x7f6a8412da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): picked mon.noname-g con 0x7f6a840a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a84138cd0 0x7f6a8412dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): picked mon.noname-h con 0x7f6a84138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6a841397c0 0x7f6a8409ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): picked mon.noname-e con 0x7f6a841397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6a84144a90 con 0x7f6a840a5b80 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6a84144910 con 0x7f6a84138cd0 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6a84144c10 con 0x7f6a841397c0 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.805+0000 7f6a8b050640 10 monclient(hunting): authenticate will time out at 1926.935987s 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6a841397c0 0x7f6a8409ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6a841397c0 0x7f6a8409ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39024/0 (socket says 172.21.15.43:39024) 2024-04-19T01:00:56.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 -- 172.21.15.43:0/3832742922 learned_addr learned my addr 172.21.15.43:0/3832742922 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): get_auth_request con 0x7f6a841397c0 auth_method 0 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a840a5b80 0x7f6a8412da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): get_auth_request con 0x7f6a840a5b80 auth_method 0 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a84138cd0 0x7f6a8412dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 10 monclient(hunting): get_auth_request con 0x7f6a84138cd0 auth_method 0 2024-04-19T01:00:56.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient(hunting): handle_auth_done global_id 144161 payload 824 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a84138cd0 msgr2=0x7f6a8412dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a84138cd0 0x7f6a8412dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a840a5b80 msgr2=0x7f6a8412da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6a840a5b80 0x7f6a8412da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 monclient: found mon.noname-e 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a840a3590 con 0x7f6a841397c0 2024-04-19T01:00:56.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a895ed640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6a841397c0 0x7f6a8409ba70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c01a500 tx=0x7f6a7c019670 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 client.?.objecter ms_handle_connect 0x7f6a841397c0 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a7c0a5e30 con 0x7f6a841397c0 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: dump: 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.822 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 5 monclient: authenticate success, global_id 144161 2024-04-19T01:00:56.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 monclient: _renew_subs 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7c0bc070 con 0x7f6a841397c0 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a8409f170 con 0x7f6a841397c0 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a7c0044b0 con 0x7f6a841397c0 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a827fc640 4 set_mon_vals no callback set 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 client.144161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: dump: 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.824 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 monclient: _renew_subs 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:00:56.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a840a3040 con 0x7f6a841397c0 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 1 librados: init done 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 0x7f6a8409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6a8406d140 con 0x7f6a8409bfb0 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6a7c004670 con 0x7f6a841397c0 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a70038e10 0x7f6a7003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(611..611 src has 501..611) ==== 6709+0+0 (secure 0 0 0) 0x7f6a7c0e4520 con 0x7f6a841397c0 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 client.144161.objecter ms_dispatch 0x7f6a84000ba0 osd_map(611..611 src has 501..611) 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 3 client.144161.objecter handle_osd_map got epochs [611,611] > 0 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 3 client.144161.objecter handle_osd_map decoding full epoch 611 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 0x7f6a8409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient: get_auth_request con 0x7f6a8409bfb0 auth_method 0 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 20 client.144161.objecter dump_active .. 0 homeless 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a70038e10 0x7f6a7003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 10 monclient: get_auth_request con 0x7f6a70038e10 auth_method 0 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 10 monclient(hunting): handle_auth_done global_id 74732 payload 824 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a840693b0 con 0x7f6a8409bfb0 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 client.144161.objecter ms_handle_connect 0x7f6a70038e10 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 client.144161.objecter ms_handle_connect 0x7f6a8409bfb0 2024-04-19T01:00:56.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 client.144161.objecter resend_mon_ops 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a83fff640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a70038e10 0x7f6a7003b2d0 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7f6a78004e30 tx=0x7f6a78009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a88dec640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 0x7f6a8409c390 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400e500 tx=0x7f6a7400e9c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a740052e0 con 0x7f6a8409bfb0 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.809+0000 7f6a817fa640 10 monclient: dump: 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.828 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6a74018070 con 0x7f6a8409bfb0 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 msgr2=0x7f6a8409c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 0x7f6a8409c390 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400e500 tx=0x7f6a7400e9c0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 msgr2=0x7f6a8409c390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.813+0000 7f6a817fa640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409bfb0 0x7f6a8409c390 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400e500 tx=0x7f6a7400e9c0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a8b050640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:56.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a8b050640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 0x7f6a841330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a8b050640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6a84068240 con 0x7f6a8409c8d0 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 0x7f6a841330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient: get_auth_request con 0x7f6a8409c8d0 auth_method 0 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 10 monclient(hunting): handle_auth_done global_id 74741 payload 824 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.817+0000 7f6a88dec640 1 -- 172.21.15.43:0/3832742922 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a8409d7d0 con 0x7f6a8409c8d0 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a88dec640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 0x7f6a841330e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400e500 tx=0x7f6a74000ee0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 client.144161.objecter ms_handle_connect 0x7f6a8409c8d0 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 client.144161.objecter resend_mon_ops 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6a740076d0 con 0x7f6a8409c8d0 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 monclient: dump: 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.833 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f6a74007260 con 0x7f6a8409c8d0 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 msgr2=0x7f6a841330e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 0x7f6a841330e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400e500 tx=0x7f6a74000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 msgr2=0x7f6a841330e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a817fa640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6a8409c8d0 0x7f6a841330e0 secure :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6a7400e500 tx=0x7f6a74000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 436436, 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:56.835 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:56.836 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:56.837 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.838 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:56.839 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:56.840 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.842 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.843 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.844 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.845 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.846 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:56.847 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.848 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.849 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.850 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:56.851 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.852 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.853 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:56.854 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:56.855 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.856 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:56.857 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:56.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 10 librados: watch_flush enter 2024-04-19T01:00:56.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 10 librados: watch_flush exit 2024-04-19T01:00:56.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 20 client.144161.objecter shutdown clearing up homeless session... 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 10 client.144161.objecter successfully canceled tick 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a70038e10 msgr2=0x7f6a7003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a70038e10 0x7f6a7003b2d0 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7f6a78004e30 tx=0x7f6a78009040 comp rx=0 tx=0).stop 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 10 monclient: shutdown 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6a841397c0 msgr2=0x7f6a8409ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 --2- 172.21.15.43:0/3832742922 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6a841397c0 0x7f6a8409ba70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c01a500 tx=0x7f6a7c019670 comp rx=0 tx=0).stop 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a895ed640 1 -- 172.21.15.43:0/3832742922 reap_dead start 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a895ed640 1 -- 172.21.15.43:0/3832742922 reap_dead start 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 shutdown_connections 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 >> 172.21.15.43:0/3832742922 conn(0x7f6a840ab2f0 msgr2=0x7f6a84130db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 shutdown_connections 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 -- 172.21.15.43:0/3832742922 wait complete. 2024-04-19T01:00:56.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.821+0000 7f6a8b050640 1 librados: shutdown 2024-04-19T01:00:56.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:56.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T01:00:56.861 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T01:00:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient: get_monmap_and_config 2024-04-19T01:00:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient: monmap: 2024-04-19T01:00:56.979 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:56.982657+0000 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:56.982657+0000 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.980 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.981 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:56.981 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:56.981 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 Processor -- start 2024-04-19T01:00:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 -- start start 2024-04-19T01:00:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient: init 2024-04-19T01:00:56.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient: _add_conns ranks=[3,4,5,2,7,6,1,10,8,0,9] 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22ec135d20 0x7f22ec136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): picked mon.noname-g con 0x7f22ec135d20 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 0x7f22ec0a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): picked mon.noname-c con 0x7f22ec135860 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec0a4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): picked mon.noname-h con 0x7f22ec0a3e80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f22ec0a1780 con 0x7f22ec135d20 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f22ec0a2400 con 0x7f22ec135860 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f22ec0a1130 con 0x7f22ec0a3e80 2024-04-19T01:00:56.984 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f2782640 10 monclient(hunting): authenticate will time out at 1927.109757s 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 0x7f22ec0a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 0x7f22ec0a3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52782/0 (socket says 172.21.15.43:52782) 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 -- 172.21.15.43:0/2961077159 learned_addr learned my addr 172.21.15.43:0/2961077159 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): get_auth_request con 0x7f22ec135860 auth_method 0 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22ebfff640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22ec135d20 0x7f22ec136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f0d1f640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec0a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f0d1f640 10 monclient(hunting): get_auth_request con 0x7f22ec0a3e80 auth_method 0 2024-04-19T01:00:56.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f0d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f0d1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22f0d1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22ebfff640 10 monclient(hunting): get_auth_request con 0x7f22ec135d20 auth_method 0 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_done global_id 134374 payload 274 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 -- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 msgr2=0x7f22ec0a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec0a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 -- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22ec135d20 msgr2=0x7f22ec136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22ec135d20 0x7f22ec136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 monclient: found mon.noname-c 2024-04-19T01:00:56.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:00:56.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.981+0000 7f22eb7fe640 1 -- 172.21.15.43:0/2961077159 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ec0a1410 con 0x7f22ec135860 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eb7fe640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 0x7f22ec0a3940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f22e00098a0 tx=0x7f22e001c4d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=3bc8c552850301f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 1 -- 172.21.15.43:0/2961077159 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22e00b0070 con 0x7f22ec135860 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: dump: 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.993 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: _finish_auth 0 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: _check_auth_tickets 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 5 monclient: authenticate success, global_id 134374 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 1 -- 172.21.15.43:0/2961077159 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e00aa3a0 con 0x7f22ec135860 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 1 -- 172.21.15.43:0/2961077159 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22e00aaa40 con 0x7f22ec135860 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 4 set_mon_vals no callback set 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eaffd640 10 monclient: dump: 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:56.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: shutdown 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- 172.21.15.43:0/2961077159 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 msgr2=0x7f22ec0a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 0x7f22ec0a3940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f22e00098a0 tx=0x7f22e001c4d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- 172.21.15.43:0/2961077159 shutdown_connections 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec0a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f22ec135860 0x7f22ec0a3940 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f22e00098a0 tx=0x7f22e001c4d0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- 172.21.15.43:0/2961077159 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f22ec135d20 0x7f22ec136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- 172.21.15.43:0/2961077159 >> 172.21.15.43:0/2961077159 conn(0x7f22ec0a85c0 msgr2=0x7f22ec0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- 172.21.15.43:0/2961077159 shutdown_connections 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- 172.21.15.43:0/2961077159 wait complete. 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: build_initial_monmap 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: monmap: 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:56.997 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:56.998 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 librados: starting msgr at 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 librados: starting objecter 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 Processor -- start 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- start start 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 librados: setting wanted keys 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 librados: calling monclient init 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: init 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient: _add_conns ranks=[5,2,0,4,7,3,8,6,1] 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec1d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): picked mon.noname-h con 0x7f22ec0a3e80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22ec135d20 0x7f22ec133830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): picked mon.noname-b con 0x7f22ec135d20 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:56.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22ec133d70 0x7f22ec11e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): picked mon.noname-a con 0x7f22ec133d70 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f22ec13f7c0 con 0x7f22ec133d70 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f22ec073460 con 0x7f22ec135d20 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f22ec13f940 con 0x7f22ec0a3e80 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f2782640 10 monclient(hunting): authenticate will time out at 1927.115804s 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22ec133d70 0x7f22ec11e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22ec133d70 0x7f22ec11e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55618/0 (socket says 172.21.15.43:55618) 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 1 -- 172.21.15.43:0/937309611 learned_addr learned my addr 172.21.15.43:0/937309611 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 10 monclient(hunting): get_auth_request con 0x7f22ec133d70 auth_method 0 2024-04-19T01:00:57.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22f0d1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.985+0000 7f22eb7fe640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22ec135d20 0x7f22ec133830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): get_auth_request con 0x7f22ec135d20 auth_method 0 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:57.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec1d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): get_auth_request con 0x7f22ec0a3e80 auth_method 0 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 10 monclient(hunting): handle_auth_done global_id 65385 payload 824 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 msgr2=0x7f22ec1d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f22ec0a3e80 0x7f22ec1d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22ec135d20 msgr2=0x7f22ec133830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22ec135d20 0x7f22ec133830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 10 monclient: _finish_hunting 0 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 monclient: found mon.noname-a 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:57.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22ec122220 con 0x7f22ec133d70 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f0d1f640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22ec133d70 0x7f22ec11e580 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f22e400d600 tx=0x7f22e400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 client.?.objecter ms_handle_connect 0x7f22ec133d70 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22e4012a60 con 0x7f22ec133d70 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: dump: 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.003 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: _finish_auth 0 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e4019070 con 0x7f22ec133d70 2024-04-19T01:00:57.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22e4012c60 con 0x7f22ec133d70 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: dump: 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.005 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 5 monclient: authenticate success, global_id 65385 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ea7fc640 4 set_mon_vals no callback set 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 monclient: _renew_subs 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22ec120a20 con 0x7f22ec133d70 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 client.65385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 monclient: _renew_subs 2024-04-19T01:00:57.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22ec0a0660 con 0x7f22ec133d70 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 1 librados: init done 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 0x7f22ec11eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f22ec06d140 con 0x7f22ec1341f0 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 0x7f22ec11eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient: get_auth_request con 0x7f22ec1341f0 auth_method 0 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:57.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f22e40125f0 con 0x7f22ec133d70 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f22d8038e10 0x7f22d803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(611..611 src has 501..611) ==== 6709+0+0 (secure 0 0 0) 0x7f22e400f040 con 0x7f22ec133d70 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 client.65385.objecter ms_dispatch 0x7f22ec000ba0 osd_map(611..611 src has 501..611) 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 3 client.65385.objecter handle_osd_map got epochs [611,611] > 0 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 3 client.65385.objecter handle_osd_map decoding full epoch 611 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 10 monclient(hunting): handle_auth_done global_id 74626 payload 824 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22ec120780 con 0x7f22ec1341f0 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 20 client.65385.objecter dump_active .. 0 homeless 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f22d8038e10 0x7f22d803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 10 monclient: get_auth_request con 0x7f22d8038e10 auth_method 0 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 client.65385.objecter ms_handle_connect 0x7f22ec1341f0 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 client.65385.objecter resend_mon_ops 2024-04-19T01:00:57.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22ebfff640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 0x7f22ec11eac0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004610 tx=0x7f22dc007700 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22dc014880 con 0x7f22ec1341f0 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 monclient: dump: 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.009 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22eb7fe640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f22d8038e10 0x7f22d803b2d0 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f22e00ad000 tx=0x7f22e00a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.989+0000 7f22e8ff9640 10 client.65385.objecter ms_handle_connect 0x7f22d8038e10 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22dc019070 con 0x7f22ec1341f0 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 msgr2=0x7f22ec11eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 0x7f22ec11eac0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004610 tx=0x7f22dc007700 comp rx=0 tx=0).stop 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 msgr2=0x7f22ec11eac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:57.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.993+0000 7f22e8ff9640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec1341f0 0x7f22ec11eac0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004610 tx=0x7f22dc007700 comp rx=0 tx=0).stop 2024-04-19T01:00:57.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22f2782640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:57.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22f2782640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 0x7f22ec0a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22f2782640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f22ec11f410 con 0x7f22ec11f000 2024-04-19T01:00:57.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 0x7f22ec0a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient: get_auth_request con 0x7f22ec11f000 auth_method 0 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 10 monclient(hunting): handle_auth_done global_id 74635 payload 824 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 1 -- 172.21.15.43:0/937309611 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22ec121a60 con 0x7f22ec11f000 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22ebfff640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 0x7f22ec0a5d30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004610 tx=0x7f22dc004a50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22e8ff9640 10 client.65385.objecter ms_handle_connect 0x7f22ec11f000 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:56.997+0000 7f22e8ff9640 10 client.65385.objecter resend_mon_ops 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f22dc014e20 con 0x7f22ec11f000 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 10 monclient: dump: 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.014 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7f22dc0149b0 con 0x7f22ec11f000 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 msgr2=0x7f22ec0a5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 0x7f22ec0a5d30 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004610 tx=0x7f22dc004a50 comp rx=0 tx=0).stop 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 msgr2=0x7f22ec0a5d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:57.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22e8ff9640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f22ec11f000 0x7f22ec0a5d30 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f22dc004610 tx=0x7f22dc004a50 comp rx=0 tx=0).stop 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 436616, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:57.016 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:57.017 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:57.018 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:57.019 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:57.020 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:57.021 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:57.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:57.026 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:57.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.031 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:57.037 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:00:57.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:57.039 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 10 librados: watch_flush enter 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 10 librados: watch_flush exit 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 20 client.65385.objecter shutdown clearing up homeless session... 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 10 client.65385.objecter successfully canceled tick 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f22d8038e10 msgr2=0x7f22d803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f22d8038e10 0x7f22d803b2d0 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f22e00ad000 tx=0x7f22e00a4f90 comp rx=0 tx=0).stop 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 10 monclient: shutdown 2024-04-19T01:00:57.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22ec133d70 msgr2=0x7f22ec11e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 --2- 172.21.15.43:0/937309611 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f22ec133d70 0x7f22ec11e580 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f22e400d600 tx=0x7f22e400dac0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f0d1f640 1 -- 172.21.15.43:0/937309611 reap_dead start 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f0d1f640 1 -- 172.21.15.43:0/937309611 reap_dead start 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 shutdown_connections 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 >> 172.21.15.43:0/937309611 conn(0x7f22ec0a85c0 msgr2=0x7f22ec134980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 shutdown_connections 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 -- 172.21.15.43:0/937309611 wait complete. 2024-04-19T01:00:57.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.001+0000 7f22f2782640 1 librados: shutdown 2024-04-19T01:00:57.042 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:57.043 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T01:00:57.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.161+0000 7f05bdebd640 10 monclient: get_monmap_and_config 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.161+0000 7f05bdebd640 10 monclient: build_initial_monmap 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.161+0000 7f05bdebd640 10 monclient: monmap: 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:57.165201+0000 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:57.165201+0000 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:57.162 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:57.163 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.161+0000 7f05bdebd640 1 Processor -- start 2024-04-19T01:00:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.161+0000 7f05bdebd640 1 -- start start 2024-04-19T01:00:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.161+0000 7f05bdebd640 10 monclient: init 2024-04-19T01:00:57.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient: _add_conns ranks=[7,6,3,4,10,0,1,9,5,8,2] 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f05b8138cd0 0x7f05b81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): picked mon.noname-i con 0x7f05b8138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 0x7f05b81395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): picked mon.noname-d con 0x7f05b80a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f05b81358b0 0x7f05b8135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): picked mon.noname-g con 0x7f05b81358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f05b80a4410 con 0x7f05b81358b0 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f05b8139b30 con 0x7f05b80a5b80 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f05b80a3dc0 con 0x7f05b8138cd0 2024-04-19T01:00:57.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient(hunting): authenticate will time out at 1927.292331s 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 0x7f05b81395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 0x7f05b81395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53852/0 (socket says 172.21.15.43:53852) 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 -- 172.21.15.43:0/3092799124 learned_addr learned my addr 172.21.15.43:0/3092799124 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): get_auth_request con 0x7f05b80a5b80 auth_method 0 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient(hunting): handle_auth_done global_id 144168 payload 274 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b7fff640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f05b81358b0 0x7f05b8135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 -- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f05b8138cd0 msgr2=0x7f05b81390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b77fe640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f05b8138cd0 0x7f05b81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f05b8138cd0 0x7f05b81390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 -- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f05b81358b0 msgr2=0x7f05b8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f05b81358b0 0x7f05b8135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 monclient: found mon.noname-d 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 -- 172.21.15.43:0/3092799124 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b80a40a0 con 0x7f05b80a5b80 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b7fff640 10 monclient: get_auth_request con 0x7f05b81358b0 auth_method 0 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b7fff640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f05b81358b0 0x7f05b8135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b77fe640 10 monclient: get_auth_request con 0x7f05b8138cd0 auth_method 0 2024-04-19T01:00:57.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b77fe640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f05b8138cd0 0x7f05b81390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b6ffd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 0x7f05b81395f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05a00098a0 tx=0x7f05a001c3e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=67438dd46b1c6f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 1 -- 172.21.15.43:0/3092799124 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05a00b0070 con 0x7f05b80a5b80 2024-04-19T01:00:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: dump: 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.175 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 1 -- 172.21.15.43:0/3092799124 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a00aa810 con 0x7f05b80a5b80 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 5 monclient: authenticate success, global_id 144168 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 4 set_mon_vals no callback set 2024-04-19T01:00:57.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 1 -- 172.21.15.43:0/3092799124 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05a00a9300 con 0x7f05b80a5b80 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05b67fc640 10 monclient: dump: 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.177 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 10 monclient: shutdown 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 -- 172.21.15.43:0/3092799124 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 msgr2=0x7f05b81395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 0x7f05b81395f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05a00098a0 tx=0x7f05a001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 -- 172.21.15.43:0/3092799124 shutdown_connections 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f05b81358b0 0x7f05b8135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f05b80a5b80 0x7f05b81395f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 --2- 172.21.15.43:0/3092799124 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f05b8138cd0 0x7f05b81390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.165+0000 7f05bdebd640 1 -- 172.21.15.43:0/3092799124 >> 172.21.15.43:0/3092799124 conn(0x7f05b80ab2f0 msgr2=0x7f05b80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- 172.21.15.43:0/3092799124 shutdown_connections 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- 172.21.15.43:0/3092799124 wait complete. 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: build_initial_monmap 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: monmap: 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:57.179 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 librados: starting msgr at 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 librados: starting objecter 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 Processor -- start 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- start start 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 librados: setting wanted keys 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 librados: calling monclient init 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: init 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:57.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _add_conns ranks=[0,1,5,2,7,8,4,6,3] 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 0x7f05b81d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): picked mon.noname-a con 0x7f05b80a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b81358b0 0x7f05b81d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): picked mon.noname-f con 0x7f05b81358b0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f05b8138cd0 0x7f05b81d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): picked mon.noname-h con 0x7f05b8138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f05b80a2010 con 0x7f05b80a5b80 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f05b80a1e90 con 0x7f05b81358b0 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f05b8142aa0 con 0x7f05b8138cd0 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient(hunting): authenticate will time out at 1927.297644s 2024-04-19T01:00:57.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 0x7f05b81d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 0x7f05b81d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55622/0 (socket says 172.21.15.43:55622) 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 -- 172.21.15.43:0/516376809 learned_addr learned my addr 172.21.15.43:0/516376809 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): get_auth_request con 0x7f05b80a5b80 auth_method 0 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b7fff640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f05b8138cd0 0x7f05b81d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b6ffd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b81358b0 0x7f05b81d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b7fff640 10 monclient(hunting): get_auth_request con 0x7f05b8138cd0 auth_method 0 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b6ffd640 10 monclient(hunting): get_auth_request con 0x7f05b81358b0 auth_method 0 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient(hunting): handle_auth_done global_id 65394 payload 824 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 -- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f05b8138cd0 msgr2=0x7f05b81d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f05b8138cd0 0x7f05b81d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 -- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b81358b0 msgr2=0x7f05b81d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b81358b0 0x7f05b81d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 monclient: found mon.noname-a 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 -- 172.21.15.43:0/516376809 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b80a05c0 con 0x7f05b80a5b80 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b77fe640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 0x7f05b81d0a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f05b000ce00 tx=0x7f05b000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 client.?.objecter ms_handle_connect 0x7f05b80a5b80 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05b000ae20 con 0x7f05b80a5b80 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: dump: 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.184 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: _finish_auth 0 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: _check_auth_tickets 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b000fb00 con 0x7f05b80a5b80 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 5 monclient: authenticate success, global_id 65394 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _renew_subs 2024-04-19T01:00:57.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b80a3590 con 0x7f05b80a5b80 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f05b000fcb0 con 0x7f05b80a5b80 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05b5ffb640 4 set_mon_vals no callback set 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 client.65394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 monclient: dump: 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:57.186 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _renew_subs 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b80a02e0 con 0x7f05b80a5b80 2024-04-19T01:00:57.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 librados: init done 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f05b000f690 con 0x7f05b80a5b80 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0598038e10 0x7f059803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b80693b0 con 0x7f05b80a5b80 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(611..611 src has 501..611) ==== 6709+0+0 (secure 0 0 0) 0x7f05b004a390 con 0x7f05b80a5b80 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 10 client.65394.objecter ms_dispatch 0x7f05b8000ba0 osd_map(611..611 src has 501..611) 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 3 client.65394.objecter handle_osd_map got epochs [611,611] > 0 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.169+0000 7f05affff640 3 client.65394.objecter handle_osd_map decoding full epoch 611 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.173+0000 7f05affff640 20 client.65394.objecter dump_active .. 0 homeless 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.173+0000 7f05b6ffd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0598038e10 0x7f059803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.173+0000 7f05b6ffd640 10 monclient: get_auth_request con 0x7f0598038e10 auth_method 0 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.173+0000 7f05affff640 10 client.65394.objecter ms_handle_connect 0x7f0598038e10 2024-04-19T01:00:57.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.173+0000 7f05b6ffd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0598038e10 0x7f059803b2d0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f05a00ad000 tx=0x7f05a00a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:57.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.177+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f05b0014030 con 0x7f05b80a5b80 2024-04-19T01:00:57.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.177+0000 7f05affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:57.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.177+0000 7f05affff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:57.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.409+0000 7f05bdebd640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T01:00:57.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.409+0000 7f05bdebd640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T01:00:57.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.409+0000 7f05bdebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:57.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.409+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f05b80008e0 con 0x7f05b80a5b80 2024-04-19T01:00:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.413+0000 7f05affff640 1 -- 172.21.15.43:0/516376809 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f05b000d530 con 0x7f05b80a5b80 2024-04-19T01:00:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.413+0000 7f05affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T01:00:57.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.413+0000 7f05affff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:57.417 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.417 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.417 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=10,kv=12,logm=51} crc {auth=192215208,config=2586702516,health=2914215846,kv=4196696685,logm=2781547565}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=1409537675,config=2586702516,health=1482733132,kv=4196696685,logm=2933720390}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=11,kv=11,logm=52} crc {auth=2180000861,config=2586702516,health=2334723672,kv=4171600807,logm=3608180357}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=192215208,config=2586702516,health=1482733132,kv=4196696685,logm=2601396993}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=192215208,config=2586702516,health=3102051209,kv=2243130063,logm=297904459}) 2024-04-19T01:00:57.418 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=25,config=2,health=11,kv=11,logm=51} crc {auth=23994073,config=2586702516,health=3562341845,kv=139348691,logm=2373241471}) 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=1700341606,config=2586702516,health=1482733132,kv=4171600807,logm=1373360337}) 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=244772752,config=2586702516,health=3562341845,kv=4196696685,logm=969258158}) 2024-04-19T01:00:57.419 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.416+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=2624554927,config=2586702516,health=1482733132,kv=414189483,logm=1577822047}) 2024-04-19T01:00:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 10 librados: watch_flush enter 2024-04-19T01:00:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 10 librados: watch_flush exit 2024-04-19T01:00:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 20 client.65394.objecter shutdown clearing up homeless session... 2024-04-19T01:00:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 10 client.65394.objecter successfully canceled tick 2024-04-19T01:00:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0598038e10 msgr2=0x7f059803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0598038e10 0x7f059803b2d0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f05a00ad000 tx=0x7f05a00a4f90 comp rx=0 tx=0).stop 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 10 monclient: shutdown 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 msgr2=0x7f05b81d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 0x7f05b81d0a30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f05b000ce00 tx=0x7f05b000a650 comp rx=0 tx=0).stop 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 shutdown_connections 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0598038e10 0x7f059803b2d0 unknown :-1 s=CLOSED pgs=3601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f05b8138cd0 0x7f05b81d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b81358b0 0x7f05b81d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 --2- 172.21.15.43:0/516376809 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f05b80a5b80 0x7f05b81d0a30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 >> 172.21.15.43:0/516376809 conn(0x7f05b80ab2f0 msgr2=0x7f05b8137460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 shutdown_connections 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 -- 172.21.15.43:0/516376809 wait complete. 2024-04-19T01:00:57.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:57.417+0000 7f05bdebd640 1 librados: shutdown 2024-04-19T01:00:57.423 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189915079}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009840221}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757318776}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2679082768}) 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.424 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4169906450}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=542122726}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=274451269}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523099811}) 2024-04-19T01:00:57.425 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.424+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3742347772}) 2024-04-19T01:00:57.431 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3284107245}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010346972}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3142061396}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2459269797}) 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.432 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2593802244}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2889307460}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1409682652}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=183242490}) 2024-04-19T01:00:57.433 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.432+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2333951044}) 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606216197}) 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942654394}) 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.439 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1973344331}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2202643014}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2398487292}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2687957865}) 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.440 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.441 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2458884446}) 2024-04-19T01:00:57.441 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.441 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1143634778}) 2024-04-19T01:00:57.441 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.436+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3505961887}) 2024-04-19T01:00:57.442 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T01:00:57.446 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.446 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.446 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976461414}) 2024-04-19T01:00:57.446 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2179318902}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780397780}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2171869713}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2628899218}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3545147683}) 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.447 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.448 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1048495742}) 2024-04-19T01:00:57.448 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.448 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4274477345}) 2024-04-19T01:00:57.448 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.444+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=217689894}) 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2155796620}) 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692640198}) 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.451 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641137057}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1425341734}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3550936661}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=974182275}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=308848380}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3993529700}) 2024-04-19T01:00:57.452 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.448+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1007194789}) 2024-04-19T01:00:57.454 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.454 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.454 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3304924107}) 2024-04-19T01:00:57.454 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.454 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1612359395}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3696843727}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=571876548}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1749524999}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2836971806}) 2024-04-19T01:00:57.455 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.456 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.456 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4263458271}) 2024-04-19T01:00:57.456 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.456 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4104810648}) 2024-04-19T01:00:57.456 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.452+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=503655286}) 2024-04-19T01:00:57.458 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.458 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.458 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692028493}) 2024-04-19T01:00:57.458 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.458 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.458 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=999301016}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2296808976}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=671760572}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1947962524}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3573997668}) 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.459 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.460 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=330005889}) 2024-04-19T01:00:57.460 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.460 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=898266882}) 2024-04-19T01:00:57.460 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.456+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2972010749}) 2024-04-19T01:00:57.462 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.462 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.462 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=162540099}) 2024-04-19T01:00:57.462 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=621299108}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466470131}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4144409878}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1361145925}) 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.463 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=918111741}) 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=253781437}) 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2434274173}) 2024-04-19T01:00:57.464 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.460+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=828149689}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2996472041}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=209284355}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969731241}) 2024-04-19T01:00:57.466 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3201708875}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3765988451}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=916192310}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1429471792}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442329364}) 2024-04-19T01:00:57.467 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.464+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=225853252}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1319592215}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626125532}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2758432317}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.470 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4156799856}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1324664752}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2301574445}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1165869112}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4024334194}) 2024-04-19T01:00:57.471 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.468+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=84788436}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250804555}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4268183606}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458808707}) 2024-04-19T01:00:57.474 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3871875076}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4142842387}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1436356413}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372658107}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3317419546}) 2024-04-19T01:00:57.475 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.472+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3866471369}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224390900}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492991507}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=259412084}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=588136232}) 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.477 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3494613814}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2684495365}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3696727674}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2941104728}) 2024-04-19T01:00:57.478 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.476+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=28983580}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665283790}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1464449643}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=199827384}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1617260757}) 2024-04-19T01:00:57.481 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2507842242}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=113321414}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2877574789}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3198061898}) 2024-04-19T01:00:57.482 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.480+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1801099859}) 2024-04-19T01:00:57.484 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.484 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.484 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551002847}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018151390}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523849893}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3780395925}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2721125334}) 2024-04-19T01:00:57.485 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=721641623}) 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3185286750}) 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1370574787}) 2024-04-19T01:00:57.486 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.484+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4081238088}) 2024-04-19T01:00:57.488 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.488 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.488 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520719166}) 2024-04-19T01:00:57.488 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.488 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.488 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4293722728}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=291746846}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3844596173}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=395376359}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1042779505}) 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.489 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.490 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=427737248}) 2024-04-19T01:00:57.490 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.490 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=355295903}) 2024-04-19T01:00:57.490 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.488+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1809120968}) 2024-04-19T01:00:57.493 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.493 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.493 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968963999}) 2024-04-19T01:00:57.493 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.493 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.493 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663501869}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1356893477}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=226862059}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1882136261}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3971425185}) 2024-04-19T01:00:57.494 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.495 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.495 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639905262}) 2024-04-19T01:00:57.495 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.495 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=320662769}) 2024-04-19T01:00:57.495 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.492+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2891706167}) 2024-04-19T01:00:57.496 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611300903}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3166672002}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=276758890}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2595075577}) 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.497 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4230852575}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3333057434}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1008979782}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=261903227}) 2024-04-19T01:00:57.498 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.496+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3208757819}) 2024-04-19T01:00:57.500 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2247019477}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717978215}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2164576065}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1414823960}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.501 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2865044730}) 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3651585259}) 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=205312598}) 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=514305216}) 2024-04-19T01:00:57.502 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.500+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=481074809}) 2024-04-19T01:00:57.504 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.504 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1915379253}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270908043}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1349160874}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4211609100}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4220119878}) 2024-04-19T01:00:57.505 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=831574737}) 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2172750608}) 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1821970676}) 2024-04-19T01:00:57.506 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.504+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2680298099}) 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2910162552}) 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4020923012}) 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.508 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43523835}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=695856354}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1875081742}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2949400476}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1618680076}) 2024-04-19T01:00:57.509 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.510 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2060056710}) 2024-04-19T01:00:57.510 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2527870361}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3480690672}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.508+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461761609}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281973466}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2635365223}) 2024-04-19T01:00:57.512 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=656665648}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1165501944}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4187544768}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3704156579}) 2024-04-19T01:00:57.513 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2953403579}) 2024-04-19T01:00:57.515 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.515 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.515 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2824830899}) 2024-04-19T01:00:57.515 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.515 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.515 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300117761}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1501619247}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4161641374}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3926584518}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=768988671}) 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.516 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.517 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=145406714}) 2024-04-19T01:00:57.517 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.517 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2589298124}) 2024-04-19T01:00:57.517 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.512+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2866348848}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=428305744}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094919837}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439273301}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.519 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2640248974}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4222922368}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3619490539}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2913575760}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1999018769}) 2024-04-19T01:00:57.520 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.516+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=876377374}) 2024-04-19T01:00:57.522 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.522 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=436354738}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=907502661}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123893347}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2761752474}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2845266450}) 2024-04-19T01:00:57.523 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3410897309}) 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1635869877}) 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2351051820}) 2024-04-19T01:00:57.524 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.520+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=214194177}) 2024-04-19T01:00:57.526 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.526 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1099831059}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454684979}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2218804946}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2955999094}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4128606155}) 2024-04-19T01:00:57.527 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3991068569}) 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=238154105}) 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=35806474}) 2024-04-19T01:00:57.528 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.524+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=942230190}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845420717}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2379956144}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3191880316}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=722289654}) 2024-04-19T01:00:57.533 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3618459925}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=767954159}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3350814875}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1902959499}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.528+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2902047566}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.534 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1435718163}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1195250578}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2115691136}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2687445020}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=577656659}) 2024-04-19T01:00:57.535 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1754200272}) 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2109581357}) 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3752459526}) 2024-04-19T01:00:57.536 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3677111702}) 2024-04-19T01:00:57.538 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.538 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.538 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445915488}) 2024-04-19T01:00:57.538 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.538 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.538 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=926184840}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536920492}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3937336270}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=361845629}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1648207434}) 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.539 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3256339919}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=173526079}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.532+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4130015024}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654376266}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2721407972}) 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.540 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873384145}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2391738794}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=160653641}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=734557726}) 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.541 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.542 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2628538969}) 2024-04-19T01:00:57.542 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.542 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033911665}) 2024-04-19T01:00:57.542 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.536+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=256139448}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=861952421}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738090173}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3999369896}) 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.544 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1460440585}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3341952132}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3865195373}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2324430647}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859894997}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.540+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1101793948}) 2024-04-19T01:00:57.545 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552663114}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2426373517}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=796408423}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3689778849}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.546 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=973531170}) 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1877067122}) 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3552435704}) 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3245021752}) 2024-04-19T01:00:57.547 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.544+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=777575945}) 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415263263}) 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945086327}) 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.548 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3879969027}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3977361590}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4078121521}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1417232091}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1517360826}) 2024-04-19T01:00:57.549 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.550 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=812993863}) 2024-04-19T01:00:57.550 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1187875447}) 2024-04-19T01:00:57.551 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.551 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.551 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3909895372}) 2024-04-19T01:00:57.551 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.551 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=879818873}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1770126336}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3704890421}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1961802960}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.552 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2303260844}) 2024-04-19T01:00:57.553 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.553 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.553 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=844300914}) 2024-04-19T01:00:57.553 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.553 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2851261853}) 2024-04-19T01:00:57.553 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.548+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1821963047}) 2024-04-19T01:00:57.554 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.554 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.554 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265687713}) 2024-04-19T01:00:57.554 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.554 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821157759}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3325049779}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2864796651}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=935224967}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.555 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1335197502}) 2024-04-19T01:00:57.556 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.556 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.556 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1647540951}) 2024-04-19T01:00:57.556 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.556 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1859022498}) 2024-04-19T01:00:57.556 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.552+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2046841596}) 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3357443968}) 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2181263361}) 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.559 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790205776}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1720700712}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1524406935}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3503931511}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.560 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1450589303}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3313710857}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2272817165}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018984703}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3987842363}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3620317405}) 2024-04-19T01:00:57.561 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=639736816}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3593216842}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1342333353}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2918685868}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3292716800}) 2024-04-19T01:00:57.562 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.556+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3654119104}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203752081}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738223575}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714771327}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.565 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1326902758}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=451000114}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2576284001}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2453038822}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1066036270}) 2024-04-19T01:00:57.566 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.560+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3278112984}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273724561}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=200660841}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699590303}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2037419581}) 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.567 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3169147249}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2740806938}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1571954496}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3035669385}) 2024-04-19T01:00:57.568 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.564+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1128103035}) 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=987083104}) 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607107689}) 2024-04-19T01:00:57.571 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3705264570}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3332845013}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1172620466}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1667474260}) 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.572 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1307927078}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=286691404}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3176782341}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613960731}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=85088484}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950108621}) 2024-04-19T01:00:57.573 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3242763156}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3285064496}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1466149050}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1513411206}) 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.574 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2165010382}) 2024-04-19T01:00:57.575 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.568+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=758997120}) 2024-04-19T01:00:57.583 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.583 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.583 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275845144}) 2024-04-19T01:00:57.583 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.583 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.583 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3151745366}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493058377}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2238085024}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3848596584}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2816200420}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.584 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1537950467}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1888898133}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.572+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3288519229}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006833311}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591107704}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334479255}) 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.585 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3299827753}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=290052743}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1676216965}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2509690820}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3410088114}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.576+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1938328033}) 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.586 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1088833112}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627559481}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2824941616}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3019331834}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3489484296}) 2024-04-19T01:00:57.587 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1569431361}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=794518796}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3373742876}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4165131056}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338101031}) 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.588 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4023120046}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210894228}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1533392375}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2097829047}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4231446335}) 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.589 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.590 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=759133105}) 2024-04-19T01:00:57.590 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.590 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3199047199}) 2024-04-19T01:00:57.590 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.580+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4123166501}) 2024-04-19T01:00:57.597 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.597 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.597 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261631899}) 2024-04-19T01:00:57.597 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2631336281}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889194647}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=690606543}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3111309768}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.598 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1557150095}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=869499776}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093114847}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.584+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3998838273}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2971697421}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430828416}) 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.599 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864414444}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3972475235}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4220790939}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3407504042}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1022108928}) 2024-04-19T01:00:57.600 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2745357798}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1565411297}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134818903}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3129373634}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508218878}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1528851619}) 2024-04-19T01:00:57.601 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1657489707}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3783412360}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2258817747}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2278711659}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.588+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2495307263}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.602 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210705417}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451834203}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403266637}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=18971288}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2246463603}) 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.603 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1732464182}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1863744806}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2401765327}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1790091114}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145783200}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155565293}) 2024-04-19T01:00:57.604 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347500083}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2059603465}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3183637174}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3608398162}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.592+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3658971200}) 2024-04-19T01:00:57.605 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=641853647}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=114362948}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2857393310}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854449907}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=510638567}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3123159927}) 2024-04-19T01:00:57.606 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3997975017}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=248216709}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4109229873}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2342640205}) 2024-04-19T01:00:57.607 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.596+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2017515711}) 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4084763649}) 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302633726}) 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.639 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1885186241}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4023350526}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=515464162}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=744126740}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2408464059}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137208035}) 2024-04-19T01:00:57.640 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1400199719}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948693925}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839863178}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1222277806}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2202848713}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.641 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1248622580}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4023663022}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1805866159}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=711371117}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.600+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2729511201}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1478236479}) 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.642 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216885652}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702764137}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=290770002}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=556047274}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1618521571}) 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.643 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=848569558}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1224876546}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2462431521}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4067342088}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899486385}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4009939216}) 2024-04-19T01:00:57.644 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1457220171}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1330912522}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3843633725}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=606207193}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=549478197}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.604+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3070664230}) 2024-04-19T01:00:57.645 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1454586973}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639885928}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2052936194}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2639796982}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2470434351}) 2024-04-19T01:00:57.646 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2172375821}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3417070966}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776455025}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.608+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1159822341}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352396344}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.647 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639962123}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3702023096}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3572399855}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2056668164}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=502491457}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.648 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3112744925}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=319462731}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1698385865}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2869974744}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294735186}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611830808}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.649 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1395183894}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1098036547}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1903126167}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2462897851}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2041541274}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.612+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2363446540}) 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.650 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133818821}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330403108}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=387038958}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1848391347}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1019139833}) 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.651 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3699809683}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1769944065}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1484186325}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1327687215}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477942269}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791821647}) 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.652 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=648666988}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3704782433}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=257535193}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4258698396}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2605227506}) 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.653 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3246819754}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.616+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1447541403}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=76594655}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223703245}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527485403}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=74764634}) 2024-04-19T01:00:57.654 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3617045388}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1485489015}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=255869280}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2987213125}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1765494087}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3067216155}) 2024-04-19T01:00:57.655 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939683527}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=158090876}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=941503649}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4119420539}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=364515226}) 2024-04-19T01:00:57.656 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=506837602}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=515240152}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.620+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2478856066}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1222417751}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4058854573}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641961346}) 2024-04-19T01:00:57.657 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3915230530}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=264912813}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2327277481}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1356706377}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3075151930}) 2024-04-19T01:00:57.658 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4133257848}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149860059}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1294523710}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63263153}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2340914295}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.659 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4077306343}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.624+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=43903334}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1743080658}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1952763840}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1935509985}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388377818}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.660 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064306339}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949440278}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209351645}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2983861336}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3329709829}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.661 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2007004028}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=647344473}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.628+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=218100803}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599674879}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2761694941}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735704698}) 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.662 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1286028968}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=669083805}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1617171088}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2733781326}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3065529774}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2649575221}) 2024-04-19T01:00:57.663 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137890655}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=40122970}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177152633}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3458850592}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.664 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=612864613}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1289164520}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=569074301}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3717979195}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.632+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4191577891}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74150302}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.665 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2381710624}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617037154}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2309645074}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1608406716}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1965634301}) 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.666 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4259350779}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248151144}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3601199058}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078539797}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=464052375}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3111405420}) 2024-04-19T01:00:57.667 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2099853290}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1567672101}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2137580194}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=670489547}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3988108605}) 2024-04-19T01:00:57.668 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.636+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=831845958}) 2024-04-19T01:00:57.801 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.801 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087404027}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567558397}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278297768}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3487606224}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.802 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=661504946}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1926374242}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2133293972}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=437887270}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3339761538}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546060714}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.803 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=758562686}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164085956}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2457455044}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=585806723}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1327434094}) 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.804 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=853748921}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=357308432}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.640+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2022713738}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239160898}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103550888}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3977097943}) 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.805 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3612524235}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1662393630}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1571414995}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2844251934}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=564241186}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1542123889}) 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.806 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4077257555}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197993046}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593810181}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2262070886}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1508323218}) 2024-04-19T01:00:57.807 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2475603307}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3842052303}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=859675520}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.644+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3539871972}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144751344}) 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.808 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028067213}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=807332506}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=790852258}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=587891376}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=858582797}) 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.809 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3148427251}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3153032303}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1455900923}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405929206}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1630541432}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.810 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011749327}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=124791103}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1116449065}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=893284126}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2385862654}) 2024-04-19T01:00:57.811 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1868948149}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.648+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=605534692}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637771827}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666589656}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776408376}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.812 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3971923522}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=444684468}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1508246473}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=746627979}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3821362633}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3189128181}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2660108746}) 2024-04-19T01:00:57.813 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2194716969}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2486049116}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2611846107}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1962590223}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.814 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1665613371}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=11284200}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2553371937}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.652+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4276454034}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189998245}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4214155646}) 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.815 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568051148}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=590149388}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3477288537}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3593302440}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.816 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4143969824}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533042291}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1478949107}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169393056}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556786764}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467330903}) 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.817 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4196581611}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=356600625}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2581731174}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2894003289}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3886196840}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.656+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1017723892}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.818 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071703782}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1300694801}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934195039}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1141386675}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=344241477}) 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.819 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1520738287}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=677221546}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=503917076}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=81108690}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197818089}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.820 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986462767}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007518742}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4199647223}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3981210007}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2178032082}) 2024-04-19T01:00:57.821 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1032715278}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=334371290}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.660+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1086074460}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2271753433}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160416963}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310757555}) 2024-04-19T01:00:57.822 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2737648739}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1594465591}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3575511718}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1289225946}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=278226865}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2247553281}) 2024-04-19T01:00:57.823 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3839737985}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2913607242}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683809397}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2392254769}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3999088517}) 2024-04-19T01:00:57.824 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1774801883}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4218277546}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1267299143}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.664+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=728458813}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231793957}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.825 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429054098}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704081360}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3076476684}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=901962658}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3270189229}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.826 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2694619755}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2452657935}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2741414304}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012647395}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545169688}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680343443}) 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.827 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=844198215}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3029378673}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4170008115}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2054313759}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1707821773}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.668+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3868421185}) 2024-04-19T01:00:57.828 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921654107}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3121270855}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3238147989}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=418153653}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4183952400}) 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2901310165}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1065248075}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3758731356}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3393197798}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=873531544}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963499736}) 2024-04-19T01:00:57.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2718964902}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1727181297}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2011254367}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2130162611}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=873721717}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2371691702}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.672+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3524156344}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66822166}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3896098795}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264788810}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.832 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2554358211}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=348723029}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=419396435}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1140828232}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=751779869}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2244722894}) 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.833 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1627674544}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527229607}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=66485115}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3774711981}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3813286741}) 2024-04-19T01:00:57.834 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2398379325}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3683845320}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063329313}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.676+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2513797699}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133582496}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692229730}) 2024-04-19T01:00:57.835 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1242620409}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=740901232}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2201855866}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3616930142}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3575440444}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1726728637}) 2024-04-19T01:00:57.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1274664690}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2714168211}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046439051}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830012016}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=371822764}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4135767685}) 2024-04-19T01:00:57.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3140878377}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=418505201}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.680+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=739242947}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1116915562}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526972760}) 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917697160}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568722824}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3747817426}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2155866933}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=643852617}) 2024-04-19T01:00:57.839 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3543432877}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2101589405}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3736262339}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4033247381}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172918052}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.840 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3989342356}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3720674998}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=221043526}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=717789077}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2419462430}) 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.841 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=429015241}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.684+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=744148934}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541356652}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695511256}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172991439}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=934742034}) 2024-04-19T01:00:57.842 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=342497828}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1747587906}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2079220447}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3594105218}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=166118958}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402742456}) 2024-04-19T01:00:57.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3004350626}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248591026}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1016359128}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1616162868}) 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1897846847}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3736472046}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2106825600}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.688+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3478141083}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2975554395}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262462545}) 2024-04-19T01:00:57.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564164030}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=457388282}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4290953492}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=39954519}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.846 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1495562159}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=821223071}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2391409188}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537674000}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005670656}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=156775225}) 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.847 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3997825492}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=243857797}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1618636098}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4027354927}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1805619684}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.692+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2186940200}) 2024-04-19T01:00:57.848 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3532111820}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2994720981}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1764880037}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=355246880}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3882626122}) 2024-04-19T01:00:57.849 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=513982630}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2024389986}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506003548}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2013880093}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542402224}) 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.850 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696675492}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1070318638}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3212822504}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1123322577}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3422263391}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.851 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=538129377}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=756380580}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.696+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4187411625}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707430372}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2695047816}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3692866004}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1456404584}) 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.852 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=477943098}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=736208275}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3584208555}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3903761815}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3228062569}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725477460}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.853 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1392342230}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=225903619}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2819257028}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.700+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3690785888}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3469329579}) 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.854 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=277891924}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3371589798}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=230129470}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456441092}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1668344713}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174499593}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.855 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=883604518}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1369757386}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=149710639}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1435314189}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2089565957}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=71486291}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.856 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3020833316}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3099958445}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877818834}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2924882853}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2343238299}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.704+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.857 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2660107162}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4190651435}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=763899212}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1111360492}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050346079}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=907759666}) 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.858 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3695917489}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1988258809}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1140675439}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3774743482}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=922476165}) 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.859 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890326140}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2823123856}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642941996}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2651729324}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298112391}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1465866178}) 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.860 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3770500606}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=16688208}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2000355971}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4200600917}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.708+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1996844422}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965504708}) 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.861 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171365919}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878911008}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1056083719}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1159529193}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.862 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3690434083}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2802204660}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4143643784}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2444835578}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343073733}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1417254853}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4010665735}) 2024-04-19T01:00:57.863 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2957995510}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3702768542}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3931395055}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3934960120}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3127626306}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.712+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1999516567}) 2024-04-19T01:00:57.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52111773}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286796478}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=779561733}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4193781799}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2613635548}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4164671470}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3430132689}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=701993861}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3820835816}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814337266}) 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3471129502}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808986420}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=976287249}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=681913013}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.867 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1355396458}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=447092316}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2473116030}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.716+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3556484961}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702567588}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4106396305}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.868 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2493816129}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3259662834}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=97999646}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3278234546}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=693509638}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1752644951}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1191435987}) 2024-04-19T01:00:57.869 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217641165}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3187015173}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429886672}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=532338881}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.870 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2270821963}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3040314613}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4108683519}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4273993603}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.720+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1939564995}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820905953}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720876532}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097045241}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1333741090}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2350542065}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=379648453}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1217770586}) 2024-04-19T01:00:57.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3013378348}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1935922065}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1984227235}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115930647}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092474173}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2690996000}) 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2782573995}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2994096846}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2330300737}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=6974894}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.724+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1040362544}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571630662}) 2024-04-19T01:00:57.874 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667995738}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=298300992}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1375785870}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3889684833}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2691255230}) 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1619926799}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=196551070}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3200853018}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931411641}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3819226238}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051456382}) 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1375630175}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=588409751}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1696254775}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1960233690}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3897827003}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.728+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2024155276}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1455941040}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252476796}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2998954694}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2558659459}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2422093080}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=261593787}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3468578250}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1782754709}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=404168785}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258719827}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4144180421}) 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.879 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107708009}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2190541020}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3805270374}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1142671064}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=314939230}) 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.880 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2531913440}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.732+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2142821609}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3449976768}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3838997366}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192107723}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1621508996}) 2024-04-19T01:00:57.881 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2976001885}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1429040136}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2850127819}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1178670134}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3048188145}) 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.882 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401831869}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1859769779}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2958999382}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2540601137}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2377442473}) 2024-04-19T01:00:57.883 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2992376347}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=267008313}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3374214530}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.736+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2481270561}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603675300}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.884 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935423888}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291216837}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4056064749}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1379270132}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=824144370}) 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3495331570}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2588908488}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2492444133}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559607883}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989982673}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662573628}) 2024-04-19T01:00:57.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=436006619}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1914297070}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3064206730}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1272936257}) 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=258488508}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.740+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2896835352}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182834544}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=954314407}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777455814}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=72217385}) 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1651872408}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=41964031}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2151554915}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3365067489}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=600903227}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2427575633}) 2024-04-19T01:00:57.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940868153}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1704953700}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2579592588}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2007531494}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3399757112}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3251611511}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1875543168}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.744+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3158306643}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3467217576}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082088576}) 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2622028975}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=842014948}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1056038843}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=203584668}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=544665129}) 2024-04-19T01:00:57.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1112027363}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=188863660}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370940612}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3364306586}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4259635854}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2855613314}) 2024-04-19T01:00:57.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1730646057}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4088576389}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3252778286}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3602609935}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.748+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4107284426}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.894 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165634035}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3053461993}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393092736}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3173657660}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2071333115}) 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.895 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3410109282}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2601674139}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1120177516}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=535799911}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1259077724}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3624805509}) 2024-04-19T01:00:57.896 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109749558}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2673293654}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2347970899}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2421292807}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2350489209}) 2024-04-19T01:00:57.897 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3890559688}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.752+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2920035475}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1312820960}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827308480}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327762679}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.898 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4080323211}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2861759389}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2755158432}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=797640342}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2251561220}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=609698256}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.899 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658541928}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020616748}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210340555}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=413580413}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=619998169}) 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.900 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=842913222}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=173328271}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2570083596}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.756+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3767680272}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750085299}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=746930556}) 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.901 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964933741}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=357613959}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1025642830}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3546699998}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=235715153}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=521361576}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3275047614}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753499552}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638060254}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1923156661}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=347533146}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3435637196}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4249302613}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2189101702}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1802653060}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1952209126}) 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.760+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886208825}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490444187}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201638598}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1156681186}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1505149979}) 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=142046872}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2255851933}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2637471902}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3610105098}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15} crc {logm=3270406676,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=194635828}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18} crc {logm=3736607309,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=1687915564,mon_config_key=1209360200,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=111714647}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=16} crc {logm=3407973436,mdsmap=3140514009,mgr=3996069054,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=777000486,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=3584283674}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=37,mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18} crc {logm=3356102196,mdsmap=3140514009,mgr=1256687556,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=4012203639,mon_config_key=4212284759,monmap=4080741109,osd_metadata=438442333,osd_pg_creating=3787281371,osd_snap=4200688773}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=36,mdsmap=3,mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18} crc {logm=3452148635,mdsmap=2840952586,mgr=2914213742,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3636954238,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=3061610263}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=90202068,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=1553540132,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=2175937022}) 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=36,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=16} crc {logm=334978354,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=4212284759,monmap=2862618235,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=16} crc {logm=3196620601,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=94321782,mon_config_key=2834389005,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=3787281371,osd_snap=1348479156}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.764+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13} crc {logm=2173609772,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=576832608,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3787281371,osd_snap=2107719021}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=3492395341,osdmap=1074987345}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=4148021793,osdmap=1149926405}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=1098360154,osdmap=1046136059}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=1098360154,osdmap=4004146558}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=1560043582,osdmap=427750308}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=1277991274,osdmap=2848965479}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=3178715265,osdmap=1447966189}) 2024-04-19T01:00:57.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=165805231,osdmap=3542246614}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.768+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osd_snap=19,osdmap=81} crc {osd_snap=1098360154,osdmap=2876748291}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=96} crc {osdmap=2222464083}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=95} crc {osdmap=3093611335}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=95} crc {osdmap=478624995}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=95} crc {osdmap=4203840006}) 2024-04-19T01:00:57.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=89} crc {osdmap=3940885587}) 2024-04-19T01:00:57.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=94} crc {osdmap=1137443121}) 2024-04-19T01:00:57.914 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.914 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.914 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=91} crc {osdmap=2926431163}) 2024-04-19T01:00:57.914 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:00:57.914 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=91} crc {osdmap=3762852647}) 2024-04-19T01:00:57.914 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:00:57.780+0000 7fa88844f640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=97} crc {osdmap=3086096194}) 2024-04-19T01:00:58.442 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T01:00:58.443 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:00:58.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient: get_monmap_and_config 2024-04-19T01:00:58.588 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient: build_initial_monmap 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient: monmap: 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:58.591747+0000 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:58.591747+0000 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:58.589 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:58.590 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:58.590 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:58.590 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:58.590 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 Processor -- start 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 -- start start 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient: init 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient: _add_conns ranks=[3,10,2,8,1,0,4,6,7,9,5] 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8598072a40 0x7f85981390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.594 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): picked mon.noname-g con 0x7f8598072a40 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f859806dd60 0x7f8598072e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): picked mon.noname-f con 0x7f859806dd60 addr v2:172.21.15.161:3300/0 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 0x7f85980a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): picked mon.noname-b con 0x7f85980a4aa0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f859806f880 con 0x7f859806dd60 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f859806f2c0 con 0x7f85980a4aa0 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f859806c4d0 con 0x7f8598072a40 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f85a0005640 10 monclient(hunting): authenticate will time out at 1928.718904s 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 0x7f85980a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 0x7f85980a4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46216/0 (socket says 172.21.15.43:46216) 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 -- 172.21.15.43:0/136822143 learned_addr learned my addr 172.21.15.43:0/136822143 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): get_auth_request con 0x7f85980a4aa0 auth_method 0 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 1 --2- 172.21.15.43:0/136822143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8598072a40 0x7f85981390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 1 --2- 172.21.15.43:0/136822143 >> v2:172.21.15.161:3300/0 conn(0x7f859806dd60 0x7f8598072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): get_auth_request con 0x7f859806dd60 auth_method 0 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): get_auth_request con 0x7f8598072a40 auth_method 0 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859d5a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859dda1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient(hunting): handle_auth_done global_id 144173 payload 274 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 -- 172.21.15.43:0/136822143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8598072a40 msgr2=0x7f85981390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 --2- 172.21.15.43:0/136822143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8598072a40 0x7f85981390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 -- 172.21.15.43:0/136822143 >> v2:172.21.15.161:3300/0 conn(0x7f859806dd60 msgr2=0x7f8598072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 --2- 172.21.15.43:0/136822143 >> v2:172.21.15.161:3300/0 conn(0x7f859806dd60 0x7f8598072e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient: _finish_hunting 0 2024-04-19T01:00:58.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 monclient: found mon.noname-b 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.589+0000 7f859e5a2640 1 -- 172.21.15.43:0/136822143 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85980737c0 con 0x7f85980a4aa0 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859e5a2640 1 --2- 172.21.15.43:0/136822143 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 0x7f85980a4f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8590009e50 tx=0x7f859001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d3845631dc70d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 1 -- 172.21.15.43:0/136822143 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85900090d0 con 0x7f85980a4aa0 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: dump: 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:58.598 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:58.599 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: _finish_auth 0 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: _check_auth_tickets 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 1 -- 172.21.15.43:0/136822143 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85900a59a0 con 0x7f85980a4aa0 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 5 monclient: authenticate success, global_id 144173 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 1 -- 172.21.15.43:0/136822143 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f85900ad3f0 con 0x7f85980a4aa0 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 4 set_mon_vals no callback set 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f859cd9f640 10 monclient: dump: 2024-04-19T01:00:58.600 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:58.601 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 10 monclient: shutdown 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 -- 172.21.15.43:0/136822143 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 msgr2=0x7f85980a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 --2- 172.21.15.43:0/136822143 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 0x7f85980a4f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8590009e50 tx=0x7f859001c8f0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 -- 172.21.15.43:0/136822143 shutdown_connections 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 --2- 172.21.15.43:0/136822143 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f85980a4aa0 0x7f85980a4f00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 --2- 172.21.15.43:0/136822143 >> v2:172.21.15.161:3300/0 conn(0x7f859806dd60 0x7f8598072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 --2- 172.21.15.43:0/136822143 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f8598072a40 0x7f85981390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 -- 172.21.15.43:0/136822143 >> 172.21.15.43:0/136822143 conn(0x7f85980a6410 msgr2=0x7f859806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 -- 172.21.15.43:0/136822143 shutdown_connections 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 -- 172.21.15.43:0/136822143 wait complete. 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 10 monclient: build_initial_monmap 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 10 monclient: monmap: 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:58.603 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 librados: starting msgr at 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 librados: starting objecter 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 Processor -- start 2024-04-19T01:00:58.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 -- start start 2024-04-19T01:00:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 librados: setting wanted keys 2024-04-19T01:00:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 1 librados: calling monclient init 2024-04-19T01:00:58.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.593+0000 7f85a0005640 10 monclient: init 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _add_conns ranks=[0,5,3,4,6,1,2,7,8] 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 0x7f85981d1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): picked mon.noname-a con 0x7f859806dd60 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8598072a40 0x7f859813e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): picked mon.noname-h con 0x7f8598072a40 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85980a4aa0 0x7f859813e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): picked mon.noname-g con 0x7f85980a4aa0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f8598142ad0 con 0x7f859806dd60 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f8598142950 con 0x7f85980a4aa0 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8598142c50 con 0x7f8598072a40 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient(hunting): authenticate will time out at 1928.724153s 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 0x7f85981d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 0x7f85981d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55628/0 (socket says 172.21.15.43:55628) 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8598072a40 0x7f859813e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85980a4aa0 0x7f859813e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 -- 172.21.15.43:0/1908848158 learned_addr learned my addr 172.21.15.43:0/1908848158 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): get_auth_request con 0x7f859806dd60 auth_method 0 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): get_auth_request con 0x7f8598072a40 auth_method 0 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): get_auth_request con 0x7f85980a4aa0 auth_method 0 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient(hunting): handle_auth_done global_id 65403 payload 824 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 -- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8598072a40 msgr2=0x7f859813e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8598072a40 0x7f859813e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 -- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85980a4aa0 msgr2=0x7f859813e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85980a4aa0 0x7f859813e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient: _finish_hunting 0 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 monclient: found mon.noname-a 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:58.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 -- 172.21.15.43:0/1908848158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85981214c0 con 0x7f859806dd60 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859e5a2640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85980a4aa0 0x7f859813e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859dda1640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 0x7f85981d1130 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f859400e930 tx=0x7f859400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 client.?.objecter ms_handle_connect 0x7f859806dd60 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8594017070 con 0x7f859806dd60 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: dump: 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.614 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:58.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8594013470 con 0x7f859806dd60 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 5 monclient: authenticate success, global_id 65403 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _renew_subs 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85981217a0 con 0x7f859806dd60 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8594013a90 con 0x7f859806dd60 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:58.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858ffff640 4 set_mon_vals no callback set 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 monclient: dump: 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:58.617 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 client.65403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _renew_subs 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f859806ea20 con 0x7f859806dd60 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 librados: init done 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f859806e4d0 con 0x7f859806dd60 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f859401a070 con 0x7f859806dd60 2024-04-19T01:00:58.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f857c038e10 0x7f857c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(613..613 src has 501..613) ==== 6709+0+0 (secure 0 0 0) 0x7f859400f040 con 0x7f859806dd60 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 client.65403.objecter ms_dispatch 0x7f8598000ba0 osd_map(613..613 src has 501..613) 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 3 client.65403.objecter handle_osd_map got epochs [613,613] > 0 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 3 client.65403.objecter handle_osd_map decoding full epoch 613 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 20 client.65403.objecter dump_active .. 0 homeless 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f857c038e10 0x7f857c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 10 monclient: get_auth_request con 0x7f857c038e10 auth_method 0 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f859d5a0640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f857c038e10 0x7f857c03b2d0 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7f859813f800 tx=0x7f8588009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.597+0000 7f858e7fc640 10 client.65403.objecter ms_handle_connect 0x7f857c038e10 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.601+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8594021070 con 0x7f859806dd60 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.601+0000 7f858e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:58.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.601+0000 7f858e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:58.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f85a0005640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:00:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f85a0005640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:00:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f85a0005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8598120f00 con 0x7f859806dd60 2024-04-19T01:00:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f858e7fc640 1 -- 172.21.15.43:0/1908848158 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f8594011350 con 0x7f859806dd60 2024-04-19T01:00:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f858e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:00:58.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.829+0000 7f858e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:58.830 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:00:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 10 librados: watch_flush enter 2024-04-19T01:00:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 10 librados: watch_flush exit 2024-04-19T01:00:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 20 client.65403.objecter shutdown clearing up homeless session... 2024-04-19T01:00:58.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 10 client.65403.objecter successfully canceled tick 2024-04-19T01:00:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f857c038e10 msgr2=0x7f857c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f857c038e10 0x7f857c03b2d0 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7f859813f800 tx=0x7f8588009190 comp rx=0 tx=0).stop 2024-04-19T01:00:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 10 monclient: shutdown 2024-04-19T01:00:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 msgr2=0x7f85981d1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 0x7f85981d1130 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f859400e930 tx=0x7f859400edf0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 shutdown_connections 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f857c038e10 0x7f857c03b2d0 unknown :-1 s=CLOSED pgs=3602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f85980a4aa0 0x7f859813e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8598072a40 0x7f859813e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 --2- 172.21.15.43:0/1908848158 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f859806dd60 0x7f85981d1130 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 >> 172.21.15.43:0/1908848158 conn(0x7f85980a6410 msgr2=0x7f85981373e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 shutdown_connections 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 -- 172.21.15.43:0/1908848158 wait complete. 2024-04-19T01:00:58.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.833+0000 7f85a0005640 1 librados: shutdown 2024-04-19T01:00:58.849 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T01:00:58.849 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T01:00:58.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T01:00:58.849 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:00:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient: get_monmap_and_config 2024-04-19T01:00:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient: build_initial_monmap 2024-04-19T01:00:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient: monmap: 2024-04-19T01:00:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.978 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:58.980852+0000 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:58.980852+0000 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:58.979 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 Processor -- start 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 -- start start 2024-04-19T01:00:58.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient: init 2024-04-19T01:00:58.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:58.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient: _add_conns ranks=[0,8,7,1,4,6,2,3,9,10,5] 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16a812f1e0 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16a812f5c0 0x7f16a813de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): picked mon.noname-e con 0x7f16a812f5c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16a812fa20 0x7f16a8140360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): picked mon.noname-i con 0x7f16a812fa20 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16a809d3c0 con 0x7f16a812f1e0 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f16a809d150 con 0x7f16a812fa20 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f16a809e260 con 0x7f16a812f5c0 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.977+0000 7f16ad2e6640 10 monclient(hunting): authenticate will time out at 1929.107960s 2024-04-19T01:00:58.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a6ffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f16a812f1e0 0x7f16a809c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43086/0 (socket says 172.21.15.43:43086) 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a6ffd640 1 -- 172.21.15.43:0/198767957 learned_addr learned my addr 172.21.15.43:0/198767957 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16a812f5c0 0x7f16a813de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): get_auth_request con 0x7f16a812f5c0 auth_method 0 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a77fe640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16a812fa20 0x7f16a8140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a77fe640 10 monclient(hunting): get_auth_request con 0x7f16a812fa20 auth_method 0 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:58.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient(hunting): handle_auth_done global_id 144170 payload 274 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 -- 172.21.15.43:0/198767957 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16a812fa20 msgr2=0x7f16a8140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16a812fa20 0x7f16a8140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 -- 172.21.15.43:0/198767957 >> v1:172.21.15.43:6789/0 conn(0x7f16a812f1e0 legacy=0x7f16a809c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient: _finish_hunting 0 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 monclient: found mon.noname-e 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 1 -- 172.21.15.43:0/198767957 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668247879 0 0) 0x7f16a809d3c0 con 0x7f16a812f1e0 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:00:58.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 -- 172.21.15.43:0/198767957 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a809d6a0 con 0x7f16a812f5c0 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a67fc640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16a812f5c0 0x7f16a813de10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16900097f0 tx=0x7f169001c3a0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=c2c482e5dbcc9f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 1 -- 172.21.15.43:0/198767957 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16900b0070 con 0x7f16a812f5c0 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: dump: 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:58.993 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: _finish_auth 0 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 5 monclient: authenticate success, global_id 144170 2024-04-19T01:00:58.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 1 -- 172.21.15.43:0/198767957 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16900aa810 con 0x7f16a812f5c0 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 4 set_mon_vals no callback set 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 1 -- 172.21.15.43:0/198767957 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16900aa3a0 con 0x7f16a812f5c0 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16a5ffb640 10 monclient: dump: 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:58.995 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 10 monclient: shutdown 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 -- 172.21.15.43:0/198767957 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16a812f5c0 msgr2=0x7f16a813de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:58.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16a812f5c0 0x7f16a813de10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f16900097f0 tx=0x7f169001c3a0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 -- 172.21.15.43:0/198767957 shutdown_connections 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f16a812fa20 0x7f16a8140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/198767957 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f16a812f5c0 0x7f16a813de10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 -- 172.21.15.43:0/198767957 >> 172.21.15.43:0/198767957 conn(0x7f16a80a3df0 msgr2=0x7f16a80a4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 -- 172.21.15.43:0/198767957 shutdown_connections 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.981+0000 7f16ad2e6640 1 -- 172.21.15.43:0/198767957 wait complete. 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: build_initial_monmap 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: monmap: 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:58.997 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 librados: starting msgr at 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 librados: starting objecter 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 Processor -- start 2024-04-19T01:00:58.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- start start 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 librados: setting wanted keys 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 librados: calling monclient init 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: init 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: _add_conns ranks=[0,5,1,8,4,6,3,2,7] 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a812f1e0 0x7f16a812e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): picked mon.noname-a con 0x7f16a812f1e0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16a812f5c0 0x7f16a812e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): picked mon.noname-h con 0x7f16a812f5c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a812fa20 0x7f16a806fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): picked mon.noname-f con 0x7f16a812fa20 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8143450 con 0x7f16a812f1e0 2024-04-19T01:00:58.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f16a81432d0 con 0x7f16a812fa20 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f16a81435d0 con 0x7f16a812f5c0 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): authenticate will time out at 1929.113554s 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a812f1e0 0x7f16a812e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a812f1e0 0x7f16a812e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55640/0 (socket says 172.21.15.43:55640) 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 -- 172.21.15.43:0/1411759142 learned_addr learned my addr 172.21.15.43:0/1411759142 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): get_auth_request con 0x7f16a812f1e0 auth_method 0 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient(hunting): handle_auth_done global_id 65421 payload 824 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a77fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a812fa20 0x7f16a806fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16a812f5c0 msgr2=0x7f16a812e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16a812f5c0 0x7f16a812e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f16a812f5c0 0x7f16a812e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a812fa20 msgr2=0x7f16a806fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a812fa20 0x7f16a806fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 monclient: found mon.noname-a 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a809c910 con 0x7f16a812f1e0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a77fe640 10 monclient: get_auth_request con 0x7f16a812fa20 auth_method 0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a77fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16a812fa20 0x7f16a806fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a6ffd640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a812f1e0 0x7f16a812e070 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f169800c450 tx=0x7f169800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 client.?.objecter ms_handle_connect 0x7f16a812f1e0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1698018070 con 0x7f16a812f1e0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: dump: 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.002 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.003 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 5 monclient: authenticate success, global_id 65421 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16980047f0 con 0x7f16a812f1e0 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: _renew_subs 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a80712c0 con 0x7f16a812f1e0 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16980049a0 con 0x7f16a812f1e0 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 monclient: dump: 2024-04-19T01:00:59.004 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.005 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a57fa640 4 set_mon_vals no callback set 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 client.65421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: _renew_subs 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a812eaf0 con 0x7f16a812f1e0 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 librados: init done 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 0x7f16a802c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16a806d140 con 0x7f16a80708e0 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1698004b40 con 0x7f16a812f1e0 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 0x7f16a802c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient: get_auth_request con 0x7f16a80708e0 auth_method 0 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1688038e10 0x7f168803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(614..614 src has 501..614) ==== 6821+0+0 (secure 0 0 0) 0x7f1698048d00 con 0x7f16a812f1e0 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 10 client.65421.objecter ms_dispatch 0x7f16a8000ba0 osd_map(614..614 src has 501..614) 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 3 client.65421.objecter handle_osd_map got epochs [614,614] > 0 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 3 client.65421.objecter handle_osd_map decoding full epoch 614 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 10 monclient(hunting): handle_auth_done global_id 65430 payload 824 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f169f7fe640 20 client.65421.objecter dump_active .. 0 homeless 2024-04-19T01:00:59.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.985+0000 7f16a67fc640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16a8073230 con 0x7f16a80708e0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f16a77fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1688038e10 0x7f168803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 client.65421.objecter ms_handle_connect 0x7f16a80708e0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 client.65421.objecter resend_mon_ops 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f16a67fc640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 0x7f16a802c450 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16a809bd20 tx=0x7f1690019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f16a77fe640 10 monclient: get_auth_request con 0x7f1688038e10 auth_method 0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16900b9030 con 0x7f16a80708e0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 monclient: dump: 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.008 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1690003620 con 0x7f16a80708e0 2024-04-19T01:00:59.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 msgr2=0x7f16a802c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 0x7f16a802c450 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16a809bd20 tx=0x7f1690019140 comp rx=0 tx=0).stop 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 msgr2=0x7f16a802c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a80708e0 0x7f16a802c450 secure :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f16a809bd20 tx=0x7f1690019140 comp rx=0 tx=0).stop 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f16a77fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1688038e10 0x7f168803b2d0 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f16a8136250 tx=0x7f16a000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.989+0000 7f169f7fe640 10 client.65421.objecter ms_handle_connect 0x7f1688038e10 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16ad2e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 0x7f16a802c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16ad2e6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f16a80a1520 con 0x7f16a8070cc0 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 0x7f16a802c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient: get_auth_request con 0x7f16a8070cc0 auth_method 0 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 10 monclient(hunting): handle_auth_done global_id 65439 payload 824 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 1 -- 172.21.15.43:0/1411759142 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f16a80768c0 con 0x7f16a8070cc0 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f16a67fc640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 0x7f16a802c990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f169001aa50 tx=0x7f16900a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 client.65421.objecter ms_handle_connect 0x7f16a8070cc0 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 client.65421.objecter resend_mon_ops 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f16900b9040 con 0x7f16a8070cc0 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 monclient: dump: 2024-04-19T01:00:59.013 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.014 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7f16900039e0 con 0x7f16a8070cc0 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 msgr2=0x7f16a802c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 0x7f16a802c990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f169001aa50 tx=0x7f16900a8000 comp rx=0 tx=0).stop 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 msgr2=0x7f16a802c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.993+0000 7f169f7fe640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a8070cc0 0x7f16a802c990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.015 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 585265, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:59.016 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.017 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.018 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:59.019 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.020 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:59.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.023 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.026 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:59.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:59.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.033 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:59.036 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:59.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.038 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:59.039 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:59.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 10 librados: watch_flush enter 2024-04-19T01:00:59.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 10 librados: watch_flush exit 2024-04-19T01:00:59.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 20 client.65421.objecter shutdown clearing up homeless session... 2024-04-19T01:00:59.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 10 client.65421.objecter successfully canceled tick 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1688038e10 msgr2=0x7f168803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1688038e10 0x7f168803b2d0 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f16a8136250 tx=0x7f16a000b040 comp rx=0 tx=0).stop 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 10 monclient: shutdown 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a812f1e0 msgr2=0x7f16a812e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 --2- 172.21.15.43:0/1411759142 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f16a812f1e0 0x7f16a812e070 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f169800c450 tx=0x7f169800c910 comp rx=0 tx=0).stop 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16a77fe640 1 -- 172.21.15.43:0/1411759142 reap_dead start 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16a77fe640 1 -- 172.21.15.43:0/1411759142 reap_dead start 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 shutdown_connections 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 >> 172.21.15.43:0/1411759142 conn(0x7f16a80a3df0 msgr2=0x7f16a809e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 shutdown_connections 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 -- 172.21.15.43:0/1411759142 wait complete. 2024-04-19T01:00:59.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:58.997+0000 7f16ad2e6640 1 librados: shutdown 2024-04-19T01:00:59.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:59.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T01:00:59.043 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:00:59.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient: get_monmap_and_config 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient: monmap: 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:59.162803+0000 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:59.162803+0000 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:59.160 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:59.161 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 Processor -- start 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 -- start start 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient: init 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient: _add_conns ranks=[3,5,7,9,2,4,1,8,10,0,6] 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d88138a50 0x7f1d88138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): picked mon.noname-g con 0x7f1d88138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d880a5bb0 0x7f1d88139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): picked mon.noname-h con 0x7f1d880a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): picked mon.noname-i con 0x7f1d88137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f1d880a1020 con 0x7f1d88138a50 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f1d881398b0 con 0x7f1d880a5bb0 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1d880a0780 con 0x7f1d88137960 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8fd7f640 10 monclient(hunting): authenticate will time out at 1929.290106s 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d88138a50 0x7f1d88138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:58034/0 (socket says 172.21.15.43:58034) 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d880a5bb0 0x7f1d88139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 -- 172.21.15.43:0/3053120922 learned_addr learned my addr 172.21.15.43:0/3053120922 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 10 monclient(hunting): get_auth_request con 0x7f1d88137960 auth_method 0 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 -- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 msgr2=0x7f1d88137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): get_auth_request con 0x7f1d88138a50 auth_method 0 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 -- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 msgr2=0x7f1d88137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.167 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): get_auth_request con 0x7f1d880a5bb0 auth_method 0 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8e31c640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.168 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_done global_id 74786 payload 274 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 -- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 msgr2=0x7f1d88137dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 -- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d88138a50 msgr2=0x7f1d88138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d88138a50 0x7f1d88138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 monclient: found mon.noname-h 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.161+0000 7f1d8d31a640 1 -- 172.21.15.43:0/3053120922 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d880a0d40 con 0x7f1d880a5bb0 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8d31a640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d880a5bb0 0x7f1d88139370 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097f0 tx=0x7f1d7c01c370 comp rx=0 tx=0).ready entity=mon.5 client_cookie=d26a669bc546e56b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.169 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 1 -- 172.21.15.43:0/3053120922 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7c0b0070 con 0x7f1d880a5bb0 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: dump: 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.170 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 1 -- 172.21.15.43:0/3053120922 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0aa810 con 0x7f1d880a5bb0 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 1 -- 172.21.15.43:0/3053120922 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7c0a9300 con 0x7f1d880a5bb0 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 5 monclient: authenticate success, global_id 74786 2024-04-19T01:00:59.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 4 set_mon_vals no callback set 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8cb19640 10 monclient: dump: 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.172 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.173 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.173 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.173 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.173 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.173 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.173 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 10 monclient: shutdown 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d880a5bb0 msgr2=0x7f1d88139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d880a5bb0 0x7f1d88139370 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097f0 tx=0x7f1d7c01c370 comp rx=0 tx=0).stop 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/3053120922 shutdown_connections 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d88137960 0x7f1d88137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1d880a5bb0 0x7f1d88139370 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/3053120922 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f1d88138a50 0x7f1d88138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/3053120922 >> 172.21.15.43:0/3053120922 conn(0x7f1d880a7cb0 msgr2=0x7f1d880a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/3053120922 shutdown_connections 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/3053120922 wait complete. 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 10 monclient: monmap: 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.181 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 librados: starting msgr at 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 librados: starting objecter 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.165+0000 7f1d8fd7f640 1 Processor -- start 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- start start 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 librados: setting wanted keys 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 librados: calling monclient init 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: init 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: _add_conns ranks=[7,2,1,3,6,4,8,0,5] 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d880a5bb0 0x7f1d881d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): picked mon.noname-i con 0x7f1d880a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88137960 0x7f1d88136c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): picked mon.noname-b con 0x7f1d88137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d88138a50 0x7f1d8813ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): picked mon.noname-f con 0x7f1d88138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1d880a2100 con 0x7f1d88138a50 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d880a1f80 con 0x7f1d88137960 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1d88142810 con 0x7f1d880a5bb0 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): authenticate will time out at 1929.296777s 2024-04-19T01:00:59.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88137960 0x7f1d88136c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88137960 0x7f1d88136c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46228/0 (socket says 172.21.15.43:46228) 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 -- 172.21.15.43:0/4060406647 learned_addr learned my addr 172.21.15.43:0/4060406647 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): get_auth_request con 0x7f1d88137960 auth_method 0 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient(hunting): handle_auth_done global_id 144182 payload 824 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d880a5bb0 0x7f1d881d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8e31c640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d88138a50 0x7f1d8813ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d880a5bb0 msgr2=0x7f1d881d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1d880a5bb0 0x7f1d881d0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d88138a50 msgr2=0x7f1d8813ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1d88138a50 0x7f1d8813ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 monclient: found mon.noname-b 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8809fc90 con 0x7f1d88137960 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8d31a640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88137960 0x7f1d88136c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c01c1a0 tx=0x7f1d7c019110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 client.?.objecter ms_handle_connect 0x7f1d88137960 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7c0b0040 con 0x7f1d88137960 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: dump: 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.186 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0b9070 con 0x7f1d88137960 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 5 monclient: authenticate success, global_id 144182 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: _renew_subs 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d8809ee60 con 0x7f1d88137960 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7c0044b0 con 0x7f1d88137960 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:59.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: dump: 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.189 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d877fe640 4 set_mon_vals no callback set 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 client.144182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: _renew_subs 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d88122370 con 0x7f1d88137960 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 librados: init done 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1d7c0aac90 con 0x7f1d88137960 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 0x7f1d8813cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8806d140 con 0x7f1d88121db0 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d74038e10 0x7f1d7403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 0x7f1d8813cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient: get_auth_request con 0x7f1d88121db0 auth_method 0 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(614..614 src has 501..614) ==== 6821+0+0 (secure 0 0 0) 0x7f1d7c0e3240 con 0x7f1d88137960 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 client.144182.objecter ms_dispatch 0x7f1d88000ba0 osd_map(614..614 src has 501..614) 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 3 client.144182.objecter handle_osd_map got epochs [614,614] > 0 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 3 client.144182.objecter handle_osd_map decoding full epoch 614 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 20 client.144182.objecter dump_active .. 0 homeless 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_done global_id 144191 payload 824 2024-04-19T01:00:59.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d88122680 con 0x7f1d88121db0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8e31c640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d74038e10 0x7f1d7403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8e31c640 10 monclient: get_auth_request con 0x7f1d74038e10 auth_method 0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d8db1b640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 0x7f1d8813cfa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d78010570 tx=0x7f1d78010a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 client.144182.objecter ms_handle_connect 0x7f1d88121db0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 client.144182.objecter resend_mon_ops 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7800f040 con 0x7f1d88121db0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: dump: 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.192 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d7801c400 con 0x7f1d88121db0 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:59.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 msgr2=0x7f1d8813cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 0x7f1d8813cfa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d78010570 tx=0x7f1d78010a30 comp rx=0 tx=0).stop 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 msgr2=0x7f1d8813cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.169+0000 7f1d867fc640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88121db0 0x7f1d8813cfa0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d78010570 tx=0x7f1d78010a30 comp rx=0 tx=0).stop 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.173+0000 7f1d867fc640 10 client.144182.objecter ms_handle_connect 0x7f1d74038e10 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.173+0000 7f1d8e31c640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d74038e10 0x7f1d7403b2d0 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7f1d880a7500 tx=0x7f1d80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 0x7f1d8813d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1d88076190 con 0x7f1d8813d4e0 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 0x7f1d8813d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient: get_auth_request con 0x7f1d8813d4e0 auth_method 0 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 10 monclient(hunting): handle_auth_done global_id 144200 payload 824 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 1 -- 172.21.15.43:0/4060406647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d881d19e0 con 0x7f1d8813d4e0 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8db1b640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 0x7f1d8813d8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801b000 tx=0x7f1d780046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 client.144182.objecter ms_handle_connect 0x7f1d8813d4e0 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 client.144182.objecter resend_mon_ops 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1d7801ca20 con 0x7f1d8813d4e0 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 monclient: dump: 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.195 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f1d7801c5b0 con 0x7f1d8813d4e0 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:59.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 msgr2=0x7f1d8813d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 0x7f1d8813d8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801b000 tx=0x7f1d780046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 msgr2=0x7f1d8813d8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d867fc640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d8813d4e0 0x7f1d8813d8f0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801b000 tx=0x7f1d780046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 23807, 2024-04-19T01:00:59.197 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 3, 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.198 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:59.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:59.200 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:59.201 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.202 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.204 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.205 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.207 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.209 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:59.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:59.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.215 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:00:59.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.219 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.219 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.219 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:59.219 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:59.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 10 librados: watch_flush enter 2024-04-19T01:00:59.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 10 librados: watch_flush exit 2024-04-19T01:00:59.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 20 client.144182.objecter shutdown clearing up homeless session... 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 10 client.144182.objecter successfully canceled tick 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d74038e10 msgr2=0x7f1d7403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1d74038e10 0x7f1d7403b2d0 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7f1d880a7500 tx=0x7f1d80009040 comp rx=0 tx=0).stop 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 10 monclient: shutdown 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88137960 msgr2=0x7f1d88136c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 --2- 172.21.15.43:0/4060406647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1d88137960 0x7f1d88136c90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c01c1a0 tx=0x7f1d7c019110 comp rx=0 tx=0).stop 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8e31c640 1 -- 172.21.15.43:0/4060406647 reap_dead start 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8e31c640 1 -- 172.21.15.43:0/4060406647 reap_dead start 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 shutdown_connections 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 >> 172.21.15.43:0/4060406647 conn(0x7f1d880a7cb0 msgr2=0x7f1d880a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 shutdown_connections 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 -- 172.21.15.43:0/4060406647 wait complete. 2024-04-19T01:00:59.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.177+0000 7f1d8fd7f640 1 librados: shutdown 2024-04-19T01:00:59.222 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:59.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T01:00:59.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:00:59.339 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient: get_monmap_and_config 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient: monmap: 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:59.343038+0000 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:59.343038+0000 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.340 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.341 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.341 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.341 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.341 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:59.341 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:59.341 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 Processor -- start 2024-04-19T01:00:59.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 -- start start 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient: init 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient: _add_conns ranks=[9,10,6,2,3,0,8,7,5,1,4] 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 0x7f3df8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): picked mon.noname-a con 0x7f3df8138a50 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3df80a5bb0 0x7f3df8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): picked mon.noname-f con 0x7f3df80a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df8137960 0x7f3df8137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): picked mon.noname-d con 0x7f3df8137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f3df80a1020 con 0x7f3df8138a50 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3df81398b0 con 0x7f3df80a5bb0 2024-04-19T01:00:59.344 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3df80a0780 con 0x7f3df8137960 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfe75a640 10 monclient(hunting): authenticate will time out at 1929.470870s 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 0x7f3df8138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 0x7f3df8138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:55668/0 (socket says 172.21.15.43:55668) 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 1 -- 172.21.15.43:0/4224083047 learned_addr learned my addr 172.21.15.43:0/4224083047 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): get_auth_request con 0x7f3df8138a50 auth_method 0 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/4224083047 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df8137960 0x7f3df8137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): get_auth_request con 0x7f3df8137960 auth_method 0 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.345 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df77fe640 1 --2- 172.21.15.43:0/4224083047 >> v2:172.21.15.161:3300/0 conn(0x7f3df80a5bb0 0x7f3df8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df77fe640 10 monclient(hunting): get_auth_request con 0x7f3df80a5bb0 auth_method 0 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient(hunting): handle_auth_done global_id 144177 payload 274 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 -- 172.21.15.43:0/4224083047 >> v2:172.21.15.161:3300/0 conn(0x7f3df80a5bb0 msgr2=0x7f3df8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/4224083047 >> v2:172.21.15.161:3300/0 conn(0x7f3df80a5bb0 0x7f3df8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 -- 172.21.15.43:0/4224083047 >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 msgr2=0x7f3df8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:59.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/4224083047 >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 0x7f3df8138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 monclient: found mon.noname-d 2024-04-19T01:00:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:00:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3dfccf7640 1 -- 172.21.15.43:0/4224083047 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df80a0d40 con 0x7f3df8137960 2024-04-19T01:00:59.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.341+0000 7f3df7fff640 1 --2- 172.21.15.43:0/4224083047 >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 0x7f3df8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/4224083047 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df8137960 0x7f3df8137df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3df806e8c0 tx=0x7f3df001c340 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e3a3a6af0b9d7739 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 1 -- 172.21.15.43:0/4224083047 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3df00b0070 con 0x7f3df8137960 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: dump: 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.359 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 1 -- 172.21.15.43:0/4224083047 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df00aa810 con 0x7f3df8137960 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 5 monclient: authenticate success, global_id 144177 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:59.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 4 set_mon_vals no callback set 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 1 -- 172.21.15.43:0/4224083047 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3df00a9300 con 0x7f3df8137960 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3df6ffd640 10 monclient: dump: 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.361 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 10 monclient: shutdown 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 -- 172.21.15.43:0/4224083047 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df8137960 msgr2=0x7f3df8137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/4224083047 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df8137960 0x7f3df8137df0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3df806e8c0 tx=0x7f3df001c340 comp rx=0 tx=0).stop 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 -- 172.21.15.43:0/4224083047 shutdown_connections 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/4224083047 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df8137960 0x7f3df8137df0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/4224083047 >> v2:172.21.15.161:3300/0 conn(0x7f3df80a5bb0 0x7f3df8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/4224083047 >> v2:172.21.15.43:3300/0 conn(0x7f3df8138a50 0x7f3df8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 -- 172.21.15.43:0/4224083047 >> 172.21.15.43:0/4224083047 conn(0x7f3df80a7cb0 msgr2=0x7f3df80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 -- 172.21.15.43:0/4224083047 shutdown_connections 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 -- 172.21.15.43:0/4224083047 wait complete. 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 10 monclient: monmap: 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.363 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 librados: starting msgr at 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 librados: starting objecter 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.345+0000 7f3dfe75a640 1 Processor -- start 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- start start 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 librados: setting wanted keys 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 librados: calling monclient init 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: init 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: _add_conns ranks=[6,7,1,4,5,0,3,2,8] 2024-04-19T01:00:59.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 0x7f3df8136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): picked mon.noname-d con 0x7f3df80a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3df8137960 0x7f3df8136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): picked mon.noname-i con 0x7f3df8137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3df8138a50 0x7f3df813c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): picked mon.noname-f con 0x7f3df8138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3df80a2100 con 0x7f3df8138a50 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3df80a1f80 con 0x7f3df80a5bb0 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3df8142810 con 0x7f3df8137960 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): authenticate will time out at 1929.476727s 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 0x7f3df8136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 0x7f3df8136a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53880/0 (socket says 172.21.15.43:53880) 2024-04-19T01:00:59.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 -- 172.21.15.43:0/2763256090 learned_addr learned my addr 172.21.15.43:0/2763256090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): get_auth_request con 0x7f3df80a5bb0 auth_method 0 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3df8137960 0x7f3df8136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3df8138a50 0x7f3df813c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient(hunting): handle_auth_done global_id 144186 payload 824 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3df8137960 msgr2=0x7f3df8136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3df8137960 0x7f3df8136fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3df8138a50 msgr2=0x7f3df813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3df8138a50 0x7f3df813c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 monclient: found mon.noname-d 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3df809ee60 con 0x7f3df80a5bb0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfccf7640 10 monclient: get_auth_request con 0x7f3df8138a50 auth_method 0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient: get_auth_request con 0x7f3df8137960 auth_method 0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3df8138a50 0x7f3df813c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3df8137960 0x7f3df8136fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df7fff640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 0x7f3df8136a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e80 tx=0x7f3de800c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 client.?.objecter ms_handle_connect 0x7f3df80a5bb0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3de800cc60 con 0x7f3df80a5bb0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: dump: 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.368 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3de8019070 con 0x7f3df80a5bb0 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 5 monclient: authenticate success, global_id 144186 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: _renew_subs 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3df8122520 con 0x7f3df80a5bb0 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3de800ce20 con 0x7f3df80a5bb0 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 client.144186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df67fc640 4 set_mon_vals no callback set 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: dump: 2024-04-19T01:00:59.369 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: _renew_subs 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3df8121c80 con 0x7f3df80a5bb0 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 librados: init done 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 0x7f3df813ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f3df806d140 con 0x7f3df8121800 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3de8012050 con 0x7f3df80a5bb0 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 0x7f3df813ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient: get_auth_request con 0x7f3df8121800 auth_method 0 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3de4038e40 0x7f3de403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(614..614 src has 501..614) ==== 6821+0+0 (secure 0 0 0) 0x7f3de8049720 con 0x7f3df80a5bb0 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 client.144186.objecter ms_dispatch 0x7f3df8000ba0 osd_map(614..614 src has 501..614) 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 3 client.144186.objecter handle_osd_map got epochs [614,614] > 0 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 3 client.144186.objecter handle_osd_map decoding full epoch 614 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 10 monclient(hunting): handle_auth_done global_id 134383 payload 824 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 20 client.144186.objecter dump_active .. 0 homeless 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3df81d1980 con 0x7f3df8121800 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 client.144186.objecter ms_handle_connect 0x7f3df8121800 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 client.144186.objecter resend_mon_ops 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df77fe640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 0x7f3df813ceb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3df813f320 tx=0x7f3dec00eba0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3de4038e40 0x7f3de403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfccf7640 10 monclient: get_auth_request con 0x7f3de4038e40 auth_method 0 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dec017070 con 0x7f3df8121800 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: dump: 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3dec00ecd0 con 0x7f3df8121800 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 msgr2=0x7f3df813ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 0x7f3df813ceb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3df813f320 tx=0x7f3dec00eba0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 msgr2=0x7f3df813ceb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df8121800 0x7f3df813ceb0 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3df813f320 tx=0x7f3dec00eba0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3df4ff9640 10 client.144186.objecter ms_handle_connect 0x7f3de4038e40 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.349+0000 7f3dfccf7640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3de4038e40 0x7f3de403b300 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 crypto rx=0x7f3df806e8c0 tx=0x7f3df00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 0x7f3df80a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f3df809fd00 con 0x7f3df813d3f0 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 0x7f3df80a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient: get_auth_request con 0x7f3df813d3f0 auth_method 0 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 10 monclient(hunting): handle_auth_done global_id 134392 payload 824 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 1 -- 172.21.15.43:0/2763256090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3df813d800 con 0x7f3df813d3f0 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 client.144186.objecter ms_handle_connect 0x7f3df813d3f0 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df77fe640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 0x7f3df80a2e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3df813f320 tx=0x7f3dec0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 client.144186.objecter resend_mon_ops 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3dec00f040 con 0x7f3df813d3f0 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 monclient: dump: 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.376 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9614 (secure 0 0 0) 0x7f3dec003620 con 0x7f3df813d3f0 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 msgr2=0x7f3df80a2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 0x7f3df80a2e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3df813f320 tx=0x7f3dec0046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 msgr2=0x7f3df80a2e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3df4ff9640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3df813d3f0 0x7f3df80a2e20 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3df813f320 tx=0x7f3dec0046d0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 164061, 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:59.378 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.379 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.380 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:59.381 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.382 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.383 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.384 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:59.385 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.386 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.387 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.388 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.389 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.390 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.391 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.392 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:59.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.396 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.397 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.398 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.399 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:59.400 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 10 librados: watch_flush enter 2024-04-19T01:00:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 10 librados: watch_flush exit 2024-04-19T01:00:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 20 client.144186.objecter shutdown clearing up homeless session... 2024-04-19T01:00:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 10 client.144186.objecter successfully canceled tick 2024-04-19T01:00:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3de4038e40 msgr2=0x7f3de403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3de4038e40 0x7f3de403b300 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 crypto rx=0x7f3df806e8c0 tx=0x7f3df00a6000 comp rx=0 tx=0).stop 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 10 monclient: shutdown 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 msgr2=0x7f3df8136a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 0x7f3df8136a80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3de8009e80 tx=0x7f3de800c490 comp rx=0 tx=0).stop 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfccf7640 1 -- 172.21.15.43:0/2763256090 reap_dead start 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 shutdown_connections 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 --2- 172.21.15.43:0/2763256090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3df80a5bb0 0x7f3df8136a80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 >> 172.21.15.43:0/2763256090 conn(0x7f3df80a7cb0 msgr2=0x7f3df806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 shutdown_connections 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 -- 172.21.15.43:0/2763256090 wait complete. 2024-04-19T01:00:59.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.357+0000 7f3dfe75a640 1 librados: shutdown 2024-04-19T01:00:59.403 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:59.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T01:00:59.403 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T01:00:59.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient: get_monmap_and_config 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient: monmap: 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:59.527092+0000 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:59.527092+0000 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:59.525 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 Processor -- start 2024-04-19T01:00:59.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 -- start start 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient: init 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient: _add_conns ranks=[3,8,1,0,2,9,6,4,5,7,10] 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa754138cd0 0x7fa7541390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): picked mon.noname-g con 0x7fa754138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 0x7fa7541395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): picked mon.noname-e con 0x7fa7540a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fa7541358b0 addr v1:172.21.15.161:6789/0 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7540a3dc0 con 0x7fa7541358b0 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa7540a4410 con 0x7fa754138cd0 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa754139b30 con 0x7fa7540a5b80 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa759b89640 10 monclient(hunting): authenticate will time out at 1929.654196s 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 0x7fa7541395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 0x7fa7541395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39048/0 (socket says 172.21.15.43:39048) 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 -- 172.21.15.43:0/999943645 learned_addr learned my addr 172.21.15.43:0/999943645 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): get_auth_request con 0x7fa7540a5b80 auth_method 0 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 1 --2- 172.21.15.43:0/999943645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa754138cd0 0x7fa7541390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): get_auth_request con 0x7fa754138cd0 auth_method 0 2024-04-19T01:00:59.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa7537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient(hunting): handle_auth_done global_id 144179 payload 274 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 -- 172.21.15.43:0/999943645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa754138cd0 msgr2=0x7fa7541390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 --2- 172.21.15.43:0/999943645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa754138cd0 0x7fa7541390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 -- 172.21.15.43:0/999943645 >> v1:172.21.15.161:6789/0 conn(0x7fa7541358b0 legacy=0x7fa754135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.530 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 monclient: found mon.noname-e 2024-04-19T01:00:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:00:59.531 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.525+0000 7fa752ffd640 1 -- 172.21.15.43:0/999943645 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7540a40a0 con 0x7fa7540a5b80 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa752ffd640 1 --2- 172.21.15.43:0/999943645 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 0x7fa7541395f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7440098a0 tx=0x7fa74401c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=bc429246fc1b6bd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 1 -- 172.21.15.43:0/999943645 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7440b1070 con 0x7fa7540a5b80 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: dump: 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.537 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.538 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 1 -- 172.21.15.43:0/999943645 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7440ab830 con 0x7fa7540a5b80 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 1 -- 172.21.15.43:0/999943645 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7440aa320 con 0x7fa7540a5b80 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 5 monclient: authenticate success, global_id 144179 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 4 set_mon_vals no callback set 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7527fc640 10 monclient: dump: 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.539 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: shutdown 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- 172.21.15.43:0/999943645 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 msgr2=0x7fa7541395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 --2- 172.21.15.43:0/999943645 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 0x7fa7541395f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa7440098a0 tx=0x7fa74401c3e0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- 172.21.15.43:0/999943645 shutdown_connections 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 --2- 172.21.15.43:0/999943645 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa7540a5b80 0x7fa7541395f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 --2- 172.21.15.43:0/999943645 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa754138cd0 0x7fa7541390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- 172.21.15.43:0/999943645 >> 172.21.15.43:0/999943645 conn(0x7fa7540ab2f0 msgr2=0x7fa7540ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- 172.21.15.43:0/999943645 shutdown_connections 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- 172.21.15.43:0/999943645 wait complete. 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: monmap: 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:59.541 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 librados: starting msgr at 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 librados: starting objecter 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 Processor -- start 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- start start 2024-04-19T01:00:59.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 librados: setting wanted keys 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 librados: calling monclient init 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: init 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient: _add_conns ranks=[4,0,6,8,7,2,3,5,1] 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa7540a5b80 0x7fa7541d0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): picked mon.noname-c con 0x7fa7540a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7541358b0 0x7fa7540a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): picked mon.noname-a con 0x7fa7541358b0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754138cd0 0x7fa7540a0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): picked mon.noname-d con 0x7fa754138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fa754143ab0 con 0x7fa7541358b0 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fa7540a20a0 con 0x7fa7540a5b80 2024-04-19T01:00:59.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa754143c30 con 0x7fa754138cd0 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa759b89640 10 monclient(hunting): authenticate will time out at 1929.659706s 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa7540a5b80 0x7fa7541d0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa752ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7541358b0 0x7fa7540a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa753fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754138cd0 0x7fa7540a0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa7540a5b80 0x7fa7541d0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52808/0 (socket says 172.21.15.43:52808) 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 1 -- 172.21.15.43:0/1124141105 learned_addr learned my addr 172.21.15.43:0/1124141105 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 10 monclient(hunting): get_auth_request con 0x7fa7540a5b80 auth_method 0 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa7537fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa752ffd640 10 monclient(hunting): get_auth_request con 0x7fa7541358b0 auth_method 0 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.544 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa752ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa753fff640 10 monclient(hunting): get_auth_request con 0x7fa754138cd0 auth_method 0 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa753fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa753fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.529+0000 7fa753fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa753fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa753fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa753fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.545 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 10 monclient(hunting): handle_auth_done global_id 134401 payload 824 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754138cd0 msgr2=0x7fa7540a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754138cd0 0x7fa7540a0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7541358b0 msgr2=0x7fa7540a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fa7541358b0 0x7fa7540a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 monclient: found mon.noname-c 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75409c1f0 con 0x7fa7540a5b80 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa7537fe640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa7540a5b80 0x7fa7541d0e10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00e870 tx=0x7fa73c00ec70 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 client.?.objecter ms_handle_connect 0x7fa7540a5b80 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa73c017070 con 0x7fa7540a5b80 2024-04-19T01:00:59.546 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: dump: 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.547 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 5 monclient: authenticate success, global_id 134401 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73c00cdb0 con 0x7fa7540a5b80 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 monclient: _renew_subs 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa73c0138e0 con 0x7fa7540a5b80 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa754134d30 con 0x7fa7540a5b80 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa751ffb640 4 set_mon_vals no callback set 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:00:59.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: dump: 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 client.134401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 monclient: _renew_subs 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fa754133fb0 con 0x7fa7540a5b80 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 1 librados: init done 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa73c01a070 con 0x7fa7540a5b80 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 0x7fa7541328f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa75406d140 con 0x7fa754132510 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 0x7fa7541328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa734038e10 0x7fa73403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient: get_auth_request con 0x7fa754132510 auth_method 0 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(614..614 src has 501..614) ==== 6821+0+0 (secure 0 0 0) 0x7fa73c00f040 con 0x7fa7540a5b80 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 client.134401.objecter ms_dispatch 0x7fa754000ba0 osd_map(614..614 src has 501..614) 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 3 client.134401.objecter handle_osd_map got epochs [614,614] > 0 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 3 client.134401.objecter handle_osd_map decoding full epoch 614 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 20 client.134401.objecter dump_active .. 0 homeless 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 10 monclient(hunting): handle_auth_done global_id 144204 payload 824 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7540a3040 con 0x7fa754132510 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa753fff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa734038e10 0x7fa73403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 client.134401.objecter ms_handle_connect 0x7fa754132510 2024-04-19T01:00:59.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 client.134401.objecter resend_mon_ops 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa752ffd640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 0x7fa7541328f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7540a4380 tx=0x7fa7440191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa753fff640 10 monclient: get_auth_request con 0x7fa734038e10 auth_method 0 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7440bb030 con 0x7fa754132510 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: dump: 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.552 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa744003620 con 0x7fa754132510 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 msgr2=0x7fa7541328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 0x7fa7541328f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7540a4380 tx=0x7fa7440191e0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 msgr2=0x7fa7541328f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132510 0x7fa7541328f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa74bfff640 10 client.134401.objecter ms_handle_connect 0x7fa734038e10 2024-04-19T01:00:59.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.533+0000 7fa753fff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa734038e10 0x7fa73403b2d0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7fa754133b20 tx=0x7fa74c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 0x7fa75412bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa754068240 con 0x7fa754132e30 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 0x7fa75412bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient: get_auth_request con 0x7fa754132e30 auth_method 0 2024-04-19T01:00:59.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 10 monclient(hunting): handle_auth_done global_id 144213 payload 824 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 1 -- 172.21.15.43:0/1124141105 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7540693b0 con 0x7fa754132e30 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 client.134401.objecter ms_handle_connect 0x7fa754132e30 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa752ffd640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 0x7fa75412bbb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa74401aaf0 tx=0x7fa7440a9000 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 client.134401.objecter resend_mon_ops 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa7440bb040 con 0x7fa754132e30 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 monclient: dump: 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.556 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7fa744003b20 con 0x7fa754132e30 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 msgr2=0x7fa75412bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 0x7fa75412bbb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa74401aaf0 tx=0x7fa7440a9000 comp rx=0 tx=0).stop 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 msgr2=0x7fa75412bbb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa74bfff640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa754132e30 0x7fa75412bbb0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa74401aaf0 tx=0x7fa7440a9000 comp rx=0 tx=0).stop 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:59.558 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24171, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T01:00:59.559 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.560 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.561 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.562 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:59.563 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.564 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.565 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.566 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:59.567 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.568 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.569 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.570 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.571 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.572 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:59.573 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.574 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.575 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.576 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:59.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.579 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.580 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:59.581 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 10 librados: watch_flush enter 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 10 librados: watch_flush exit 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 20 client.134401.objecter shutdown clearing up homeless session... 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 10 client.134401.objecter successfully canceled tick 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa734038e10 msgr2=0x7fa73403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa734038e10 0x7fa73403b2d0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7fa754133b20 tx=0x7fa74c009040 comp rx=0 tx=0).stop 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 10 monclient: shutdown 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.583 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa7540a5b80 msgr2=0x7fa7541d0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 --2- 172.21.15.43:0/1124141105 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fa7540a5b80 0x7fa7541d0e10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00e870 tx=0x7fa73c00ec70 comp rx=0 tx=0).stop 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa753fff640 1 -- 172.21.15.43:0/1124141105 reap_dead start 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa753fff640 1 -- 172.21.15.43:0/1124141105 reap_dead start 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 shutdown_connections 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 >> 172.21.15.43:0/1124141105 conn(0x7fa7540ab2f0 msgr2=0x7fa7540a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 shutdown_connections 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 -- 172.21.15.43:0/1124141105 wait complete. 2024-04-19T01:00:59.584 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.541+0000 7fa759b89640 1 librados: shutdown 2024-04-19T01:00:59.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:59.584 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T01:00:59.585 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:00:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.701+0000 7f75d7ff4640 10 monclient: get_monmap_and_config 2024-04-19T01:00:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.701+0000 7f75d7ff4640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.701+0000 7f75d7ff4640 10 monclient: monmap: 2024-04-19T01:00:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.702 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:59.705571+0000 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:59.705571+0000 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.703 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:59.704 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.701+0000 7f75d7ff4640 1 Processor -- start 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.701+0000 7f75d7ff4640 1 -- start start 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.701+0000 7f75d7ff4640 10 monclient: init 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient: _add_conns ranks=[6,0,8,1,4,9,2,7,3,10,5] 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 0x7f75d009be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): picked mon.noname-d con 0x7f75d009ba70 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75d009c390 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d013ae30 0x7f75d013d220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): picked mon.noname-e con 0x7f75d013ae30 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:00:59.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d009def0 con 0x7f75d009c390 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f75d009d240 con 0x7f75d009ba70 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f75d009ea80 con 0x7f75d013ae30 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient(hunting): authenticate will time out at 1929.832565s 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 0x7f75d009be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 0x7f75d009be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53916/0 (socket says 172.21.15.43:53916) 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d558f640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f75d009c390 0x7f75d0139710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43096/0 (socket says 172.21.15.43:43096) 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 -- 172.21.15.43:0/3662987467 learned_addr learned my addr 172.21.15.43:0/3662987467 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): get_auth_request con 0x7f75d009ba70 auth_method 0 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.708 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 1 --2- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d013ae30 0x7f75d013d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): get_auth_request con 0x7f75d013ae30 auth_method 0 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d6591640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient(hunting): handle_auth_done global_id 144222 payload 274 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 -- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d013ae30 msgr2=0x7f75d013d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 --2- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d013ae30 0x7f75d013d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.709 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 -- 172.21.15.43:0/3662987467 >> v1:172.21.15.43:6789/0 conn(0x7f75d009c390 legacy=0x7f75d0139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 monclient: found mon.noname-d 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 -- 172.21.15.43:0/3662987467 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d009e4c0 con 0x7f75d009ba70 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d5d90640 1 --2- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 0x7f75d009be50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098a0 tx=0x7f75cc01c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7c8d755663f0f682 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 1 -- 172.21.15.43:0/3662987467 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f75cc0b0070 con 0x7f75d009ba70 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:00:59.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: dump: 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.711 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 1 -- 172.21.15.43:0/3662987467 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc0aa810 con 0x7f75d009ba70 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 5 monclient: authenticate success, global_id 144222 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d7ff4640 4 set_mon_vals no callback set 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 1 -- 172.21.15.43:0/3662987467 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f75cc0a9300 con 0x7f75d009ba70 2024-04-19T01:00:59.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.705+0000 7f75d4d8e640 10 monclient: dump: 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.713 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: shutdown 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 msgr2=0x7f75d009be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 0x7f75d009be50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098a0 tx=0x7f75cc01c440 comp rx=0 tx=0).stop 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/3662987467 shutdown_connections 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d013ae30 0x7f75d013d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/3662987467 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f75d009ba70 0x7f75d009be50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/3662987467 >> 172.21.15.43:0/3662987467 conn(0x7f75d00a77d0 msgr2=0x7f75d00a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/3662987467 shutdown_connections 2024-04-19T01:00:59.714 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/3662987467 wait complete. 2024-04-19T01:00:59.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: monmap: 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.722 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 librados: starting msgr at 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 librados: starting objecter 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 Processor -- start 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- start start 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 librados: setting wanted keys 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 librados: calling monclient init 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: init 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: _add_conns ranks=[2,3,7,6,5,0,8,4,1] 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f75d009ba70 0x7f75d01d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): picked mon.noname-b con 0x7f75d009ba70 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75d009c390 0x7f75d01369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): picked mon.noname-g con 0x7f75d009c390 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f75d013ae30 0x7f75d0070be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): picked mon.noname-i con 0x7f75d013ae30 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f75d009f630 con 0x7f75d009ba70 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f75d009f4b0 con 0x7f75d009c390 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f75d00a11d0 con 0x7f75d013ae30 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): authenticate will time out at 1929.837742s 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f75d009ba70 0x7f75d01d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f75d009ba70 0x7f75d01d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46256/0 (socket says 172.21.15.43:46256) 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 -- 172.21.15.43:0/4017271162 learned_addr learned my addr 172.21.15.43:0/4017271162 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): get_auth_request con 0x7f75d009ba70 auth_method 0 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75d009c390 0x7f75d01369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d6591640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f75d013ae30 0x7f75d0070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient(hunting): handle_auth_done global_id 144209 payload 824 2024-04-19T01:00:59.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f75d013ae30 msgr2=0x7f75d0070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f75d013ae30 0x7f75d0070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75d009c390 msgr2=0x7f75d01369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75d009c390 0x7f75d01369a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 monclient: found mon.noname-b 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d0123070 con 0x7f75d009ba70 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient: get_auth_request con 0x7f75d009c390 auth_method 0 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f75d009c390 0x7f75d01369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d5d90640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f75d009ba70 0x7f75d01d06b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f75cc01aaf0 tx=0x7f75cc0191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 client.?.objecter ms_handle_connect 0x7f75d009ba70 2024-04-19T01:00:59.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f75cc0b9070 con 0x7f75d009ba70 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: dump: 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.727 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc0044a0 con 0x7f75d009ba70 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 5 monclient: authenticate success, global_id 144209 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: _renew_subs 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d0123350 con 0x7f75d009ba70 2024-04-19T01:00:59.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f75cc0b0040 con 0x7f75d009ba70 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 monclient: dump: 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.729 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c3fff640 4 set_mon_vals no callback set 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 client.144209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: _renew_subs 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d009d3c0 con 0x7f75d009ba70 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 librados: init done 2024-04-19T01:00:59.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 0x7f75d002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f75d006d140 con 0x7f75d00718e0 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f75cc0aacf0 con 0x7f75d009ba70 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 0x7f75d002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient: get_auth_request con 0x7f75d00718e0 auth_method 0 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f75b4038e10 0x7f75b403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(614..614 src has 501..614) ==== 6821+0+0 (secure 0 0 0) 0x7f75cc0e2990 con 0x7f75d009ba70 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 10 client.144209.objecter ms_dispatch 0x7f75d0000ba0 osd_map(614..614 src has 501..614) 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 3 client.144209.objecter handle_osd_map got epochs [614,614] > 0 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 3 client.144209.objecter handle_osd_map decoding full epoch 614 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 10 monclient(hunting): handle_auth_done global_id 144197 payload 824 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75d558f640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75d00693b0 con 0x7f75d00718e0 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.709+0000 7f75c27fc640 20 client.144209.objecter dump_active .. 0 homeless 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 client.144209.objecter ms_handle_connect 0x7f75d00718e0 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 client.144209.objecter resend_mon_ops 2024-04-19T01:00:59.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75d558f640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 0x7f75d002c450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000f70 tx=0x7f75c400d950 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f75c40047f0 con 0x7f75d00718e0 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75d6591640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f75b4038e10 0x7f75b403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 monclient: dump: 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.733 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75d6591640 10 monclient: get_auth_request con 0x7f75b4038e10 auth_method 0 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f75c40072e0 con 0x7f75d00718e0 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 msgr2=0x7f75d002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 0x7f75d002c450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000f70 tx=0x7f75c400d950 comp rx=0 tx=0).stop 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 msgr2=0x7f75d002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d00718e0 0x7f75d002c450 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000f70 tx=0x7f75c400d950 comp rx=0 tx=0).stop 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75c27fc640 10 client.144209.objecter ms_handle_connect 0x7f75b4038e10 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.713+0000 7f75d6591640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f75b4038e10 0x7f75b403b2d0 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f75c8007380 tx=0x7f75c800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 0x7f75d002c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f75d0068240 con 0x7f75d0071cc0 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 0x7f75d002c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient: get_auth_request con 0x7f75d0071cc0 auth_method 0 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 10 monclient(hunting): handle_auth_done global_id 144206 payload 824 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 1 -- 172.21.15.43:0/4017271162 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f75d0072820 con 0x7f75d0071cc0 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d558f640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 0x7f75d002c990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000fc0 tx=0x7f75c4000ee0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 client.144209.objecter ms_handle_connect 0x7f75d0071cc0 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 client.144209.objecter resend_mon_ops 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f75c401aae0 con 0x7f75d0071cc0 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 monclient: dump: 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.736 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f75c401a670 con 0x7f75d0071cc0 2024-04-19T01:00:59.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 msgr2=0x7f75d002c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 0x7f75d002c990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000fc0 tx=0x7f75c4000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 msgr2=0x7f75d002c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75c27fc640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f75d0071cc0 0x7f75d002c990 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f75c4000fc0 tx=0x7f75c4000ee0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.738 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24347, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.739 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.740 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.741 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:59.742 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.743 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.744 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.745 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.746 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.747 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.748 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.749 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.750 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.751 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:59.752 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.753 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:59.754 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.755 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.756 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.757 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.758 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.759 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:59.760 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 10 librados: watch_flush enter 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 10 librados: watch_flush exit 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 20 client.144209.objecter shutdown clearing up homeless session... 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.717+0000 7f75d7ff4640 10 client.144209.objecter successfully canceled tick 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f75b4038e10 msgr2=0x7f75b403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f75b4038e10 0x7f75b403b2d0 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f75c8007380 tx=0x7f75c800b040 comp rx=0 tx=0).stop 2024-04-19T01:00:59.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 10 monclient: shutdown 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f75d009ba70 msgr2=0x7f75d01d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 --2- 172.21.15.43:0/4017271162 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f75d009ba70 0x7f75d01d06b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f75cc01aaf0 tx=0x7f75cc0191e0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d6591640 1 -- 172.21.15.43:0/4017271162 reap_dead start 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d6591640 1 -- 172.21.15.43:0/4017271162 reap_dead start 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 shutdown_connections 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 >> 172.21.15.43:0/4017271162 conn(0x7f75d00a77d0 msgr2=0x7f75d00a6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 shutdown_connections 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 -- 172.21.15.43:0/4017271162 wait complete. 2024-04-19T01:00:59.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.721+0000 7f75d7ff4640 1 librados: shutdown 2024-04-19T01:00:59.763 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:59.764 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T01:00:59.764 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:00:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient: get_monmap_and_config 2024-04-19T01:00:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient: monmap: 2024-04-19T01:00:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.887 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:00:59.890645+0000 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:00:59.890645+0000 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.888 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:00:59.889 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:00:59.889 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 Processor -- start 2024-04-19T01:00:59.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 -- start start 2024-04-19T01:00:59.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient: init 2024-04-19T01:00:59.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient: _add_conns ranks=[3,0,9,1,2,10,6,4,8,7,5] 2024-04-19T01:00:59.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a80a1fc0 0x7f70a80a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): picked mon.noname-g con 0x7f70a80a1fc0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a80a1280 addr v1:172.21.15.43:6789/0 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f70a813eb30 0x7f70a8140f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): picked mon.noname-a con 0x7f70a813eb30 addr v2:172.21.15.43:3300/0 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a80a05e0 con 0x7f70a80a1280 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f70a806c1a0 con 0x7f70a813eb30 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70a80a08c0 con 0x7f70a80a1fc0 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70ad044640 10 monclient(hunting): authenticate will time out at 1930.017965s 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a659c640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f70a80a1280 0x7f70a80a3e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43104/0 (socket says 172.21.15.43:43104) 2024-04-19T01:00:59.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a659c640 1 -- 172.21.15.43:0/2636737293 learned_addr learned my addr 172.21.15.43:0/2636737293 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 1 --2- 172.21.15.43:0/2636737293 >> v2:172.21.15.43:3300/0 conn(0x7f70a813eb30 0x7f70a8140f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 1 --2- 172.21.15.43:0/2636737293 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a80a1fc0 0x7f70a80a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): get_auth_request con 0x7f70a813eb30 auth_method 0 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request con 0x7f70a80a1fc0 auth_method 0 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient(hunting): handle_auth_done global_id 65475 payload 274 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 1 -- 172.21.15.43:0/2636737293 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a80a1fc0 msgr2=0x7f70a80a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 1 --2- 172.21.15.43:0/2636737293 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a80a1fc0 0x7f70a80a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 1 -- 172.21.15.43:0/2636737293 >> v1:172.21.15.43:6789/0 conn(0x7f70a80a1280 legacy=0x7f70a80a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 1 monclient: found mon.noname-a 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.889+0000 7f70a759e640 1 -- 172.21.15.43:0/2636737293 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a80a0020 con 0x7f70a813eb30 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a759e640 1 --2- 172.21.15.43:0/2636737293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a813eb30 0x7f70a8140f20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f70980097c0 tx=0x7f709801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3c6549514cb205 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 1 -- 172.21.15.43:0/2636737293 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70980b0070 con 0x7f70a813eb30 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: dump: 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.897 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 1 -- 172.21.15.43:0/2636737293 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70980aa810 con 0x7f70a813eb30 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 1 -- 172.21.15.43:0/2636737293 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70980a9300 con 0x7f70a813eb30 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 5 monclient: authenticate success, global_id 65475 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: get_monmap_and_config success 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 4 set_mon_vals no callback set 2024-04-19T01:00:59.898 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70a5d9b640 10 monclient: dump: 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.899 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: shutdown 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- 172.21.15.43:0/2636737293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a813eb30 msgr2=0x7f70a8140f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 --2- 172.21.15.43:0/2636737293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a813eb30 0x7f70a8140f20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f70980097c0 tx=0x7f709801c360 comp rx=0 tx=0).stop 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- 172.21.15.43:0/2636737293 shutdown_connections 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 --2- 172.21.15.43:0/2636737293 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f70a813eb30 0x7f70a8140f20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 --2- 172.21.15.43:0/2636737293 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a80a1fc0 0x7f70a80a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- 172.21.15.43:0/2636737293 >> 172.21.15.43:0/2636737293 conn(0x7f70a80a85c0 msgr2=0x7f70a80a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- 172.21.15.43:0/2636737293 shutdown_connections 2024-04-19T01:00:59.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- 172.21.15.43:0/2636737293 wait complete. 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: build_initial_monmap 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: monmap: 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:00:59.907 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 librados: starting msgr at 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 librados: starting objecter 2024-04-19T01:00:59.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 Processor -- start 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- start start 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 librados: setting wanted keys 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 librados: calling monclient init 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: init 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: _reopen_session rank -1 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient: _add_conns ranks=[1,2,3,6,7,4,5,8,0] 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a80a1280 0x7f70a81d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): picked mon.noname-f con 0x7f70a80a1280 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a80a1fc0 0x7f70a81d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): picked mon.noname-b con 0x7f70a80a1fc0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a813eb30 0x7f70a81d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): picked mon.noname-g con 0x7f70a813eb30 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:00:59.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70a80a4590 con 0x7f70a80a1280 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f70a80a4710 con 0x7f70a80a1fc0 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f70a8143c80 con 0x7f70a813eb30 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): _renew_subs 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.893+0000 7f70ad044640 10 monclient(hunting): authenticate will time out at 1930.023850s 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a80a1280 0x7f70a81d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a80a1fc0 0x7f70a81d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a80a1280 0x7f70a81d0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:42222/0 (socket says 172.21.15.43:42222) 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 1 -- 172.21.15.43:0/3765958033 learned_addr learned my addr 172.21.15.43:0/3765958033 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request con 0x7f70a80a1280 auth_method 0 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): get_auth_request con 0x7f70a80a1fc0 auth_method 0 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a813eb30 0x7f70a81d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): get_auth_request con 0x7f70a813eb30 auth_method 0 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.911 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient(hunting): handle_auth_done global_id 144218 payload 824 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a813eb30 msgr2=0x7f70a81d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f70a813eb30 0x7f70a81d16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a80a1280 msgr2=0x7f70a81d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a80a1280 0x7f70a81d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient: _finish_hunting 0 2024-04-19T01:00:59.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 monclient: found mon.noname-b 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a813ce30 con 0x7f70a80a1fc0 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a659c640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a80a1fc0 0x7f70a81d11b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c390 tx=0x7f709c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 client.?.objecter ms_handle_connect 0x7f70a80a1fc0 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f709c017070 con 0x7f70a80a1fc0 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: dump: 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.913 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:00:59.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f709c0070d0 con 0x7f70a80a1fc0 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 5 monclient: authenticate success, global_id 144218 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 monclient: _renew_subs 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a80732f0 con 0x7f70a80a1fc0 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f709c010300 con 0x7f70a80a1fc0 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 client.144218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: dump: 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.915 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a4d99640 4 set_mon_vals no callback set 2024-04-19T01:00:59.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 monclient: _renew_subs 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a813d6d0 con 0x7f70a80a1fc0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 1 librados: init done 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 0x7f70a8136eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70a80a1c00 con 0x7f70a8136ad0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f709c005af0 con 0x7f70a80a1fc0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 0x7f709403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(614..614 src has 501..614) ==== 6821+0+0 (secure 0 0 0) 0x7f709c048880 con 0x7f70a80a1fc0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 client.144218.objecter ms_dispatch 0x7f70a8000ba0 osd_map(614..614 src has 501..614) 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 3 client.144218.objecter handle_osd_map got epochs [614,614] > 0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 3 client.144218.objecter handle_osd_map decoding full epoch 614 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 0x7f70a8136eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 20 client.144218.objecter dump_active .. 0 homeless 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient: get_auth_request con 0x7f70a8136ad0 auth_method 0 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 0x7f709403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 10 monclient: get_auth_request con 0x7f7094038e10 auth_method 0 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_done global_id 144226 payload 824 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70a809ec50 con 0x7f70a8136ad0 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 client.144218.objecter ms_handle_connect 0x7f70a8136ad0 2024-04-19T01:00:59.918 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 client.144218.objecter resend_mon_ops 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a6d9d640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 0x7f70a8136eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009f30 tx=0x7f70a000f640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 msgr2=0x7f709403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 msgr2=0x7f709403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 0x7f709403b2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f70980ad000 tx=0x7f70980a6000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f70a759e640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 0x7f709403b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70a0019070 con 0x7f70a8136ad0 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.897+0000 7f708f7fe640 10 monclient: dump: 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.919 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f70a0003bb0 con 0x7f70a8136ad0 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:00:59.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 msgr2=0x7f70a8136eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 0x7f70a8136eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009f30 tx=0x7f70a000f640 comp rx=0 tx=0).stop 2024-04-19T01:00:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 msgr2=0x7f70a8136eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.901+0000 7f708f7fe640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a8136ad0 0x7f70a8136eb0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009f30 tx=0x7f70a000f640 comp rx=0 tx=0).stop 2024-04-19T01:00:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70ad044640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:00:59.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70ad044640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 0x7f70a809ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70ad044640 10 monclient(hunting): start opening mon connection 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f70a806d140 con 0x7f70a81373f0 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 0x7f70a809ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient: get_auth_request con 0x7f70a81373f0 auth_method 0 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 10 monclient(hunting): handle_auth_done global_id 144235 payload 824 2024-04-19T01:00:59.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 1 -- 172.21.15.43:0/3765958033 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70a813e680 con 0x7f70a81373f0 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f70a6d9d640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 0x7f70a809ba70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009f30 tx=0x7f70a0004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f708f7fe640 10 client.144218.objecter ms_handle_connect 0x7f70a81373f0 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.909+0000 7f708f7fe640 10 client.144218.objecter resend_mon_ops 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f70a001ead0 con 0x7f70a81373f0 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 10 monclient: dump: 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:00:59.924 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:00:59.925 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 24543, 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:00:59.926 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.927 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.928 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:00:59.929 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:00:59.930 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:00:59.931 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.932 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:00:59.933 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.934 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.935 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.936 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.937 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.938 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.939 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:00:59.940 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.941 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:00:59.942 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.943 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:00:59.944 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.945 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.946 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:00:59.947 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f70a001e660 con 0x7f70a81373f0 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 msgr2=0x7f70a809ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 0x7f70a809ba70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009f30 tx=0x7f70a0004a50 comp rx=0 tx=0).stop 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 msgr2=0x7f70a809ba70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f708f7fe640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f70a81373f0 0x7f70a809ba70 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009f30 tx=0x7f70a0004a50 comp rx=0 tx=0).stop 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 10 librados: watch_flush enter 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 10 librados: watch_flush exit 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 20 client.144218.objecter shutdown clearing up homeless session... 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 10 client.144218.objecter successfully canceled tick 2024-04-19T01:00:59.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 msgr2=0x7f709403b2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7094038e10 0x7f709403b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 10 monclient: shutdown 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a80a1fc0 msgr2=0x7f70a81d11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 --2- 172.21.15.43:0/3765958033 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f70a80a1fc0 0x7f70a81d11b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c390 tx=0x7f709c00c850 comp rx=0 tx=0).stop 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70a759e640 1 -- 172.21.15.43:0/3765958033 reap_dead start 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70a759e640 1 -- 172.21.15.43:0/3765958033 reap_dead start 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 shutdown_connections 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 >> 172.21.15.43:0/3765958033 conn(0x7f70a80a85c0 msgr2=0x7f70a813ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 shutdown_connections 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 -- 172.21.15.43:0/3765958033 wait complete. 2024-04-19T01:00:59.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:00:59.913+0000 7f70ad044640 1 librados: shutdown 2024-04-19T01:00:59.951 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:00:59.951 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T01:00:59.951 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T01:01:00.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient: get_monmap_and_config 2024-04-19T01:01:00.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient: monmap: 2024-04-19T01:01:00.068 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:00.071673+0000 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:00.071673+0000 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.069 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.070 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:00.070 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:00.070 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 Processor -- start 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 -- start start 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient: init 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient: _add_conns ranks=[3,5,2,10,6,0,9,7,4,1,8] 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): picked mon.noname-g con 0x7f7a200a4f50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:00.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7a200731a0 0x7f7a200a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): picked mon.noname-h con 0x7f7a200731a0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 0x7f7a200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a200a3370 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7a200a5330 con 0x7f7a200a3370 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7a20070660 con 0x7f7a200a4f50 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7a2009bc10 con 0x7f7a200731a0 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a27b1c640 10 monclient(hunting): authenticate will time out at 1930.199471s 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 0x7f7a200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 0x7f7a200a37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:46278/0 (socket says 172.21.15.43:46278) 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 1 -- 172.21.15.43:0/2228146281 learned_addr learned my addr 172.21.15.43:0/2228146281 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): get_auth_request con 0x7f7a200a3370 auth_method 0 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a260b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a258b8640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a250b7640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7a200731a0 0x7f7a200a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a258b8640 10 monclient(hunting): get_auth_request con 0x7f7a200a4f50 auth_method 0 2024-04-19T01:01:00.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a258b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a258b8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.069+0000 7f7a258b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a250b7640 10 monclient(hunting): get_auth_request con 0x7f7a200731a0 auth_method 0 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a250b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a250b7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a250b7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 10 monclient(hunting): handle_auth_done global_id 144227 payload 274 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 -- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7a200731a0 msgr2=0x7f7a200a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7a200731a0 0x7f7a200a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 -- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a200a4f50 msgr2=0x7f7a200a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 monclient: found mon.noname-b 2024-04-19T01:01:00.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:01:00.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 -- 172.21.15.43:0/2228146281 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a20133900 con 0x7f7a200a3370 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a260b9640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 0x7f7a200a37f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009e80 tx=0x7f7a1801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=840a267edca42f0c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 1 -- 172.21.15.43:0/2228146281 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a180090d0 con 0x7f7a200a3370 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: dump: 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.083 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 1 -- 172.21.15.43:0/2228146281 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a180a5a70 con 0x7f7a200a3370 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 5 monclient: authenticate success, global_id 144227 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 1 -- 172.21.15.43:0/2228146281 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a180a5c20 con 0x7f7a200a3370 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 4 set_mon_vals no callback set 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a248b6640 10 monclient: dump: 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.085 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 10 monclient: shutdown 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2228146281 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 msgr2=0x7f7a200a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 0x7f7a200a37f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009e80 tx=0x7f7a1801c8b0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2228146281 shutdown_connections 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a3370 0x7f7a200a37f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7a200731a0 0x7f7a200a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2228146281 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2228146281 >> 172.21.15.43:0/2228146281 conn(0x7f7a200a4460 msgr2=0x7f7a200a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2228146281 shutdown_connections 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2228146281 wait complete. 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 10 monclient: monmap: 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.088 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 librados: starting msgr at 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.073+0000 7f7a27b1c640 1 librados: starting objecter 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 Processor -- start 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- start start 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 librados: setting wanted keys 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 librados: calling monclient init 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: init 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: _add_conns ranks=[8,7,2,1,6,4,3,0,5] 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7a200731a0 0x7f7a201301a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): picked mon.noname-e con 0x7f7a200731a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7a200a3370 0x7f7a200a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): picked mon.noname-i con 0x7f7a200a3370 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:00.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): picked mon.noname-b con 0x7f7a200a4f50 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7a20140700 con 0x7f7a200a4f50 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7a20140580 con 0x7f7a200a3370 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7a20140880 con 0x7f7a200731a0 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): authenticate will time out at 1930.204648s 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7a200731a0 0x7f7a201301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7a200731a0 0x7f7a201301a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39072/0 (socket says 172.21.15.43:39072) 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): get_auth_request con 0x7f7a200a4f50 auth_method 0 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 1 -- 172.21.15.43:0/2901028236 learned_addr learned my addr 172.21.15.43:0/2901028236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): get_auth_request con 0x7f7a200731a0 auth_method 0 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7a200a3370 0x7f7a200a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 10 monclient(hunting): get_auth_request con 0x7f7a200a3370 auth_method 0 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient(hunting): handle_auth_done global_id 144236 payload 824 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7a200731a0 msgr2=0x7f7a201301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7a200731a0 0x7f7a201301a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7a200a3370 msgr2=0x7f7a200a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7a200a3370 0x7f7a200a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 monclient: found mon.noname-b 2024-04-19T01:01:00.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2009f210 con 0x7f7a200a4f50 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7a200731a0 0x7f7a201301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a260b9640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1240 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009e50 tx=0x7f7a180196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 client.?.objecter ms_handle_connect 0x7f7a200a4f50 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a180bc070 con 0x7f7a200a4f50 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: dump: 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.093 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a18003e10 con 0x7f7a200a4f50 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a180a5810 con 0x7f7a200a4f50 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 monclient: dump: 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.095 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a17fff640 4 set_mon_vals no callback set 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 5 monclient: authenticate success, global_id 144236 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: _renew_subs 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2009d3c0 con 0x7f7a200a4f50 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 client.144236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:00.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: _renew_subs 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2009ef30 con 0x7f7a200a4f50 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 librados: init done 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7a180039a0 con 0x7f7a200a4f50 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 0x7f7a201ce810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7a2006d140 con 0x7f7a2009ce70 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7a04038e00 0x7f7a0403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(615..615 src has 501..615) ==== 6765+0+0 (secure 0 0 0) 0x7f7a180e3d10 con 0x7f7a200a4f50 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 10 client.144236.objecter ms_dispatch 0x7f7a20000ba0 osd_map(615..615 src has 501..615) 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 3 client.144236.objecter handle_osd_map got epochs [615,615] > 0 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 3 client.144236.objecter handle_osd_map decoding full epoch 615 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a167fc640 20 client.144236.objecter dump_active .. 0 homeless 2024-04-19T01:01:00.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 0x7f7a201ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient: get_auth_request con 0x7f7a2009ce70 auth_method 0 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7a04038e00 0x7f7a0403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a250b7640 10 monclient: get_auth_request con 0x7f7a04038e00 auth_method 0 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 10 monclient(hunting): handle_auth_done global_id 19857 payload 824 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.077+0000 7f7a258b8640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a2009f610 con 0x7f7a2009ce70 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a258b8640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 0x7f7a201ce810 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7a10010cb0 tx=0x7f7a1000e5a0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 client.144236.objecter ms_handle_connect 0x7f7a2009ce70 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 client.144236.objecter resend_mon_ops 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a250b7640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7a04038e00 0x7f7a0403b2c0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f7a200a7640 tx=0x7f7a1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 client.144236.objecter ms_handle_connect 0x7f7a04038e00 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a10018070 con 0x7f7a2009ce70 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 monclient: dump: 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.099 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7a1001d450 con 0x7f7a2009ce70 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 msgr2=0x7f7a201ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 0x7f7a201ce810 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7a10010cb0 tx=0x7f7a1000e5a0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 msgr2=0x7f7a201ce810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:00.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.081+0000 7f7a167fc640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a2009ce70 0x7f7a201ce810 secure :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f7a10010cb0 tx=0x7f7a1000e5a0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a27b1c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 0x7f7a201cf160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a27b1c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f7a201cf6a0 con 0x7f7a201ced50 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 0x7f7a201cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient: get_auth_request con 0x7f7a201ced50 auth_method 0 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.085+0000 7f7a258b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a258b8640 10 monclient(hunting): handle_auth_done global_id 19866 payload 824 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a258b8640 1 -- 172.21.15.43:0/2901028236 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7a2009e280 con 0x7f7a201ced50 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a258b8640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 0x7f7a201cf160 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f7a10010cb0 tx=0x7f7a100046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 client.144236.objecter ms_handle_connect 0x7f7a201ced50 2024-04-19T01:01:00.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 client.144236.objecter resend_mon_ops 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7a1000eea0 con 0x7f7a201ced50 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 monclient: dump: 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.104 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9277 (secure 0 0 0) 0x7f7a10014990 con 0x7f7a201ced50 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 msgr2=0x7f7a201cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 0x7f7a201cf160 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f7a10010cb0 tx=0x7f7a100046d0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 msgr2=0x7f7a201cf160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:00.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a167fc640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f7a201ced50 0x7f7a201cf160 secure :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f7a10010cb0 tx=0x7f7a100046d0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 1003570, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:01:00.106 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.107 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.108 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:00.109 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.110 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:00.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.117 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.119 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:00.120 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.123 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.124 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:00.125 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.126 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:00.127 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 10 librados: watch_flush enter 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 10 librados: watch_flush exit 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 20 client.144236.objecter shutdown clearing up homeless session... 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 10 client.144236.objecter successfully canceled tick 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7a04038e00 msgr2=0x7f7a0403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7a04038e00 0x7f7a0403b2c0 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f7a200a7640 tx=0x7f7a1c008040 comp rx=0 tx=0).stop 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 10 monclient: shutdown 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a4f50 msgr2=0x7f7a200a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 --2- 172.21.15.43:0/2901028236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7a200a4f50 0x7f7a200a1240 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009e50 tx=0x7f7a180196b0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a260b9640 1 -- 172.21.15.43:0/2901028236 reap_dead start 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a260b9640 1 -- 172.21.15.43:0/2901028236 reap_dead start 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 shutdown_connections 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 >> 172.21.15.43:0/2901028236 conn(0x7f7a200a4460 msgr2=0x7f7a200a65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 shutdown_connections 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 -- 172.21.15.43:0/2901028236 wait complete. 2024-04-19T01:01:00.130 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.089+0000 7f7a27b1c640 1 librados: shutdown 2024-04-19T01:01:00.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:01:00.130 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T01:01:00.131 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:01:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient: get_monmap_and_config 2024-04-19T01:01:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient: monmap: 2024-04-19T01:01:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.248 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:00.251360+0000 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:00.251360+0000 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.249 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.250 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:00.250 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:00.250 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 Processor -- start 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 -- start start 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient: init 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient: _add_conns ranks=[8,6,10,7,1,2,9,4,3,0,5] 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 0x7f23401390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): picked mon.noname-e con 0x7f2340138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:00.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f23400a5b80 0x7f23401395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): picked mon.noname-d con 0x7f23400a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f23401358b0 0x7f2340135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): picked mon.noname-f con 0x7f23401358b0 addr v2:172.21.15.161:3300/0 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f23400a4410 con 0x7f23401358b0 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2340139b30 con 0x7f23400a5b80 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f23400a3dc0 con 0x7f2340138cd0 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2347794640 10 monclient(hunting): authenticate will time out at 1930.378659s 2024-04-19T01:01:00.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 0x7f23401390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f23400a5b80 0x7f23401395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 0x7f23401390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39088/0 (socket says 172.21.15.43:39088) 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 -- 172.21.15.43:0/927686321 learned_addr learned my addr 172.21.15.43:0/927686321 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): get_auth_request con 0x7f2340138cd0 auth_method 0 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): get_auth_request con 0x7f23400a5b80 auth_method 0 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345d31640 1 --2- 172.21.15.43:0/927686321 >> v2:172.21.15.161:3300/0 conn(0x7f23401358b0 0x7f2340135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345d31640 10 monclient(hunting): get_auth_request con 0x7f23401358b0 auth_method 0 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345d31640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345d31640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient(hunting): handle_auth_done global_id 144224 payload 274 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 -- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f23400a5b80 msgr2=0x7f23401395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 --2- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f23400a5b80 0x7f23401395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 -- 172.21.15.43:0/927686321 >> v2:172.21.15.161:3300/0 conn(0x7f23401358b0 msgr2=0x7f2340135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 --2- 172.21.15.43:0/927686321 >> v2:172.21.15.161:3300/0 conn(0x7f23401358b0 0x7f2340135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 monclient: found mon.noname-e 2024-04-19T01:01:00.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:01:00.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.249+0000 7f2345530640 1 -- 172.21.15.43:0/927686321 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23400a40a0 con 0x7f2340138cd0 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2345530640 1 --2- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 0x7f23401390b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2338009870 tx=0x7f233801c3c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=8454060b3f63b375 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 1 -- 172.21.15.43:0/927686321 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23380b0070 con 0x7f2340138cd0 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: dump: 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 1 -- 172.21.15.43:0/927686321 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23380aa810 con 0x7f2340138cd0 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 1 -- 172.21.15.43:0/927686321 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23380a9300 con 0x7f2340138cd0 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 5 monclient: authenticate success, global_id 144224 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 4 set_mon_vals no callback set 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f233ffff640 10 monclient: dump: 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.265 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 10 monclient: shutdown 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 -- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 msgr2=0x7f23401390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 --2- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 0x7f23401390b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2338009870 tx=0x7f233801c3c0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 -- 172.21.15.43:0/927686321 shutdown_connections 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 --2- 172.21.15.43:0/927686321 >> v2:172.21.15.161:3300/0 conn(0x7f23401358b0 0x7f2340135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 --2- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f23400a5b80 0x7f23401395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 --2- 172.21.15.43:0/927686321 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2340138cd0 0x7f23401390b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 -- 172.21.15.43:0/927686321 >> 172.21.15.43:0/927686321 conn(0x7f23400ab2f0 msgr2=0x7f23400ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 -- 172.21.15.43:0/927686321 shutdown_connections 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 1 -- 172.21.15.43:0/927686321 wait complete. 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.253+0000 7f2347794640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: monmap: 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:00.267 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.268 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 librados: starting msgr at 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 librados: starting objecter 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 Processor -- start 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- start start 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 librados: setting wanted keys 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 librados: calling monclient init 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: init 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: _add_conns ranks=[8,5,2,7,4,6,3,0,1] 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23400a5b80 0x7f2340134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): picked mon.noname-e con 0x7f23400a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:00.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401358b0 0x7f23401351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): picked mon.noname-h con 0x7f23401358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2340138cd0 0x7f23401d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): picked mon.noname-b con 0x7f2340138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f23400a2010 con 0x7f2340138cd0 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f23400a1e90 con 0x7f23401358b0 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f2340142aa0 con 0x7f23400a5b80 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): authenticate will time out at 1930.385188s 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23400a5b80 0x7f2340134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23400a5b80 0x7f2340134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:39092/0 (socket says 172.21.15.43:39092) 2024-04-19T01:01:00.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 -- 172.21.15.43:0/2450386629 learned_addr learned my addr 172.21.15.43:0/2450386629 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): get_auth_request con 0x7f23400a5b80 auth_method 0 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2340138cd0 0x7f23401d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): get_auth_request con 0x7f2340138cd0 auth_method 0 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401358b0 0x7f23401351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): get_auth_request con 0x7f23401358b0 auth_method 0 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient(hunting): handle_auth_done global_id 144233 payload 824 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401358b0 msgr2=0x7f23401351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401358b0 0x7f23401351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2340138cd0 msgr2=0x7f23401d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2340138cd0 0x7f23401d0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 monclient: found mon.noname-e 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:01:00.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2340137d20 con 0x7f23400a5b80 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345530640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23400a5b80 0x7f2340134c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23380ae040 tx=0x7f23380191c0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 client.?.objecter ms_handle_connect 0x7f23400a5b80 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23380b8020 con 0x7f23400a5b80 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: dump: 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.273 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23380aad70 con 0x7f23400a5b80 2024-04-19T01:01:00.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23380b0040 con 0x7f23400a5b80 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 monclient: dump: 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.275 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 5 monclient: authenticate success, global_id 144233 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233f7fe640 4 set_mon_vals no callback set 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: _renew_subs 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23401356e0 con 0x7f23400a5b80 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 client.144233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: _renew_subs 2024-04-19T01:01:00.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f234009e5d0 con 0x7f23400a5b80 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 librados: init done 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 0x7f23401d1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f234006d140 con 0x7f23400a06c0 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f23380bb070 con 0x7f23400a5b80 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 0x7f23401d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f232c038e10 0x7f232c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient: get_auth_request con 0x7f23400a06c0 auth_method 0 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(615..615 src has 501..615) ==== 6765+0+0 (secure 0 0 0) 0x7f23380e26a0 con 0x7f23400a5b80 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 10 client.144233.objecter ms_dispatch 0x7f2340000ba0 osd_map(615..615 src has 501..615) 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 3 client.144233.objecter handle_osd_map got epochs [615,615] > 0 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 3 client.144233.objecter handle_osd_map decoding full epoch 615 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f233dffb640 20 client.144233.objecter dump_active .. 0 homeless 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f232c038e10 0x7f232c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 10 monclient(hunting): handle_auth_done global_id 74813 payload 824 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2344d2f640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23400693b0 con 0x7f23400a06c0 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.257+0000 7f2345d31640 10 monclient: get_auth_request con 0x7f232c038e10 auth_method 0 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f2344d2f640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 0x7f23401d1180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23400a7360 tx=0x7f23340115a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 client.144233.objecter ms_handle_connect 0x7f23400a06c0 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 client.144233.objecter resend_mon_ops 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2334003e10 con 0x7f23400a06c0 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 monclient: dump: 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.279 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 client.144233.objecter ms_handle_connect 0x7f232c038e10 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f2345d31640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f232c038e10 0x7f232c03b2d0 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7f23400a03b0 tx=0x7f2330008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23340039a0 con 0x7f23400a06c0 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:00.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 msgr2=0x7f23401d1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 0x7f23401d1180 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23400a7360 tx=0x7f23340115a0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 msgr2=0x7f23401d1180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:00.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.261+0000 7f233dffb640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23400a06c0 0x7f23401d1180 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f23400a7360 tx=0x7f23340115a0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.265+0000 7f2347794640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.265+0000 7f2347794640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 0x7f234013bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.265+0000 7f2347794640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.265+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f2340068240 con 0x7f23401d16c0 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 0x7f234013bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient: get_auth_request con 0x7f23401d16c0 auth_method 0 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 10 monclient(hunting): handle_auth_done global_id 74822 payload 824 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 1 -- 172.21.15.43:0/2450386629 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f234009ee80 con 0x7f23401d16c0 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2344d2f640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 0x7f234013bcc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f234006cb10 tx=0x7f2334004a50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 client.144233.objecter ms_handle_connect 0x7f23401d16c0 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 client.144233.objecter resend_mon_ops 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f23340048a0 con 0x7f23401d16c0 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 monclient: dump: 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.284 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.285 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f2334004430 con 0x7f23401d16c0 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 msgr2=0x7f234013bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 0x7f234013bcc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f234006cb10 tx=0x7f2334004a50 comp rx=0 tx=0).stop 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 msgr2=0x7f234013bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f233dffb640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f23401d16c0 0x7f234013bcc0 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f234006cb10 tx=0x7f2334004a50 comp rx=0 tx=0).stop 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:00.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 439884, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:01:00.287 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 4, 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.288 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.289 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:00.290 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.291 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:00.293 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.298 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:00.299 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.300 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.301 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.302 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.303 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.304 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.305 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:00.306 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:00.307 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 10 librados: watch_flush enter 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 10 librados: watch_flush exit 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 20 client.144233.objecter shutdown clearing up homeless session... 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 10 client.144233.objecter successfully canceled tick 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f232c038e10 msgr2=0x7f232c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f232c038e10 0x7f232c03b2d0 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7f23400a03b0 tx=0x7f2330008040 comp rx=0 tx=0).stop 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 10 monclient: shutdown 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23400a5b80 msgr2=0x7f2340134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 --2- 172.21.15.43:0/2450386629 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f23400a5b80 0x7f2340134c60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f23380ae040 tx=0x7f23380191c0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2345d31640 1 -- 172.21.15.43:0/2450386629 reap_dead start 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2345d31640 1 -- 172.21.15.43:0/2450386629 reap_dead start 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 shutdown_connections 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 >> 172.21.15.43:0/2450386629 conn(0x7f23400ab2f0 msgr2=0x7f2340137610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 shutdown_connections 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 -- 172.21.15.43:0/2450386629 wait complete. 2024-04-19T01:01:00.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.269+0000 7f2347794640 1 librados: shutdown 2024-04-19T01:01:00.310 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:01:00.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T01:01:00.311 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient: get_monmap_and_config 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient: monmap: 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:00.431369+0000 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:00.431369+0000 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:00.429 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 Processor -- start 2024-04-19T01:01:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 -- start start 2024-04-19T01:01:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient: init 2024-04-19T01:01:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient: _add_conns ranks=[0,3,4,8,10,7,5,9,6,1,2] 2024-04-19T01:01:00.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e440a2ac0 addr v1:172.21.15.43:6789/0 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e44136e80 0x7f6e4413bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): picked mon.noname-g con 0x7f6e44136e80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6e4413c220 0x7f6e4413e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4413c220 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e440a35b0 con 0x7f6e440a2ac0 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6e440718c0 con 0x7f6e44136e80 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f6e44072fb0 con 0x7f6e4413c220 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e4957d640 10 monclient(hunting): authenticate will time out at 1930.558536s 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e42ffd640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6e440a2ac0 0x7f6e440a2ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:43112/0 (socket says 172.21.15.43:43112) 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e42ffd640 1 -- 172.21.15.43:0/4143947064 learned_addr learned my addr 172.21.15.43:0/4143947064 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:01:00.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6e4413c220 0x7f6e4413e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): get_auth_request con 0x7f6e4413c220 auth_method 0 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e427fc640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e44136e80 0x7f6e4413bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e427fc640 10 monclient(hunting): get_auth_request con 0x7f6e44136e80 auth_method 0 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e427fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient(hunting): handle_auth_done global_id 134410 payload 274 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 1 -- 172.21.15.43:0/4143947064 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e44136e80 msgr2=0x7f6e4413bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e44136e80 0x7f6e4413bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e41ffb640 1 -- 172.21.15.43:0/4143947064 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223804492 0 0) 0x7f6e440a35b0 con 0x7f6e440a2ac0 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 1 -- 172.21.15.43:0/4143947064 >> v1:172.21.15.43:6789/0 conn(0x7f6e440a2ac0 legacy=0x7f6e440a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 1 monclient: found mon.noname-c 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e437fe640 1 -- 172.21.15.43:0/4143947064 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e44073290 con 0x7f6e4413c220 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.429+0000 7f6e41ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e437fe640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6e4413c220 0x7f6e4413e610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6e340097f0 tx=0x7f6e3401c3a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6d324831cb7c2cb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 1 -- 172.21.15.43:0/4143947064 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e340b0070 con 0x7f6e4413c220 2024-04-19T01:01:00.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: dump: 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.444 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 1 -- 172.21.15.43:0/4143947064 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e340aa810 con 0x7f6e4413c220 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 5 monclient: authenticate success, global_id 134410 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 1 -- 172.21.15.43:0/4143947064 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e340a9300 con 0x7f6e4413c220 2024-04-19T01:01:00.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 4 set_mon_vals no callback set 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e41ffb640 10 monclient: dump: 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.446 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: shutdown 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- 172.21.15.43:0/4143947064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6e4413c220 msgr2=0x7f6e4413e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6e4413c220 0x7f6e4413e610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6e340097f0 tx=0x7f6e3401c3a0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- 172.21.15.43:0/4143947064 shutdown_connections 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6e4413c220 0x7f6e4413e610 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 --2- 172.21.15.43:0/4143947064 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e44136e80 0x7f6e4413bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- 172.21.15.43:0/4143947064 >> 172.21.15.43:0/4143947064 conn(0x7f6e440a63e0 msgr2=0x7f6e4406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- 172.21.15.43:0/4143947064 shutdown_connections 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- 172.21.15.43:0/4143947064 wait complete. 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: monmap: 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:00.448 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 librados: starting msgr at 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 librados: starting objecter 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 Processor -- start 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- start start 2024-04-19T01:01:00.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 librados: setting wanted keys 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 librados: calling monclient init 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: init 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient: _add_conns ranks=[3,7,5,0,6,4,8,2,1] 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e440a2ac0 0x7f6e4413aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): picked mon.noname-g con 0x7f6e440a2ac0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e44136e80 0x7f6e441ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): picked mon.noname-i con 0x7f6e44136e80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6e4413c220 0x7f6e44138750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): picked mon.noname-h con 0x7f6e4413c220 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6e44141370 con 0x7f6e440a2ac0 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6e4409fd50 con 0x7f6e4413c220 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6e441414f0 con 0x7f6e44136e80 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.433+0000 7f6e4957d640 10 monclient(hunting): authenticate will time out at 1930.563770s 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e44136e80 0x7f6e441ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6e4413c220 0x7f6e44138750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e44136e80 0x7f6e441ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:58072/0 (socket says 172.21.15.43:58072) 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 1 -- 172.21.15.43:0/779959247 learned_addr learned my addr 172.21.15.43:0/779959247 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): get_auth_request con 0x7f6e44136e80 auth_method 0 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e440a2ac0 0x7f6e4413aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): get_auth_request con 0x7f6e4413c220 auth_method 0 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): get_auth_request con 0x7f6e440a2ac0 auth_method 0 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient(hunting): handle_auth_done global_id 19884 payload 824 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e44136e80 msgr2=0x7f6e441ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e44136e80 0x7f6e441ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6e4413c220 msgr2=0x7f6e44138750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6e4413c220 0x7f6e44138750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 monclient: found mon.noname-g 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4411f6e0 con 0x7f6e440a2ac0 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e42ffd640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e440a2ac0 0x7f6e4413aef0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00c770 tx=0x7f6e3c00cc30 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 client.?.objecter ms_handle_connect 0x7f6e440a2ac0 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e3c018070 con 0x7f6e440a2ac0 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: dump: 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.454 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 5 monclient: authenticate success, global_id 19884 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 monclient: _renew_subs 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T01:01:00.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e4411fc40 con 0x7f6e440a2ac0 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c004d80 con 0x7f6e440a2ac0 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e417fa640 4 set_mon_vals no callback set 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 client.19884.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 monclient: _renew_subs 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 monclient: _send_mon_message to mon.g at v2:172.21.15.161:3301/0 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e4413a390 con 0x7f6e440a2ac0 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 1 librados: init done 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e3c00aa50 con 0x7f6e440a2ac0 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: dump: 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 0x7f6e44138c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6e4406d140 con 0x7f6e441ceef0 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 0x7f6e44138c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient: get_auth_request con 0x7f6e441ceef0 auth_method 0 2024-04-19T01:01:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.3 v2:172.21.15.161:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6e3c00abf0 con 0x7f6e440a2ac0 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 10 monclient(hunting): handle_auth_done global_id 74671 payload 824 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e4411fa80 con 0x7f6e441ceef0 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6e24038e10 0x7f6e2403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.3 v2:172.21.15.161:3301/0 5 ==== osd_map(615..615 src has 501..615) ==== 6765+0+0 (secure 0 0 0) 0x7f6e3c048f90 con 0x7f6e440a2ac0 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 client.19884.objecter ms_dispatch 0x7f6e44000ba0 osd_map(615..615 src has 501..615) 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 3 client.19884.objecter handle_osd_map got epochs [615,615] > 0 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 3 client.19884.objecter handle_osd_map decoding full epoch 615 2024-04-19T01:01:00.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 20 client.19884.objecter dump_active .. 0 homeless 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 client.19884.objecter ms_handle_connect 0x7f6e441ceef0 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e427fc640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 0x7f6e44138c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c010860 tx=0x7f6e2c010d20 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 client.19884.objecter resend_mon_ops 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6e24038e10 0x7f6e2403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e437fe640 10 monclient: get_auth_request con 0x7f6e24038e10 auth_method 0 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e2c018070 con 0x7f6e441ceef0 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.437+0000 7f6e3b7fe640 10 monclient: dump: 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.460 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 10 client.19884.objecter ms_handle_connect 0x7f6e24038e10 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e437fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6e24038e10 0x7f6e2403b2d0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f6e340ad000 tx=0x7f6e340a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e2c00e9e0 con 0x7f6e441ceef0 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:00.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 msgr2=0x7f6e44138c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 0x7f6e44138c90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c010860 tx=0x7f6e2c010d20 comp rx=0 tx=0).stop 2024-04-19T01:01:00.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 msgr2=0x7f6e44138c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:00.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.441+0000 7f6e3b7fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441ceef0 0x7f6e44138c90 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c010860 tx=0x7f6e2c010d20 comp rx=0 tx=0).stop 2024-04-19T01:01:00.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e4957d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e4957d640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 0x7f6e4409ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e4957d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6e44073570 con 0x7f6e441391d0 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 0x7f6e4409ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient: get_auth_request con 0x7f6e441391d0 auth_method 0 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.445+0000 7f6e427fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.449+0000 7f6e427fc640 10 monclient(hunting): handle_auth_done global_id 74680 payload 824 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.449+0000 7f6e427fc640 1 -- 172.21.15.43:0/779959247 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e441395e0 con 0x7f6e441391d0 2024-04-19T01:01:00.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.449+0000 7f6e3b7fe640 10 client.19884.objecter ms_handle_connect 0x7f6e441391d0 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.449+0000 7f6e3b7fe640 10 client.19884.objecter resend_mon_ops 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.449+0000 7f6e427fc640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 0x7f6e4409ca80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c010860 tx=0x7f6e2c0046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6e2c02a020 con 0x7f6e441391d0 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 10 monclient: dump: 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.465 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7f6e2c0133d0 con 0x7f6e441391d0 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 msgr2=0x7f6e4409ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 0x7f6e4409ca80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c010860 tx=0x7f6e2c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 msgr2=0x7f6e4409ca80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e3b7fe640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6e441391d0 0x7f6e4409ca80 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c010860 tx=0x7f6e2c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 110, 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 440072, 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:00.467 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.469 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:00.470 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:00.471 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:00.472 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.474 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.475 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.476 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.477 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.478 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.479 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:00.480 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.481 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.482 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.483 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:00.484 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.485 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.486 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.487 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.488 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:00.489 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 10 librados: watch_flush enter 2024-04-19T01:01:00.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 10 librados: watch_flush exit 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 20 client.19884.objecter shutdown clearing up homeless session... 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 10 client.19884.objecter successfully canceled tick 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6e24038e10 msgr2=0x7f6e2403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6e24038e10 0x7f6e2403b2d0 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f6e340ad000 tx=0x7f6e340a6000 comp rx=0 tx=0).stop 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 10 monclient: shutdown 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e440a2ac0 msgr2=0x7f6e4413aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 --2- 172.21.15.43:0/779959247 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6e440a2ac0 0x7f6e4413aef0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c00c770 tx=0x7f6e3c00cc30 comp rx=0 tx=0).stop 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e437fe640 1 -- 172.21.15.43:0/779959247 reap_dead start 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e437fe640 1 -- 172.21.15.43:0/779959247 reap_dead start 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 shutdown_connections 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 >> 172.21.15.43:0/779959247 conn(0x7f6e440a63e0 msgr2=0x7f6e4413c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.457+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 shutdown_connections 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.461+0000 7f6e4957d640 1 -- 172.21.15.43:0/779959247 wait complete. 2024-04-19T01:01:00.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.461+0000 7f6e4957d640 1 librados: shutdown 2024-04-19T01:01:00.493 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:01:00.494 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['g', 'i'] 2024-04-19T01:01:00.494 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-19T01:01:00.494 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.g 2024-04-19T01:01:00.494 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-04-19T01:01:00.494 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-04-19T01:01:00.494 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:01:00.541 INFO:tasks.ceph.mon.g:Stopped 2024-04-19T01:01:00.541 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-04-19T01:01:00.541 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-04-19T01:01:00.541 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-04-19T01:01:00.541 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:01:00.576 INFO:tasks.ceph.mon.i:Stopped 2024-04-19T01:01:00.576 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 7 2024-04-19T01:01:00.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:01:00.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient: get_monmap_and_config 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient: monmap: 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:00.707915+0000 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:00.707915+0000 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.705 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.706 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.707 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.707 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:00.707 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:00.707 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 Processor -- start 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 -- start start 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient: init 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient: _add_conns ranks=[6,9,5,7,3,8,1,0,4,2,10] 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 0x7f3a38136f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): picked mon.noname-d con 0x7f3a380a2ac0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:00.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3a380a2ea0 0x7f3a380a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): picked mon.noname-a con 0x7f3a380a2ea0 addr v2:172.21.15.43:3300/0 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3a380a32c0 0x7f3a380a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): picked mon.noname-h con 0x7f3a380a32c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f3a38071890 con 0x7f3a380a2ea0 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f3a38071cb0 con 0x7f3a380a32c0 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3a38072fd0 con 0x7f3a380a2ac0 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3ea1d640 10 monclient(hunting): authenticate will time out at 1930.835221s 2024-04-19T01:01:00.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 0x7f3a38136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f3a380a2ea0 0x7f3a380a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 0x7f3a38136f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53942/0 (socket says 172.21.15.43:53942) 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 -- 172.21.15.43:0/810939452 learned_addr learned my addr 172.21.15.43:0/810939452 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): get_auth_request con 0x7f3a380a2ac0 auth_method 0 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): get_auth_request con 0x7f3a380a2ea0 auth_method 0 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.717 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient(hunting): handle_auth_done global_id 144240 payload 274 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3cfba640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3a380a32c0 0x7f3a380a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 -- 172.21.15.43:0/810939452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3a380a32c0 msgr2=0x7f3a380a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3a380a32c0 0x7f3a380a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 -- 172.21.15.43:0/810939452 >> v2:172.21.15.43:3300/0 conn(0x7f3a380a2ea0 msgr2=0x7f3a380a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 --2- 172.21.15.43:0/810939452 >> v2:172.21.15.43:3300/0 conn(0x7f3a380a2ea0 0x7f3a380a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 monclient: found mon.noname-d 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.718 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a37fff640 1 -- 172.21.15.43:0/810939452 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a380732b0 con 0x7f3a380a2ac0 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3cfba640 10 monclient: get_auth_request con 0x7f3a380a32c0 auth_method 0 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a377fe640 1 --2- 172.21.15.43:0/810939452 >> v2:172.21.15.43:3300/0 conn(0x7f3a380a2ea0 0x7f3a380a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.705+0000 7f3a3cfba640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3a380a32c0 0x7f3a380a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a37fff640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 0x7f3a38136f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009870 tx=0x7f3a2801c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=de8086c0961f2b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 1 -- 172.21.15.43:0/810939452 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a280b0070 con 0x7f3a380a2ac0 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: dump: 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.719 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 1 -- 172.21.15.43:0/810939452 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a280aa810 con 0x7f3a380a2ac0 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 1 -- 172.21.15.43:0/810939452 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a280a9300 con 0x7f3a380a2ac0 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 5 monclient: authenticate success, global_id 144240 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 4 set_mon_vals no callback set 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:01:00.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a36ffd640 10 monclient: dump: 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.722 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 10 monclient: shutdown 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/810939452 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 msgr2=0x7f3a38136f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 0x7f3a38136f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3a28009870 tx=0x7f3a2801c420 comp rx=0 tx=0).stop 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/810939452 shutdown_connections 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3a380a32c0 0x7f3a380a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/810939452 >> v2:172.21.15.43:3300/0 conn(0x7f3a380a2ea0 0x7f3a380a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/810939452 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ac0 0x7f3a38136f40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/810939452 >> 172.21.15.43:0/810939452 conn(0x7f3a380a62f0 msgr2=0x7f3a3806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/810939452 shutdown_connections 2024-04-19T01:01:00.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/810939452 wait complete. 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 10 monclient: build_initial_monmap 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 10 monclient: monmap: 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:00.724 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 librados: starting msgr at 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 librados: starting objecter 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 Processor -- start 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 -- start start 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 librados: setting wanted keys 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 1 librados: calling monclient init 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.709+0000 7f3a3ea1d640 10 monclient: init 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _add_conns ranks=[3,6,7,0,4,1,2,8,5] 2024-04-19T01:01:00.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a380a2ac0 0x7f3a3809ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): picked mon.noname-g con 0x7f3a380a2ac0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 0x7f3a3809f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): picked mon.noname-d con 0x7f3a380a2ea0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3a380a32c0 0x7f3a381cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): picked mon.noname-i con 0x7f3a380a32c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3a38140690 con 0x7f3a380a2ac0 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f3a38140810 con 0x7f3a380a2ea0 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3a38140990 con 0x7f3a380a32c0 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient(hunting): authenticate will time out at 1930.840410s 2024-04-19T01:01:00.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 0x7f3a3809f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 0x7f3a3809f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53946/0 (socket says 172.21.15.43:53946) 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 -- 172.21.15.43:0/3911013851 learned_addr learned my addr 172.21.15.43:0/3911013851 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): get_auth_request con 0x7f3a380a2ea0 auth_method 0 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a37fff640 1 -- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a380a2ac0 msgr2=0x7f3a3809ed00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3301/0 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3cfba640 1 -- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3a380a32c0 msgr2=0x7f3a381cea20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a37fff640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a380a2ac0 0x7f3a3809ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3cfba640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3a380a32c0 0x7f3a381cea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:00.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient(hunting): handle_auth_done global_id 144249 payload 824 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 -- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3a380a32c0 msgr2=0x7f3a381cea20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3a380a32c0 0x7f3a381cea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 -- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a380a2ac0 msgr2=0x7f3a3809ed00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a380a2ac0 0x7f3a3809ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 monclient: found mon.noname-d 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 -- 172.21.15.43:0/3911013851 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a3812d690 con 0x7f3a380a2ea0 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a377fe640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 0x7f3a3809f240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000efc0 tx=0x7f3a20009f90 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 client.?.objecter ms_handle_connect 0x7f3a380a2ea0 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:00.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a20018020 con 0x7f3a380a2ea0 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: dump: 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.729 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: _finish_auth 0 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a200148f0 con 0x7f3a380a2ea0 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3a2000ce60 con 0x7f3a380a2ea0 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:01:00.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 monclient: dump: 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:00.731 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a367fc640 4 set_mon_vals no callback set 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 5 monclient: authenticate success, global_id 144249 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _renew_subs 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a3812be90 con 0x7f3a380a2ea0 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 client.144249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _renew_subs 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3809ff90 con 0x7f3a380a2ea0 2024-04-19T01:01:00.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 librados: init done 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3a2001b020 con 0x7f3a380a2ea0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a38136550 con 0x7f3a380a2ea0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3a18038e10 0x7f3a1803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(615..615 src has 501..615) ==== 6765+0+0 (secure 0 0 0) 0x7f3a2004a3d0 con 0x7f3a380a2ea0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 client.144249.objecter ms_dispatch 0x7f3a38000ba0 osd_map(615..615 src has 501..615) 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 3 client.144249.objecter handle_osd_map got epochs [615,615] > 0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 3 client.144249.objecter handle_osd_map decoding full epoch 615 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 20 client.144249.objecter dump_active .. 0 homeless 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a37fff640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3a18038e10 0x7f3a1803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a37fff640 10 monclient: get_auth_request con 0x7f3a18038e10 auth_method 0 2024-04-19T01:01:00.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a37fff640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3a18038e10 0x7f3a1803b2d0 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f3a280ad000 tx=0x7f3a28008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.713+0000 7f3a34ff9640 10 client.144249.objecter ms_handle_connect 0x7f3a18038e10 2024-04-19T01:01:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.717+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3a20022030 con 0x7f3a380a2ea0 2024-04-19T01:01:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.717+0000 7f3a34ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:00.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.717+0000 7f3a34ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a3ea1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:01:00.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a3ea1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:00.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a3ea1d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:01:00.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3a38076440 con 0x7f3a380a2ea0 2024-04-19T01:01:00.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a34ff9640 1 -- 172.21.15.43:0/3911013851 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f3a20010040 con 0x7f3a380a2ea0 2024-04-19T01:01:00.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a34ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:00.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.945+0000 7f3a34ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:00.946 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:01:00.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 10 librados: watch_flush enter 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 10 librados: watch_flush exit 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 20 client.144249.objecter shutdown clearing up homeless session... 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 10 client.144249.objecter successfully canceled tick 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3a18038e10 msgr2=0x7f3a1803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3a18038e10 0x7f3a1803b2d0 secure :-1 s=READY pgs=3611 cs=0 l=1 rev1=1 crypto rx=0x7f3a280ad000 tx=0x7f3a28008040 comp rx=0 tx=0).stop 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 10 monclient: shutdown 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 msgr2=0x7f3a3809f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 0x7f3a3809f240 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3a2000efc0 tx=0x7f3a20009f90 comp rx=0 tx=0).stop 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 shutdown_connections 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3a18038e10 0x7f3a1803b2d0 unknown :-1 s=CLOSED pgs=3611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3a380a32c0 0x7f3a381cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f3a380a2ea0 0x7f3a3809f240 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 --2- 172.21.15.43:0/3911013851 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3a380a2ac0 0x7f3a3809ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:00.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 >> 172.21.15.43:0/3911013851 conn(0x7f3a380a62f0 msgr2=0x7f3a3812fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 shutdown_connections 2024-04-19T01:01:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 -- 172.21.15.43:0/3911013851 wait complete. 2024-04-19T01:01:00.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:00.949+0000 7f3a3ea1d640 1 librados: shutdown 2024-04-19T01:01:03.967 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.101+0000 7f14a108f640 10 monclient: get_monmap_and_config 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.101+0000 7f14a108f640 10 monclient: build_initial_monmap 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.101+0000 7f14a108f640 10 monclient: monmap: 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:04.105258+0000 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:04.105258+0000 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:04.102 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:04.103 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:04.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.101+0000 7f14a108f640 1 Processor -- start 2024-04-19T01:01:04.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.101+0000 7f14a108f640 1 -- start start 2024-04-19T01:01:04.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.101+0000 7f14a108f640 10 monclient: init 2024-04-19T01:01:04.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:04.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient: _add_conns ranks=[5,7,3,10,0,2,8,1,9,6,4] 2024-04-19T01:01:04.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 0x7f149c138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): picked mon.noname-h con 0x7f149c138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f149c0a5bb0 0x7f149c139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): picked mon.noname-i con 0x7f149c0a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f149c137960 0x7f149c137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): picked mon.noname-g con 0x7f149c137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f149c0a1020 con 0x7f149c137960 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f149c1398b0 con 0x7f149c138a50 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f149c0a0780 con 0x7f149c0a5bb0 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:04.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f14a108f640 10 monclient(hunting): authenticate will time out at 1934.232388s 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149a59c640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f149c0a5bb0 msgr2=0x7f149c139370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149b59e640 1 -- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f149c137960 msgr2=0x7f149c137dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3301/0 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149a59c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f149c0a5bb0 0x7f149c139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149b59e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f149c137960 0x7f149c137dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 0x7f149c138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 0x7f149c138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:47064/0 (socket says 172.21.15.43:47064) 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 -- 172.21.15.43:0/1424993960 learned_addr learned my addr 172.21.15.43:0/1424993960 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): get_auth_request con 0x7f149c138a50 auth_method 0 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:04.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient(hunting): handle_auth_done global_id 74840 payload 274 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 -- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f149c0a5bb0 msgr2=0x7f149c139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f149c0a5bb0 0x7f149c139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 -- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f149c137960 msgr2=0x7f149c137dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f149c137960 0x7f149c137dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient: _finish_hunting 0 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 monclient: found mon.noname-h 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:01:04.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 -- 172.21.15.43:0/1424993960 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149c0a0d40 con 0x7f149c138a50 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.105+0000 7f149ad9d640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 0x7f149c138e30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f148c0098a0 tx=0x7f148c01c450 comp rx=0 tx=0).ready entity=mon.5 client_cookie=4a1f0a314f3157ca server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 1 -- 172.21.15.43:0/1424993960 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f148c0b2070 con 0x7f149c138a50 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: dump: 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:04.114 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: _finish_auth 0 2024-04-19T01:01:04.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 1 -- 172.21.15.43:0/1424993960 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148c0ab7b0 con 0x7f149c138a50 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 5 monclient: authenticate success, global_id 74840 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 4 set_mon_vals no callback set 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 1 -- 172.21.15.43:0/1424993960 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f148c0ab9e0 con 0x7f149c138a50 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f1499d9b640 10 monclient: dump: 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.116 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: shutdown 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:04.117 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 msgr2=0x7f149c138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 0x7f149c138e30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f148c0098a0 tx=0x7f148c01c450 comp rx=0 tx=0).stop 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/1424993960 shutdown_connections 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f149c137960 0x7f149c137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f149c0a5bb0 0x7f149c139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- 172.21.15.43:0/1424993960 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c138a50 0x7f149c138e30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/1424993960 >> 172.21.15.43:0/1424993960 conn(0x7f149c0a7cb0 msgr2=0x7f149c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/1424993960 shutdown_connections 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/1424993960 wait complete. 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: build_initial_monmap 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: monmap: 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:04.118 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 librados: starting msgr at 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 librados: starting objecter 2024-04-19T01:01:04.119 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 Processor -- start 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- start start 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 librados: setting wanted keys 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 librados: calling monclient init 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: init 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: _add_conns ranks=[2,5,4,6,7,3,1,0,8] 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 0x7f149c13a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): picked mon.noname-b con 0x7f149c0a5bb0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c137960 0x7f149c13a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): picked mon.noname-h con 0x7f149c137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f149c138a50 0x7f149c1d0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): picked mon.noname-c con 0x7f149c138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f149c0a2100 con 0x7f149c0a5bb0 2024-04-19T01:01:04.120 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f149c0a1f80 con 0x7f149c138a50 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f149c142810 con 0x7f149c137960 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient(hunting): authenticate will time out at 1934.238514s 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f149c138a50 0x7f149c1d0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 0x7f149c13a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f149c138a50 0x7f149c1d0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:52290/0 (socket says 172.21.15.43:52290) 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 1 -- 172.21.15.43:0/3443107853 learned_addr learned my addr 172.21.15.43:0/3443107853 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): get_auth_request con 0x7f149c138a50 auth_method 0 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:04.121 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 0x7f149c13a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:39694/0 (socket says 172.21.15.43:39694) 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): get_auth_request con 0x7f149c0a5bb0 auth_method 0 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149a59c640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c137960 0x7f149c13a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149a59c640 10 monclient(hunting): get_auth_request con 0x7f149c137960 auth_method 0 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:04.122 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient(hunting): handle_auth_done global_id 144254 payload 824 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 -- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c137960 msgr2=0x7f149c13a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c137960 0x7f149c13a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 -- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f149c138a50 msgr2=0x7f149c1d0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f149c138a50 0x7f149c1d0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient: _finish_hunting 0 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 monclient: found mon.noname-b 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 -- 172.21.15.43:0/3443107853 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149c13c940 con 0x7f149c0a5bb0 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149ad9d640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 0x7f149c13a070 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f149c0a11a0 tx=0x7f148c0191f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 client.?.objecter ms_handle_connect 0x7f149c0a5bb0 2024-04-19T01:01:04.123 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f148c0b2070 con 0x7f149c0a5bb0 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: dump: 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:04.124 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: _finish_auth 0 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: _check_auth_tickets 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148c003620 con 0x7f149c0a5bb0 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 5 monclient: authenticate success, global_id 144254 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: _renew_subs 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:01:04.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149c09ee60 con 0x7f149c0a5bb0 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f148c003c40 con 0x7f149c0a5bb0 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14837fe640 10 monclient: dump: 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:04.126 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 client.144254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: _renew_subs 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f149959a640 4 set_mon_vals no callback set 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f149c09e680 con 0x7f149c0a5bb0 2024-04-19T01:01:04.127 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.109+0000 7f14a108f640 1 librados: init done 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f148c0bf060 con 0x7f149c0a5bb0 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1488038e10 0x7f148803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(615..615 src has 501..615) ==== 6765+0+0 (secure 0 0 0) 0x7f148c0e3330 con 0x7f149c0a5bb0 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14a108f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 10 client.144254.objecter ms_dispatch 0x7f149c000ba0 osd_map(615..615 src has 501..615) 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 3 client.144254.objecter handle_osd_map got epochs [615,615] > 0 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 3 client.144254.objecter handle_osd_map decoding full epoch 615 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14a108f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14a108f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f149c13b1b0 con 0x7f149c0a5bb0 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 20 client.144254.objecter dump_active .. 0 homeless 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f149a59c640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1488038e10 0x7f148803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:04.128 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f149a59c640 10 monclient: get_auth_request con 0x7f1488038e10 auth_method 0 2024-04-19T01:01:04.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f149a59c640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1488038e10 0x7f148803b2d0 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f149c09e110 tx=0x7f1490008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:04.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.113+0000 7f14837fe640 10 client.144254.objecter ms_handle_connect 0x7f1488038e10 2024-04-19T01:01:04.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.117+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f148c016190 con 0x7f149c0a5bb0 2024-04-19T01:01:04.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.117+0000 7f14837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:04.129 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.117+0000 7f14837fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:04.342 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.341+0000 7f14a108f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:01:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.341+0000 7f14a108f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.341+0000 7f14a108f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:01:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.341+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f149c076190 con 0x7f149c0a5bb0 2024-04-19T01:01:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14837fe640 1 -- 172.21.15.43:0/3443107853 <== mon.2 v2:172.21.15.43:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f148c0d1e20 con 0x7f149c0a5bb0 2024-04-19T01:01:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:04.343 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14837fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:04.344 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:01:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 10 librados: watch_flush enter 2024-04-19T01:01:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 10 librados: watch_flush exit 2024-04-19T01:01:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 20 client.144254.objecter shutdown clearing up homeless session... 2024-04-19T01:01:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 10 client.144254.objecter successfully canceled tick 2024-04-19T01:01:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1488038e10 msgr2=0x7f148803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:04.346 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1488038e10 0x7f148803b2d0 secure :-1 s=READY pgs=3612 cs=0 l=1 rev1=1 crypto rx=0x7f149c09e110 tx=0x7f1490008040 comp rx=0 tx=0).stop 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 10 monclient: shutdown 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 msgr2=0x7f149c13a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 0x7f149c13a070 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f149c0a11a0 tx=0x7f148c0191f0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 shutdown_connections 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1488038e10 0x7f148803b2d0 unknown :-1 s=CLOSED pgs=3612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f149c138a50 0x7f149c1d0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f149c137960 0x7f149c13a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 --2- 172.21.15.43:0/3443107853 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f149c0a5bb0 0x7f149c13a070 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 >> 172.21.15.43:0/3443107853 conn(0x7f149c0a7cb0 msgr2=0x7f149c0a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 shutdown_connections 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 -- 172.21.15.43:0/3443107853 wait complete. 2024-04-19T01:01:04.347 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:04.345+0000 7f14a108f640 1 librados: shutdown 2024-04-19T01:01:07.364 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:01:07.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient: get_monmap_and_config 2024-04-19T01:01:07.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient: build_initial_monmap 2024-04-19T01:01:07.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient: monmap: 2024-04-19T01:01:07.497 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:07.500599+0000 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:07.500599+0000 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:07.498 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:07.499 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:07.499 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:07.499 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:07.499 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 Processor -- start 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 -- start start 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient: init 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient: _add_conns ranks=[10,5,1,3,9,4,2,6,0,7,8] 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdcbc138a50 0x7fdcbc138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): picked mon.noname-f con 0x7fdcbc138a50 addr v2:172.21.15.161:3300/0 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 0x7fdcbc139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): picked mon.noname-h con 0x7fdcbc0a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:07.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdcbc137960 addr v1:172.21.15.161:6789/0 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcbc0a0d40 con 0x7fdcbc137960 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdcbc0a1020 con 0x7fdcbc138a50 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdcbc1398b0 con 0x7fdcbc0a5bb0 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.497+0000 7fdcc0a5d640 10 monclient(hunting): authenticate will time out at 1937.627690s 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 0x7fdcbc139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdcbc138a50 0x7fdcbc138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 0x7fdcbc139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:47076/0 (socket says 172.21.15.43:47076) 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/3669571580 learned_addr learned my addr 172.21.15.43:0/3669571580 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): get_auth_request con 0x7fdcbc0a5bb0 auth_method 0 2024-04-19T01:01:07.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): get_auth_request con 0x7fdcbc138a50 auth_method 0 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_done global_id 74858 payload 274 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/3669571580 >> v2:172.21.15.161:3300/0 conn(0x7fdcbc138a50 msgr2=0x7fdcbc138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:07.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 --2- 172.21.15.43:0/3669571580 >> v2:172.21.15.161:3300/0 conn(0x7fdcbc138a50 0x7fdcbc138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/3669571580 >> v1:172.21.15.161:6789/0 conn(0x7fdcbc137960 legacy=0x7fdcbc137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 monclient: found mon.noname-h 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/3669571580 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc0a0a60 con 0x7fdcbc0a5bb0 2024-04-19T01:01:07.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcba59c640 1 --2- 172.21.15.43:0/3669571580 >> v2:172.21.15.161:3300/0 conn(0x7fdcbc138a50 0x7fdcbc138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb9d9b640 1 --2- 172.21.15.43:0/3669571580 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 0x7fdcbc139370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009870 tx=0x7fdcb001c420 comp rx=0 tx=0).ready entity=mon.5 client_cookie=6b9da30ac601372a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 1 -- 172.21.15.43:0/3669571580 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb00b0070 con 0x7fdcbc0a5bb0 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: dump: 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:07.510 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:07.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: _finish_auth 0 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: _check_auth_tickets 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 5 monclient: authenticate success, global_id 74858 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 1 -- 172.21.15.43:0/3669571580 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb00aa810 con 0x7fdcbc0a5bb0 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 1 -- 172.21.15.43:0/3669571580 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb00a9300 con 0x7fdcbc0a5bb0 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 4 set_mon_vals no callback set 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcb959a640 10 monclient: dump: 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.512 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 10 monclient: shutdown 2024-04-19T01:01:07.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/3669571580 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 msgr2=0x7fdcbc139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/3669571580 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 0x7fdcbc139370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0009870 tx=0x7fdcb001c420 comp rx=0 tx=0).stop 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/3669571580 shutdown_connections 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/3669571580 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc0a5bb0 0x7fdcbc139370 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/3669571580 >> v2:172.21.15.161:3300/0 conn(0x7fdcbc138a50 0x7fdcbc138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.501+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/3669571580 >> 172.21.15.43:0/3669571580 conn(0x7fdcbc0a7cb0 msgr2=0x7fdcbc0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/3669571580 shutdown_connections 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/3669571580 wait complete. 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: build_initial_monmap 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: monmap: 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:07.514 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 librados: starting msgr at 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 librados: starting objecter 2024-04-19T01:01:07.515 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 Processor -- start 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- start start 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 librados: setting wanted keys 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 librados: calling monclient init 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: init 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: _add_conns ranks=[1,5,7,3,8,2,6,0,4] 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc0a5bb0 0x7fdcbc1369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): picked mon.noname-f con 0x7fdcbc0a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 0x7fdcbc12dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): picked mon.noname-h con 0x7fdcbc137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcbc138a50 0x7fdcbc12e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): picked mon.noname-i con 0x7fdcbc138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:07.516 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdcbc143bb0 con 0x7fdcbc0a5bb0 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdcbc143a30 con 0x7fdcbc137960 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdcbc143d30 con 0x7fdcbc138a50 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient(hunting): authenticate will time out at 1937.633483s 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcbad9d640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcbc138a50 msgr2=0x7fdcbc12e010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcbad9d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcbc138a50 0x7fdcbc12e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 0x7fdcbc12dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc0a5bb0 0x7fdcbc1369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 0x7fdcbc12dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:47088/0 (socket says 172.21.15.43:47088) 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/649333150 learned_addr learned my addr 172.21.15.43:0/649333150 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): get_auth_request con 0x7fdcbc137960 auth_method 0 2024-04-19T01:01:07.517 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): get_auth_request con 0x7fdcbc0a5bb0 auth_method 0 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient(hunting): handle_auth_done global_id 74867 payload 824 2024-04-19T01:01:07.518 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcbc138a50 msgr2=0x7fdcbc12e010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcbc138a50 0x7fdcbc12e010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc0a5bb0 msgr2=0x7fdcbc1369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc0a5bb0 0x7fdcbc1369c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 monclient: found mon.noname-h 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 -- 172.21.15.43:0/649333150 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcbc12ef20 con 0x7fdcbc137960 2024-04-19T01:01:07.519 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcba59c640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc0a5bb0 0x7fdcbc1369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb9d9b640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 0x7fdcbc12dad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00ae040 tx=0x7fdcb00191c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 client.?.objecter ms_handle_connect 0x7fdcbc137960 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb00b0040 con 0x7fdcbc137960 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: dump: 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:07.520 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: _finish_auth 0 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: _check_auth_tickets 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:07.521 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb00b9070 con 0x7fdcbc137960 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 5 monclient: authenticate success, global_id 74867 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: _renew_subs 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcbc09fc90 con 0x7fdcbc137960 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdcb00aacb0 con 0x7fdcbc137960 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 client.74867.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcaaffd640 10 monclient: dump: 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:07.522 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: _renew_subs 2024-04-19T01:01:07.523 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcbc09e8c0 con 0x7fdcbc137960 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcc0a5d640 1 librados: init done 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.505+0000 7fdcb8d99640 4 set_mon_vals no callback set 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcc0a5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcc0a5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcc0a5d640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcbc0008e0 con 0x7fdcbc137960 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdcb0004040 con 0x7fdcbc137960 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc9c038e10 0x7fdc9c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(615..615 src has 501..615) ==== 6765+0+0 (secure 0 0 0) 0x7fdcb00e3170 con 0x7fdcbc137960 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 10 client.74867.objecter ms_dispatch 0x7fdcbc000ba0 osd_map(615..615 src has 501..615) 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 3 client.74867.objecter handle_osd_map got epochs [615,615] > 0 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 3 client.74867.objecter handle_osd_map decoding full epoch 615 2024-04-19T01:01:07.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 20 client.74867.objecter dump_active .. 0 homeless 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcba59c640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc9c038e10 0x7fdc9c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcba59c640 10 monclient: get_auth_request con 0x7fdc9c038e10 auth_method 0 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcba59c640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc9c038e10 0x7fdc9c03b2d0 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009580 tx=0x7fdcac009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.509+0000 7fdcaaffd640 10 client.74867.objecter ms_handle_connect 0x7fdc9c038e10 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.521+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdcb0016130 con 0x7fdcbc137960 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.521+0000 7fdcaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:07.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.521+0000 7fdcaaffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:07.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:01:07.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:07.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:01:07.749 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdcbc09dff0 con 0x7fdcbc137960 2024-04-19T01:01:07.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcaaffd640 1 -- 172.21.15.43:0/649333150 <== mon.5 v2:172.21.15.161:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3194 (secure 0 0 0) 0x7fdcb00aa7e0 con 0x7fdcbc137960 2024-04-19T01:01:07.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:07.750 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcaaffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:07.750 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":110,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 10 librados: watch_flush enter 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 10 librados: watch_flush exit 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 20 client.74867.objecter shutdown clearing up homeless session... 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.749+0000 7fdcc0a5d640 10 client.74867.objecter successfully canceled tick 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc9c038e10 msgr2=0x7fdc9c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc9c038e10 0x7fdc9c03b2d0 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fdcac009580 tx=0x7fdcac009190 comp rx=0 tx=0).stop 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 10 monclient: shutdown 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:07.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 msgr2=0x7fdcbc12dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 0x7fdcbc12dad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdcb00ae040 tx=0x7fdcb00191c0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 shutdown_connections 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc9c038e10 0x7fdc9c03b2d0 unknown :-1 s=CLOSED pgs=3613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdcbc138a50 0x7fdcbc12e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdcbc137960 0x7fdcbc12dad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 --2- 172.21.15.43:0/649333150 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdcbc0a5bb0 0x7fdcbc1369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 >> 172.21.15.43:0/649333150 conn(0x7fdcbc0a7cb0 msgr2=0x7fdcbc06d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 shutdown_connections 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 -- 172.21.15.43:0/649333150 wait complete. 2024-04-19T01:01:07.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:07.753+0000 7fdcc0a5d640 1 librados: shutdown 2024-04-19T01:01:10.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient: get_monmap_and_config 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient: build_initial_monmap 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient: monmap: 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:10.904083+0000 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:10.904083+0000 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:10.901 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:10.902 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 1 Processor -- start 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 1 -- start start 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient: init 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient: _add_conns ranks=[1,0,6,2,9,4,8,3,5,10,7] 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f83c4138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83c40a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:10.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient(hunting): picked mon.noname-d con 0x7f83c41397c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c40a3dc0 con 0x7f83c40a5b80 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83c40a40a0 con 0x7f83c4138cd0 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f83c40a4410 con 0x7f83c41397c0 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83ca720640 10 monclient(hunting): authenticate will time out at 1941.031405s 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c37fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f83c40a5b80 0x7f83c413e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45302/0 (socket says 172.21.15.43:45302) 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 learned_addr learned my addr 172.21.15.43:0/1209129908 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83c41397c0 auth_method 0 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:10.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:10.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:10.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:10.901+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83c41397c0 auth_method 0 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:11.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:11.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.105+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:01:11.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83c41397c0 auth_method 0 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:11.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:11.509+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83c41397c0 auth_method 0 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:12.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:12.313+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): tick 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 monclient(hunting): continuing hunt 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 msgr2=0x7f83c41419a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83c41397c0 0x7f83c41419a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 >> v1:172.21.15.161:6789/0 conn(0x7f83c4138cd0 legacy=0x7f83c41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 >> v1:172.21.15.43:6789/0 conn(0x7f83c40a5b80 legacy=0x7f83c413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient: _add_conns ranks=[7,10,4,2,3,8,5,9] 2024-04-19T01:01:13.904 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 0x7f83b0005c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): picked mon.noname-i con 0x7f83b0003830 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): picked mon.noname-f con 0x7f83b00062e0 addr v2:172.21.15.161:3300/0 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f83b0008e20 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f83c40a4410 con 0x7f83b00062e0 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f83b000ba20 con 0x7f83b0008e20 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.901+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f83b000bc20 con 0x7f83b0003830 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c1ffb640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:13.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 msgr2=0x7f83b0005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 0x7f83b0005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 10 monclient(hunting): get_auth_request con 0x7f83b00062e0 auth_method 0 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83b0008e20 auth_method 0 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:13.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:01:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:13.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:13.905+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:14.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 msgr2=0x7f83b0005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:14.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 0x7f83b0005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83b0008e20 auth_method 0 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 10 monclient(hunting): get_auth_request con 0x7f83b00062e0 auth_method 0 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:14.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:01:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:14.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.109+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:01:14.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 msgr2=0x7f83b0005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:14.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 0x7f83b0005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:01:14.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:14.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83b0008e20 auth_method 0 2024-04-19T01:01:14.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:14.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 10 monclient(hunting): get_auth_request con 0x7f83b00062e0 auth_method 0 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:14.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:14.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:14.513+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 msgr2=0x7f83b0005c40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 0x7f83b0005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83b0008e20 auth_method 0 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:15.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 10 monclient(hunting): get_auth_request con 0x7f83b00062e0 auth_method 0 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:15.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:15.317+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): tick 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 monclient(hunting): continuing hunt 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 msgr2=0x7f83b0005c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f83b0003830 0x7f83b0005c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.905 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 msgr2=0x7f83b000b2e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f83b0008e20 0x7f83b000b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> v2:172.21.15.161:3300/0 conn(0x7f83b00062e0 0x7f83b0008770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 reap_dead start 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient: _add_conns ranks=[8,2,5,9,3] 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 1 -- 172.21.15.43:0/1209129908 reap_dead start 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83b0003c30 0x7f83b00092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): picked mon.noname-e con 0x7f83b0003c30 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83b0009810 0x7f83b0008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.906 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f83b0009810 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83b0009bf0 0x7f83b000a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): picked mon.noname-h con 0x7f83b0009bf0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f83c40a4410 con 0x7f83b0009810 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f83b000ba20 con 0x7f83b0009bf0 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:16.907 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 1 -- 172.21.15.43:0/1209129908 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f83b000bc20 con 0x7f83b0003c30 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c1ffb640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83b0003c30 0x7f83b00092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): get_auth_request con 0x7f83b0003c30 auth_method 0 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83b0009810 0x7f83b0008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83b0009bf0 0x7f83b000a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): get_auth_request con 0x7f83b0009810 auth_method 0 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:16.908 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83b0009bf0 auth_method 0 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:16.909 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient(hunting): handle_auth_done global_id 144242 payload 274 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83b0009bf0 msgr2=0x7f83b000a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83b0009bf0 0x7f83b000a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83b0009810 msgr2=0x7f83b0008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83b0009810 0x7f83b0008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient: _finish_hunting 0 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 monclient: found mon.noname-e 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:01:16.910 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.905+0000 7f83c3fff640 1 -- 172.21.15.43:0/1209129908 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c40a2ae0 con 0x7f83b0003c30 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c3fff640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83b0003c30 0x7f83b00092d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f83b4001a70 tx=0x7f83b401c780 comp rx=0 tx=0).ready entity=mon.8 client_cookie=923841710dc21f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 1 -- 172.21.15.43:0/1209129908 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83b40074e0 con 0x7f83b0003c30 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: dump: 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:16.912 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:16.913 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 1 -- 172.21.15.43:0/1209129908 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83b40057d0 con 0x7f83b0003c30 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 1 -- 172.21.15.43:0/1209129908 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83b4005df0 con 0x7f83b0003c30 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83c2ffd640 10 monclient: dump: 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.914 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83ca720640 5 monclient: authenticate success, global_id 144242 2024-04-19T01:01:16.915 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83ca720640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.909+0000 7f83ca720640 4 set_mon_vals no callback set 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 10 monclient: shutdown 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 -- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83b0003c30 msgr2=0x7f83b00092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83b0003c30 0x7f83b00092d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f83b4001a70 tx=0x7f83b401c780 comp rx=0 tx=0).stop 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 -- 172.21.15.43:0/1209129908 shutdown_connections 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f83b0009bf0 0x7f83b000a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f83b0009810 0x7f83b0008770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 --2- 172.21.15.43:0/1209129908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f83b0003c30 0x7f83b00092d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 -- 172.21.15.43:0/1209129908 >> 172.21.15.43:0/1209129908 conn(0x7f83c40ab2f0 msgr2=0x7f83c40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 -- 172.21.15.43:0/1209129908 shutdown_connections 2024-04-19T01:01:16.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 -- 172.21.15.43:0/1209129908 wait complete. 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 10 monclient: build_initial_monmap 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 10 monclient: monmap: 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:16.922 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 librados: starting msgr at 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 librados: starting objecter 2024-04-19T01:01:16.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 Processor -- start 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 -- start start 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 librados: setting wanted keys 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 1 librados: calling monclient init 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.913+0000 7f83ca720640 10 monclient: init 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _add_conns ranks=[1,6,0,4,5,3,2,7,8] 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83b0003c30 0x7f83c41d0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): picked mon.noname-f con 0x7f83b0003c30 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 0x7f83c4134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): picked mon.noname-d con 0x7f83b0009810 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b0009bf0 0x7f83c4134e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): picked mon.noname-a con 0x7f83b0009bf0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:01:16.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f83c41360c0 con 0x7f83b0009bf0 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f83c40a20a0 con 0x7f83b0003c30 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f83c4136240 con 0x7f83b0009810 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient(hunting): authenticate will time out at 1947.044732s 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 0x7f83c4134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 0x7f83c4134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59138/0 (socket says 172.21.15.43:59138) 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 1 -- 172.21.15.43:0/2789257002 learned_addr learned my addr 172.21.15.43:0/2789257002 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): get_auth_request con 0x7f83b0009810 auth_method 0 2024-04-19T01:01:16.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b0009bf0 0x7f83c4134e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request con 0x7f83b0009bf0 auth_method 0 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:16.926 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83b0003c30 0x7f83c41d0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): get_auth_request con 0x7f83b0003c30 auth_method 0 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient(hunting): handle_auth_done global_id 65511 payload 824 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 -- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 msgr2=0x7f83c4134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 0x7f83c4134950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.927 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 -- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83b0003c30 msgr2=0x7f83c41d0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83b0003c30 0x7f83c41d0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient: _finish_hunting 0 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 monclient: found mon.noname-a 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 -- 172.21.15.43:0/2789257002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83c40a3040 con 0x7f83b0009bf0 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c37fe640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 0x7f83c4134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:16.928 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c8cbd640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b0009bf0 0x7f83c4134e90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f83c40a4590 tx=0x7f83bc009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 client.?.objecter ms_handle_connect 0x7f83b0009bf0 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83bc018bc0 con 0x7f83b0009bf0 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: dump: 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:16.929 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: _finish_auth 0 2024-04-19T01:01:16.930 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc018da0 con 0x7f83b0009bf0 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 5 monclient: authenticate success, global_id 65511 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _renew_subs 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83c40a3590 con 0x7f83b0009bf0 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f83bc021ca0 con 0x7f83b0009bf0 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 monclient: dump: 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:16.931 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:16.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c27fc640 4 set_mon_vals no callback set 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 client.65511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _renew_subs 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83c4121770 con 0x7f83b0009bf0 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 librados: init done 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f83bc015040 con 0x7f83b0009bf0 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83c40693b0 con 0x7f83b0009bf0 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f83b003b230 0x7f83b003d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(617..617 src has 501..617) ==== 6384+0+0 (secure 0 0 0) 0x7f83bc01e070 con 0x7f83b0009bf0 2024-04-19T01:01:16.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 10 client.65511.objecter ms_dispatch 0x7f83c4000ba0 osd_map(617..617 src has 501..617) 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 3 client.65511.objecter handle_osd_map got epochs [617,617] > 0 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 3 client.65511.objecter handle_osd_map decoding full epoch 617 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c0ff9640 20 client.65511.objecter dump_active .. 0 homeless 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f83b003b230 0x7f83b003d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.917+0000 7f83c3fff640 10 monclient: get_auth_request con 0x7f83b003b230 auth_method 0 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.921+0000 7f83c0ff9640 10 client.65511.objecter ms_handle_connect 0x7f83b003b230 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.921+0000 7f83c3fff640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f83b003b230 0x7f83b003d6f0 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7f83b40a9000 tx=0x7f83b4007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.925+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f83c40a3040 con 0x7f83b0009bf0 2024-04-19T01:01:16.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.925+0000 7f83c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:16.935 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:16.925+0000 7f83c0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:17.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83ca720640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:01:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83ca720640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83ca720640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f83c41353d0 con 0x7f83b0009bf0 2024-04-19T01:01:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83c0ff9640 1 -- 172.21.15.43:0/2789257002 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3181 (secure 0 0 0) 0x7f83bc022500 con 0x7f83b0009bf0 2024-04-19T01:01:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:01:17.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.153+0000 7f83c0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:17.155 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":112,"quorum":[0,1,2,4,5,6,8],"quorum_names":["a","f","b","c","h","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:01:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 10 librados: watch_flush enter 2024-04-19T01:01:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 10 librados: watch_flush exit 2024-04-19T01:01:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 20 client.65511.objecter shutdown clearing up homeless session... 2024-04-19T01:01:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 10 client.65511.objecter successfully canceled tick 2024-04-19T01:01:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f83b003b230 msgr2=0x7f83b003d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f83b003b230 0x7f83b003d6f0 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7f83b40a9000 tx=0x7f83b4007040 comp rx=0 tx=0).stop 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 10 monclient: shutdown 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b0009bf0 msgr2=0x7f83c4134e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b0009bf0 0x7f83c4134e90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f83c40a4590 tx=0x7f83bc009f80 comp rx=0 tx=0).stop 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 shutdown_connections 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f83b003b230 0x7f83b003d6f0 unknown :-1 s=CLOSED pgs=3614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f83b0009bf0 0x7f83c4134e90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f83b0009810 0x7f83c4134950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 --2- 172.21.15.43:0/2789257002 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83b0003c30 0x7f83c41d0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 >> 172.21.15.43:0/2789257002 conn(0x7f83c40ab2f0 msgr2=0x7f83c40a95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 shutdown_connections 2024-04-19T01:01:17.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 -- 172.21.15.43:0/2789257002 wait complete. 2024-04-19T01:01:17.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.157+0000 7f83ca720640 1 librados: shutdown 2024-04-19T01:01:17.174 INFO:tasks.mon_thrash.ceph_manager:quorum is size 7 2024-04-19T01:01:17.174 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon a 2024-04-19T01:01:17.174 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:01:17.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient: get_monmap_and_config 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient: monmap: 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:17.295036+0000 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:17.295036+0000 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:17.292 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:17.293 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 Processor -- start 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 -- start start 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient: init 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient: _add_conns ranks=[4,3,9,7,1,8,10,6,2,5,0] 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 0x7fd6081390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): picked mon.noname-c con 0x7fd608138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6080a5b80 0x7fd6081395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): picked mon.noname-g con 0x7fd6080a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:17.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd6081358b0 0x7fd608135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): picked mon.noname-a con 0x7fd6081358b0 addr v2:172.21.15.43:3300/0 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fd6080a4410 con 0x7fd6081358b0 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd608139b30 con 0x7fd6080a5b80 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd6080a3dc0 con 0x7fd608138cd0 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd60da1f640 10 monclient(hunting): authenticate will time out at 1947.422109s 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 0x7fd6081390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6067fc640 1 -- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6080a5b80 msgr2=0x7fd6081395f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3301/0 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 0x7fd6081390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47038/0 (socket says 172.21.15.43:47038) 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 -- 172.21.15.43:0/3665922479 learned_addr learned my addr 172.21.15.43:0/3665922479 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): get_auth_request con 0x7fd608138cd0 auth_method 0 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6067fc640 1 --2- 172.21.15.43:0/3665922479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6080a5b80 0x7fd6081395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 1 --2- 172.21.15.43:0/3665922479 >> v2:172.21.15.43:3300/0 conn(0x7fd6081358b0 0x7fd608135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): get_auth_request con 0x7fd6081358b0 auth_method 0 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd6077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient(hunting): handle_auth_done global_id 134428 payload 274 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 -- 172.21.15.43:0/3665922479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6080a5b80 msgr2=0x7fd6081395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3665922479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6080a5b80 0x7fd6081395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 -- 172.21.15.43:0/3665922479 >> v2:172.21.15.43:3300/0 conn(0x7fd6081358b0 msgr2=0x7fd608135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3665922479 >> v2:172.21.15.43:3300/0 conn(0x7fd6081358b0 0x7fd608135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 monclient: found mon.noname-c 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:01:17.298 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.293+0000 7fd606ffd640 1 -- 172.21.15.43:0/3665922479 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080a40a0 con 0x7fd608138cd0 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3665922479 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 0x7fd6081390b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009870 tx=0x7fd5f801c3e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=6d5cdf6da32f5693 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 1 -- 172.21.15.43:0/3665922479 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5f80b2070 con 0x7fd608138cd0 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: dump: 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.305 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 1 -- 172.21.15.43:0/3665922479 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5f80ab7f0 con 0x7fd608138cd0 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 5 monclient: authenticate success, global_id 134428 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 4 set_mon_vals no callback set 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 1 -- 172.21.15.43:0/3665922479 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5f80abe50 con 0x7fd608138cd0 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd605ffb640 10 monclient: dump: 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.307 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: shutdown 2024-04-19T01:01:17.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- 172.21.15.43:0/3665922479 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 msgr2=0x7fd6081390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3665922479 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 0x7fd6081390b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd5f8009870 tx=0x7fd5f801c3e0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- 172.21.15.43:0/3665922479 shutdown_connections 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3665922479 >> v2:172.21.15.43:3300/0 conn(0x7fd6081358b0 0x7fd608135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3665922479 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd6080a5b80 0x7fd6081395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3665922479 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd608138cd0 0x7fd6081390b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- 172.21.15.43:0/3665922479 >> 172.21.15.43:0/3665922479 conn(0x7fd6080ab2f0 msgr2=0x7fd6080ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- 172.21.15.43:0/3665922479 shutdown_connections 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- 172.21.15.43:0/3665922479 wait complete. 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: monmap: 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.309 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.310 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 librados: starting msgr at 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 librados: starting objecter 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 Processor -- start 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- start start 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 librados: setting wanted keys 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 librados: calling monclient init 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: init 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient: _add_conns ranks=[5,0,8,4,7,2,1,3,6] 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6080a5b80 0x7fd6081d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): picked mon.noname-h con 0x7fd6080a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 0x7fd60813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): picked mon.noname-a con 0x7fd6081358b0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd608138cd0 0x7fd60813f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): picked mon.noname-e con 0x7fd608138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd6080a2010 con 0x7fd6081358b0 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fd6080a1e90 con 0x7fd6080a5b80 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd608142aa0 con 0x7fd608138cd0 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd60da1f640 10 monclient(hunting): authenticate will time out at 1947.427789s 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 0x7fd60813ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 0x7fd60813ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46584/0 (socket says 172.21.15.43:46584) 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 1 -- 172.21.15.43:0/3096377291 learned_addr learned my addr 172.21.15.43:0/3096377291 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 10 monclient(hunting): get_auth_request con 0x7fd6081358b0 auth_method 0 2024-04-19T01:01:17.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.297+0000 7fd6067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd608138cd0 0x7fd60813f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): get_auth_request con 0x7fd608138cd0 auth_method 0 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6080a5b80 0x7fd6081d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): get_auth_request con 0x7fd6080a5b80 auth_method 0 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 10 monclient(hunting): handle_auth_done global_id 65529 payload 824 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd608138cd0 msgr2=0x7fd60813f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd608138cd0 0x7fd60813f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6080a5b80 msgr2=0x7fd6081d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fd6080a5b80 0x7fd6081d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 monclient: found mon.noname-a 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6080a3040 con 0x7fd6081358b0 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6067fc640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 0x7fd60813ecf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b550 tx=0x7fd5fc00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 client.?.objecter ms_handle_connect 0x7fd6081358b0 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5fc013020 con 0x7fd6081358b0 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: dump: 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.315 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc003bc0 con 0x7fd6081358b0 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 5 monclient: authenticate success, global_id 65529 2024-04-19T01:01:17.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 monclient: _renew_subs 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6081350a0 con 0x7fd6081358b0 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5fc003d70 con 0x7fd6081358b0 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: dump: 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.317 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd604ff9640 4 set_mon_vals no callback set 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 client.65529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 monclient: _renew_subs 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60813a010 con 0x7fd6081358b0 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 1 librados: init done 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 0x7fd608133960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd60806d140 con 0x7fd608133580 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd5fc016020 con 0x7fd6081358b0 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 0x7fd608133960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient: get_auth_request con 0x7fd608133580 auth_method 0 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd5f4038e10 0x7fd5f403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(617..617 src has 501..617) ==== 6384+0+0 (secure 0 0 0) 0x7fd5fc049000 con 0x7fd6081358b0 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 client.65529.objecter ms_dispatch 0x7fd608000ba0 osd_map(617..617 src has 501..617) 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 3 client.65529.objecter handle_osd_map got epochs [617,617] > 0 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 3 client.65529.objecter handle_osd_map decoding full epoch 617 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 10 monclient(hunting): handle_auth_done global_id 65538 payload 824 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6080a3590 con 0x7fd608133580 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 20 client.65529.objecter dump_active .. 0 homeless 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 client.65529.objecter ms_handle_connect 0x7fd608133580 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 0x7fd608133960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd6080a4380 tx=0x7fd5f80191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 client.65529.objecter resend_mon_ops 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd5f4038e10 0x7fd5f403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 10 monclient: get_auth_request con 0x7fd5f4038e10 auth_method 0 2024-04-19T01:01:17.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5f80bb030 con 0x7fd608133580 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: dump: 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.321 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd5f80b3040 con 0x7fd608133580 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 msgr2=0x7fd608133960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 0x7fd608133960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd6080a4380 tx=0x7fd5f80191e0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 msgr2=0x7fd608133960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133580 0x7fd608133960 secure :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd6080a4380 tx=0x7fd5f80191e0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd5effff640 10 client.65529.objecter ms_handle_connect 0x7fd5f4038e10 2024-04-19T01:01:17.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.301+0000 7fd6077fe640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd5f4038e10 0x7fd5f403b2d0 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7fd60813afb0 tx=0x7fd60000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 0x7fd608137990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd608068240 con 0x7fd608133ea0 2024-04-19T01:01:17.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 0x7fd608137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient: get_auth_request con 0x7fd608133ea0 auth_method 0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 10 monclient(hunting): handle_auth_done global_id 65547 payload 824 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 1 -- 172.21.15.43:0/3096377291 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6080693b0 con 0x7fd608133ea0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 client.65529.objecter ms_handle_connect 0x7fd608133ea0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 client.65529.objecter resend_mon_ops 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd606ffd640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 0x7fd608137990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd5f801cc80 tx=0x7fd5f80a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd5f80bb070 con 0x7fd608133ea0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.325 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 monclient: dump: 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.326 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9422 (secure 0 0 0) 0x7fd5f80ca480 con 0x7fd608133ea0 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 msgr2=0x7fd608137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 0x7fd608137990 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd5f801cc80 tx=0x7fd5f80a9000 comp rx=0 tx=0).stop 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 msgr2=0x7fd608137990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd5effff640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd608133ea0 0x7fd608137990 secure :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd5f801cc80 tx=0x7fd5f80a9000 comp rx=0 tx=0).stop 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 603581, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:17.328 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.329 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.330 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:17.331 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.332 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:17.333 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.334 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.335 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.336 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.337 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:17.338 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.339 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:17.340 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.341 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.342 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.343 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.344 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.345 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:17.346 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.347 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.348 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.349 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:01:17.350 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.350 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.350 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.350 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:17.350 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 10 librados: watch_flush enter 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 10 librados: watch_flush exit 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 20 client.65529.objecter shutdown clearing up homeless session... 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 10 client.65529.objecter successfully canceled tick 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd5f4038e10 msgr2=0x7fd5f403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd5f4038e10 0x7fd5f403b2d0 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7fd60813afb0 tx=0x7fd60000b040 comp rx=0 tx=0).stop 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 10 monclient: shutdown 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 msgr2=0x7fd60813ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 0x7fd60813ecf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00b550 tx=0x7fd5fc00ba10 comp rx=0 tx=0).stop 2024-04-19T01:01:17.352 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd6077fe640 1 -- 172.21.15.43:0/3096377291 reap_dead start 2024-04-19T01:01:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 shutdown_connections 2024-04-19T01:01:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 --2- 172.21.15.43:0/3096377291 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd6081358b0 0x7fd60813ecf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 >> 172.21.15.43:0/3096377291 conn(0x7fd6080ab2f0 msgr2=0x7fd6080a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 shutdown_connections 2024-04-19T01:01:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 -- 172.21.15.43:0/3096377291 wait complete. 2024-04-19T01:01:17.353 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.309+0000 7fd60da1f640 1 librados: shutdown 2024-04-19T01:01:17.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:17.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon b 2024-04-19T01:01:17.353 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:01:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient: get_monmap_and_config 2024-04-19T01:01:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient: monmap: 2024-04-19T01:01:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.479 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:17.482620+0000 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:17.482620+0000 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:17.480 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:17.481 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 Processor -- start 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 -- start start 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient: init 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient: _add_conns ranks=[8,4,10,1,5,2,9,7,3,6,0] 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 0x7fe8e40a2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): picked mon.noname-e con 0x7fe8e40a2a80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:17.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe8e4136ed0 0x7fe8e40a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): picked mon.noname-c con 0x7fe8e4136ed0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe8e41372f0 0x7fe8e40a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): picked mon.noname-f con 0x7fe8e41372f0 addr v2:172.21.15.161:3300/0 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e4072fd0 con 0x7fe8e41372f0 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe8e40732b0 con 0x7fe8e4136ed0 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe8e4073590 con 0x7fe8e40a2a80 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e9d5c640 10 monclient(hunting): authenticate will time out at 1947.609786s 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 0x7fe8e40a2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe8e4136ed0 0x7fe8e40a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 0x7fe8e40a2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:46808/0 (socket says 172.21.15.43:46808) 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 -- 172.21.15.43:0/3076994130 learned_addr learned my addr 172.21.15.43:0/3076994130 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): get_auth_request con 0x7fe8e40a2a80 auth_method 0 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): get_auth_request con 0x7fe8e4136ed0 auth_method 0 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_done global_id 144260 payload 274 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 -- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe8e4136ed0 msgr2=0x7fe8e40a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 --2- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe8e4136ed0 0x7fe8e40a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 -- 172.21.15.43:0/3076994130 >> v2:172.21.15.161:3300/0 conn(0x7fe8e41372f0 msgr2=0x7fe8e40a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e3fff640 1 --2- 172.21.15.43:0/3076994130 >> v2:172.21.15.161:3300/0 conn(0x7fe8e41372f0 0x7fe8e40a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 --2- 172.21.15.43:0/3076994130 >> v2:172.21.15.161:3300/0 conn(0x7fe8e41372f0 0x7fe8e40a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 monclient: found mon.noname-e 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e37fe640 1 -- 172.21.15.43:0/3076994130 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e406c5b0 con 0x7fe8e40a2a80 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e3fff640 10 monclient: get_auth_request con 0x7fe8e41372f0 auth_method 0 2024-04-19T01:01:17.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.481+0000 7fe8e3fff640 1 --2- 172.21.15.43:0/3076994130 >> v2:172.21.15.161:3300/0 conn(0x7fe8e41372f0 0x7fe8e40a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:01:17.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 --2- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 0x7fe8e40a2e60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc009870 tx=0x7fe8dc01c4b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=1d59698fbcdd9362 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 1 -- 172.21.15.43:0/3076994130 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8dc0b0070 con 0x7fe8e40a2a80 2024-04-19T01:01:17.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:01:17.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: dump: 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.494 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 1 -- 172.21.15.43:0/3076994130 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8dc0aa810 con 0x7fe8e40a2a80 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 5 monclient: authenticate success, global_id 144260 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 4 set_mon_vals no callback set 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 1 -- 172.21.15.43:0/3076994130 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8dc0a9300 con 0x7fe8e40a2a80 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:17.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e27fc640 10 monclient: dump: 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: shutdown 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 msgr2=0x7fe8e40a2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 0x7fe8e40a2e60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc009870 tx=0x7fe8dc01c4b0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/3076994130 shutdown_connections 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/3076994130 >> v2:172.21.15.161:3300/0 conn(0x7fe8e41372f0 0x7fe8e40a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe8e4136ed0 0x7fe8e40a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/3076994130 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe8e40a2a80 0x7fe8e40a2e60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/3076994130 >> 172.21.15.43:0/3076994130 conn(0x7fe8e40a6410 msgr2=0x7fe8e406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/3076994130 shutdown_connections 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/3076994130 wait complete. 2024-04-19T01:01:17.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: monmap: 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.498 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 librados: starting msgr at 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 librados: starting objecter 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 Processor -- start 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- start start 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 librados: setting wanted keys 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 librados: calling monclient init 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: init 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient: _add_conns ranks=[0,3,5,6,1,4,2,7,8] 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e40a2a80 0x7fe8e41354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): picked mon.noname-a con 0x7fe8e40a2a80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:01:17.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8e4136ed0 0x7fe8e41359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): picked mon.noname-g con 0x7fe8e4136ed0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe8e41372f0 0x7fe8e406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): picked mon.noname-h con 0x7fe8e41372f0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e4140620 con 0x7fe8e40a2a80 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e40a5930 con 0x7fe8e4136ed0 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe8e41407a0 con 0x7fe8e41372f0 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e9d5c640 10 monclient(hunting): authenticate will time out at 1947.615087s 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e2ffd640 1 -- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8e4136ed0 msgr2=0x7fe8e41359e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3301/0 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e40a2a80 0x7fe8e41354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e2ffd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8e4136ed0 0x7fe8e41359e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:17.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e40a2a80 0x7fe8e41354a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46618/0 (socket says 172.21.15.43:46618) 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 -- 172.21.15.43:0/2031591219 learned_addr learned my addr 172.21.15.43:0/2031591219 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): get_auth_request con 0x7fe8e40a2a80 auth_method 0 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e3fff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe8e41372f0 0x7fe8e406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient(hunting): handle_auth_done global_id 65556 payload 824 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe8e41372f0 msgr2=0x7fe8e406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe8e41372f0 0x7fe8e406d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8e4136ed0 msgr2=0x7fe8e41359e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe8e4136ed0 0x7fe8e41359e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 monclient: found mon.noname-a 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e409ff00 con 0x7fe8e40a2a80 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e3fff640 10 monclient: get_auth_request con 0x7fe8e41372f0 auth_method 0 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e3fff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe8e41372f0 0x7fe8e406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8e37fe640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e40a2a80 0x7fe8e41354a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc01c2e0 tx=0x7fe8dc019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8cbfff640 10 client.?.objecter ms_handle_connect 0x7fe8e40a2a80 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8cbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.485+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8dc0b9070 con 0x7fe8e40a2a80 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: dump: 2024-04-19T01:01:17.502 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.503 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 5 monclient: authenticate success, global_id 65556 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8dc0044b0 con 0x7fe8e40a2a80 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8dc0b0040 con 0x7fe8e40a2a80 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 monclient: _renew_subs 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e406f3e0 con 0x7fe8e40a2a80 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e17fa640 4 set_mon_vals no callback set 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 client.65556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:17.504 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: dump: 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.505 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 monclient: _renew_subs 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e413bd00 con 0x7fe8e40a2a80 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 1 librados: init done 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 0x7fe8e406e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e406d140 con 0x7fe8e406dd40 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe8dc004040 con 0x7fe8e40a2a80 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 0x7fe8e406e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient: get_auth_request con 0x7fe8e406dd40 auth_method 0 2024-04-19T01:01:17.506 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe8d0038e30 0x7fe8d003b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(617..617 src has 501..617) ==== 6384+0+0 (secure 0 0 0) 0x7fe8dc0e27f0 con 0x7fe8e40a2a80 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 client.65556.objecter ms_dispatch 0x7fe8e4000ba0 osd_map(617..617 src has 501..617) 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 3 client.65556.objecter handle_osd_map got epochs [617,617] > 0 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 3 client.65556.objecter handle_osd_map decoding full epoch 617 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_done global_id 144272 payload 824 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8e4072400 con 0x7fe8e406dd40 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 20 client.65556.objecter dump_active .. 0 homeless 2024-04-19T01:01:17.507 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e3fff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe8d0038e30 0x7fe8d003b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e2ffd640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 0x7fe8e406e120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800ef40 tx=0x7fe8d800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 client.65556.objecter ms_handle_connect 0x7fe8e406dd40 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 client.65556.objecter resend_mon_ops 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e3fff640 10 monclient: get_auth_request con 0x7fe8d0038e30 auth_method 0 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8d8018030 con 0x7fe8e406dd40 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: dump: 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.508 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe8d800cda0 con 0x7fe8e406dd40 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 msgr2=0x7fe8e406e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 0x7fe8e406e120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800ef40 tx=0x7fe8d800c490 comp rx=0 tx=0).stop 2024-04-19T01:01:17.509 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 msgr2=0x7fe8e406e120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406dd40 0x7fe8e406e120 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe8d800ef40 tx=0x7fe8d800c490 comp rx=0 tx=0).stop 2024-04-19T01:01:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8cbfff640 10 client.65556.objecter ms_handle_connect 0x7fe8d0038e30 2024-04-19T01:01:17.510 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.489+0000 7fe8e3fff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe8d0038e30 0x7fe8d003b2f0 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4006fd0 tx=0x7fe8d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.511 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 0x7fe8e4137860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e413d8b0 con 0x7fe8e406e660 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 0x7fe8e4137860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient: get_auth_request con 0x7fe8e406e660 auth_method 0 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 10 monclient(hunting): handle_auth_done global_id 144281 payload 824 2024-04-19T01:01:17.512 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 1 -- 172.21.15.43:0/2031591219 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe8e413cd10 con 0x7fe8e406e660 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e2ffd640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 0x7fe8e4137860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8000f90 tx=0x7fe8d8004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 client.65556.objecter ms_handle_connect 0x7fe8e406e660 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 client.65556.objecter resend_mon_ops 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe8d801e240 con 0x7fe8e406e660 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 monclient: dump: 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.513 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9253 (secure 0 0 0) 0x7fe8d801e240 con 0x7fe8e406e660 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 msgr2=0x7fe8e4137860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 0x7fe8e4137860 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe8d8000f90 tx=0x7fe8d8004ac0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 msgr2=0x7fe8e4137860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.514 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8cbfff640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fe8e406e660 0x7fe8e4137860 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 42127, 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:17.515 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.516 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.517 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:17.518 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.523 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:17.525 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.527 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.528 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.529 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.530 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.531 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.532 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.533 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.534 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.535 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.536 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:17.536 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 10 librados: watch_flush enter 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 10 librados: watch_flush exit 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 20 client.65556.objecter shutdown clearing up homeless session... 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 10 client.65556.objecter successfully canceled tick 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe8d0038e30 msgr2=0x7fe8d003b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe8d0038e30 0x7fe8d003b2f0 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4006fd0 tx=0x7fe8d4008040 comp rx=0 tx=0).stop 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 10 monclient: shutdown 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e40a2a80 msgr2=0x7fe8e41354a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 --2- 172.21.15.43:0/2031591219 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe8e40a2a80 0x7fe8e41354a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc01c2e0 tx=0x7fe8dc019250 comp rx=0 tx=0).stop 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e3fff640 1 -- 172.21.15.43:0/2031591219 reap_dead start 2024-04-19T01:01:17.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e3fff640 1 -- 172.21.15.43:0/2031591219 reap_dead start 2024-04-19T01:01:17.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 shutdown_connections 2024-04-19T01:01:17.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 >> 172.21.15.43:0/2031591219 conn(0x7fe8e40a6410 msgr2=0x7fe8e40a85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 shutdown_connections 2024-04-19T01:01:17.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 -- 172.21.15.43:0/2031591219 wait complete. 2024-04-19T01:01:17.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.497+0000 7fe8e9d5c640 1 librados: shutdown 2024-04-19T01:01:17.539 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:17.539 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon c 2024-04-19T01:01:17.539 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient: get_monmap_and_config 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient: monmap: 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:17.667225+0000 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:17.667225+0000 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.664 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:17.665 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:17.666 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 Processor -- start 2024-04-19T01:01:17.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 -- start start 2024-04-19T01:01:17.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient: init 2024-04-19T01:01:17.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient: _add_conns ranks=[1,7,10,3,9,5,6,8,4,0,2] 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f2b080a2b80 addr v1:172.21.15.161:6789/0 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b08136f40 0x7f2b0813bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): picked mon.noname-i con 0x7f2b08136f40 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2b08137320 0x7f2b0813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): picked mon.noname-f con 0x7f2b08137320 addr v2:172.21.15.161:3300/0 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b08073690 con 0x7f2b080a2b80 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2b080a07a0 con 0x7f2b08137320 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f2b080730d0 con 0x7f2b08136f40 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b0d2c0640 10 monclient(hunting): authenticate will time out at 1947.795000s 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b067fc640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b08136f40 msgr2=0x7f2b0813bc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b067fc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b08136f40 0x7f2b0813bc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b06ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2b080a2b80 0x7f2b080a2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:33636/0 (socket says 172.21.15.43:33636) 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b06ffd640 1 -- 172.21.15.43:0/3057159289 learned_addr learned my addr 172.21.15.43:0/3057159289 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 1 --2- 172.21.15.43:0/3057159289 >> v2:172.21.15.161:3300/0 conn(0x7f2b08137320 0x7f2b0813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): get_auth_request con 0x7f2b08137320 auth_method 0 2024-04-19T01:01:17.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b05ffb640 1 -- 172.21.15.43:0/3057159289 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006266659 0 0) 0x7f2af8001120 con 0x7f2b080a2b80 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b05ffb640 10 monclient(hunting): _negotiate 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b05ffb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b05ffb640 10 monclient(hunting): my global_id is 144289 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.665+0000 7f2b05ffb640 1 -- 172.21.15.43:0/3057159289 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2af0003530 con 0x7f2b080a2b80 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 10 monclient(hunting): handle_auth_done global_id 144280 payload 274 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 1 -- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b08136f40 msgr2=0x7f2b0813bc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 1 --2- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b08136f40 0x7f2b0813bc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 1 -- 172.21.15.43:0/3057159289 >> v1:172.21.15.161:6789/0 conn(0x7f2b080a2b80 legacy=0x7f2b080a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 1 monclient: found mon.noname-f 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T01:01:17.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 1 -- 172.21.15.43:0/3057159289 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b080733b0 con 0x7f2b08137320 2024-04-19T01:01:17.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b077fe640 1 --2- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b08137320 0x7f2b0813e1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009d90 tx=0x7f2b0001c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bdbc34162735070 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 1 -- 172.21.15.43:0/3057159289 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b000b2070 con 0x7f2b08137320 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: dump: 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.681 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 1 -- 172.21.15.43:0/3057159289 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b000a5a90 con 0x7f2b08137320 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 1 -- 172.21.15.43:0/3057159289 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b000a5c40 con 0x7f2b08137320 2024-04-19T01:01:17.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b0d2c0640 5 monclient: authenticate success, global_id 144280 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b0d2c0640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b0d2c0640 4 set_mon_vals no callback set 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.669+0000 7f2b05ffb640 10 monclient: dump: 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.683 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: shutdown 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b08137320 msgr2=0x7f2b0813e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b08137320 0x7f2b0813e1a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b00009d90 tx=0x7f2b0001c850 comp rx=0 tx=0).stop 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3057159289 shutdown_connections 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2b08137320 0x7f2b0813e1a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3057159289 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b08136f40 0x7f2b0813bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3057159289 >> 172.21.15.43:0/3057159289 conn(0x7f2b080a9900 msgr2=0x7f2b080a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3057159289 shutdown_connections 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3057159289 wait complete. 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: monmap: 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:17.685 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 librados: starting msgr at 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 librados: starting objecter 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 Processor -- start 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- start start 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 librados: setting wanted keys 2024-04-19T01:01:17.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 librados: calling monclient init 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: init 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: _add_conns ranks=[7,6,8,0,3,1,5,4,2] 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b080a2b80 0x7f2b0809eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): picked mon.noname-i con 0x7f2b080a2b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b08136f40 0x7f2b08135b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): picked mon.noname-d con 0x7f2b08136f40 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 0x7f2b08136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): picked mon.noname-e con 0x7f2b08137320 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f2b08141290 con 0x7f2b08136f40 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f2b08141110 con 0x7f2b080a2b80 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f2b08141410 con 0x7f2b08137320 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): authenticate will time out at 1947.801982s 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b080a2b80 msgr2=0x7f2b0809eac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b080a2b80 0x7f2b0809eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 0x7f2b08136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 0x7f2b08136070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:46814/0 (socket says 172.21.15.43:46814) 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 -- 172.21.15.43:0/3010362090 learned_addr learned my addr 172.21.15.43:0/3010362090 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): get_auth_request con 0x7f2b08137320 auth_method 0 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.688 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b08136f40 0x7f2b08135b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): get_auth_request con 0x7f2b08136f40 auth_method 0 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient(hunting): handle_auth_done global_id 144269 payload 824 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b080a2b80 msgr2=0x7f2b0809eac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f2b080a2b80 0x7f2b0809eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b08136f40 msgr2=0x7f2b08135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b08136f40 0x7f2b08135b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 monclient: found mon.noname-e 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b081d0930 con 0x7f2b08137320 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b067fc640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2b08136f40 0x7f2b08135b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b077fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 0x7f2b08136070 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b0001a480 tx=0x7f2b000195f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 client.?.objecter ms_handle_connect 0x7f2b08137320 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b000b2050 con 0x7f2b08137320 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:01:17.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: dump: 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.691 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 5 monclient: authenticate success, global_id 144269 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: _renew_subs 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b000be070 con 0x7f2b08137320 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b08072500 con 0x7f2b08137320 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2b00003c70 con 0x7f2b08137320 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 client.144269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 10 monclient: dump: 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.693 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b04ff9640 4 set_mon_vals no callback set 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: _renew_subs 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b0809f640 con 0x7f2b08137320 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 librados: init done 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 0x7f2b0811fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f2b08122270 con 0x7f2b0811f750 2024-04-19T01:01:17.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 0x7f2b0811fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f2b00004040 con 0x7f2b08137320 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 10 monclient: get_auth_request con 0x7f2b0811f750 auth_method 0 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.673+0000 7f2b06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2af4038e40 0x7f2af403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(618..618 src has 501..618) ==== 6757+0+0 (secure 0 0 0) 0x7f2b000e4670 con 0x7f2b08137320 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 client.144269.objecter ms_dispatch 0x7f2b08000ba0 osd_map(618..618 src has 501..618) 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 3 client.144269.objecter handle_osd_map got epochs [618,618] > 0 2024-04-19T01:01:17.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 3 client.144269.objecter handle_osd_map decoding full epoch 618 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_done global_id 134446 payload 824 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b06ffd640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b08069630 con 0x7f2b0811f750 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 20 client.144269.objecter dump_active .. 0 homeless 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b067fc640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2af4038e40 0x7f2af403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b06ffd640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 0x7f2b0811fb30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2af8004880 tx=0x7f2af800cb10 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 client.144269.objecter ms_handle_connect 0x7f2b0811f750 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 client.144269.objecter resend_mon_ops 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b067fc640 10 monclient: get_auth_request con 0x7f2af4038e40 auth_method 0 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2af8017070 con 0x7f2b0811f750 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 monclient: dump: 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.696 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.697 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2af800ce50 con 0x7f2b0811f750 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 msgr2=0x7f2b0811fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 0x7f2b0811fb30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2af8004880 tx=0x7f2af800cb10 comp rx=0 tx=0).stop 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 msgr2=0x7f2b0811fb30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b0811f750 0x7f2b0811fb30 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2af8004880 tx=0x7f2af800cb10 comp rx=0 tx=0).stop 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2aef7fe640 10 client.144269.objecter ms_handle_connect 0x7f2af4038e40 2024-04-19T01:01:17.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.677+0000 7f2b067fc640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2af4038e40 0x7f2af403b300 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7f2afc006e50 tx=0x7f2afc006da0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b0d2c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 0x7f2b080a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b0d2c0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f2b0806d140 con 0x7f2b08120070 2024-04-19T01:01:17.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 0x7f2b080a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient: get_auth_request con 0x7f2b08120070 auth_method 0 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 10 monclient(hunting): handle_auth_done global_id 134455 payload 824 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 1 -- 172.21.15.43:0/3010362090 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b08121d40 con 0x7f2b08120070 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2b06ffd640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 0x7f2b080a3a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2af8014000 tx=0x7f2af800f000 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2aef7fe640 10 client.144269.objecter ms_handle_connect 0x7f2b08120070 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.681+0000 7f2aef7fe640 10 client.144269.objecter resend_mon_ops 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2af8007500 con 0x7f2b08120070 2024-04-19T01:01:17.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 10 monclient: dump: 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.702 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9592 (secure 0 0 0) 0x7f2af8007500 con 0x7f2b08120070 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 msgr2=0x7f2b080a3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 0x7f2b080a3a00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2af8014000 tx=0x7f2af800f000 comp rx=0 tx=0).stop 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 msgr2=0x7f2b080a3a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2aef7fe640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f2b08120070 0x7f2b080a3a00 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2af8014000 tx=0x7f2af800f000 comp rx=0 tx=0).stop 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 182389, 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:17.704 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.705 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.706 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:17.707 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.708 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.709 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:17.712 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.714 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:17.715 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.716 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.717 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.718 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:17.718 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:17.718 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.718 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.718 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.719 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:17.719 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.719 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.719 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.719 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:17.720 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.720 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.720 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.720 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.720 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.721 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.722 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.723 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.724 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.725 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.726 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.727 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.728 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.729 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:01:17.730 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:17.731 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:17.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 10 librados: watch_flush enter 2024-04-19T01:01:17.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 10 librados: watch_flush exit 2024-04-19T01:01:17.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 20 client.144269.objecter shutdown clearing up homeless session... 2024-04-19T01:01:17.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 10 client.144269.objecter successfully canceled tick 2024-04-19T01:01:17.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2af4038e40 msgr2=0x7f2af403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f2af4038e40 0x7f2af403b300 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7f2afc006e50 tx=0x7f2afc006da0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 10 monclient: shutdown 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 msgr2=0x7f2b08136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b077fe640 1 -- 172.21.15.43:0/3010362090 reap_dead start 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 0x7f2b08136070 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2b0001a480 tx=0x7f2b000195f0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 shutdown_connections 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 --2- 172.21.15.43:0/3010362090 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2b08137320 0x7f2b08136070 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 >> 172.21.15.43:0/3010362090 conn(0x7f2b080a9900 msgr2=0x7f2b080a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 shutdown_connections 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 -- 172.21.15.43:0/3010362090 wait complete. 2024-04-19T01:01:17.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.685+0000 7f2b0d2c0640 1 librados: shutdown 2024-04-19T01:01:17.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:17.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon d 2024-04-19T01:01:17.735 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.853+0000 7f09f79d7640 10 monclient: get_monmap_and_config 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.853+0000 7f09f79d7640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient: monmap: 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:17.858474+0000 2024-04-19T01:01:17.855 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:17.858474+0000 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.856 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:17.857 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:17.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 Processor -- start 2024-04-19T01:01:17.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 -- start start 2024-04-19T01:01:17.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient: init 2024-04-19T01:01:17.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient: _add_conns ranks=[5,8,6,0,7,9,4,2,10,3,1] 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): picked mon.noname-h con 0x7f09f0138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f09f00a5bb0 0x7f09f0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): picked mon.noname-e con 0x7f09f00a5bb0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 0x7f09f0137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): picked mon.noname-d con 0x7f09f0137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f09f00a1020 con 0x7f09f0138a50 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f09f01398b0 con 0x7f09f0137960 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f09f00a0780 con 0x7f09f00a5bb0 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f79d7640 10 monclient(hunting): authenticate will time out at 1947.985553s 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 0x7f09f0137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f09f00a5bb0 0x7f09f0139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 0x7f09f0137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59162/0 (socket says 172.21.15.43:59162) 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 1 -- 172.21.15.43:0/691625793 learned_addr learned my addr 172.21.15.43:0/691625793 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): get_auth_request con 0x7f09f0137960 auth_method 0 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): get_auth_request con 0x7f09f00a5bb0 auth_method 0 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5773640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5773640 10 monclient(hunting): get_auth_request con 0x7f09f0138a50 auth_method 0 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5773640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5773640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5773640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient(hunting): handle_auth_done global_id 144278 payload 274 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 -- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 msgr2=0x7f09f0137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 0x7f09f0137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 -- 172.21.15.43:0/691625793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 msgr2=0x7f09f0138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 monclient: found mon.noname-e 2024-04-19T01:01:17.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f4f72640 1 -- 172.21.15.43:0/691625793 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00a0d40 con 0x7f09f00a5bb0 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.857+0000 7f09f5f74640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 0x7f09f0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f4f72640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f09f00a5bb0 0x7f09f0139370 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097f0 tx=0x7f09e001c370 comp rx=0 tx=0).ready entity=mon.8 client_cookie=91545875aba81314 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 1 -- 172.21.15.43:0/691625793 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09e00b0070 con 0x7f09f00a5bb0 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: dump: 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.866 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.867 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 1 -- 172.21.15.43:0/691625793 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e00aa810 con 0x7f09f00a5bb0 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 1 -- 172.21.15.43:0/691625793 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09e00a9300 con 0x7f09f00a5bb0 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 5 monclient: authenticate success, global_id 144278 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 4 set_mon_vals no callback set 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09e7fff640 10 monclient: dump: 2024-04-19T01:01:17.868 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.869 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: shutdown 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f09f00a5bb0 msgr2=0x7f09f0139370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f09f00a5bb0 0x7f09f0139370 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097f0 tx=0x7f09e001c370 comp rx=0 tx=0).stop 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- 172.21.15.43:0/691625793 shutdown_connections 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0137960 0x7f09f0137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f09f00a5bb0 0x7f09f0139370 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- 172.21.15.43:0/691625793 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- 172.21.15.43:0/691625793 >> 172.21.15.43:0/691625793 conn(0x7f09f00a7cb0 msgr2=0x7f09f00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- 172.21.15.43:0/691625793 shutdown_connections 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- 172.21.15.43:0/691625793 wait complete. 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: build_initial_monmap 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: monmap: 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:17.871 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 librados: starting msgr at 2024-04-19T01:01:17.872 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 librados: starting objecter 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 Processor -- start 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- start start 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 librados: setting wanted keys 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 librados: calling monclient init 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: init 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient: _add_conns ranks=[1,3,5,4,6,0,2,8,7] 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a5bb0 0x7f09f00a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): picked mon.noname-f con 0x7f09f00a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:01:17.880 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09f0137960 0x7f09f009df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): picked mon.noname-g con 0x7f09f0137960 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0135700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): picked mon.noname-h con 0x7f09f0138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00a2100 con 0x7f09f00a5bb0 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f09f00a1f80 con 0x7f09f0137960 2024-04-19T01:01:17.881 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f09f0142810 con 0x7f09f0138a50 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f79d7640 10 monclient(hunting): authenticate will time out at 1947.991475s 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f4f72640 1 -- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09f0137960 msgr2=0x7f09f009df30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3301/0 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f4f72640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09f0137960 0x7f09f009df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5773640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a5bb0 0x7f09f00a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0135700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0135700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:50324/0 (socket says 172.21.15.43:50324) 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5773640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a5bb0 0x7f09f00a4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:48080/0 (socket says 172.21.15.43:48080) 2024-04-19T01:01:17.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 1 -- 172.21.15.43:0/2867515957 learned_addr learned my addr 172.21.15.43:0/2867515957 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 10 monclient(hunting): get_auth_request con 0x7f09f0138a50 auth_method 0 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5f74640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5773640 10 monclient(hunting): get_auth_request con 0x7f09f00a5bb0 auth_method 0 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5773640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5773640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.861+0000 7f09f5773640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5f74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5f74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5f74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 10 monclient(hunting): handle_auth_done global_id 144298 payload 824 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 msgr2=0x7f09f0135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0135700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09f0137960 msgr2=0x7f09f009df30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f09f0137960 0x7f09f009df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 10 monclient: _finish_hunting 0 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 monclient: found mon.noname-f 2024-04-19T01:01:17.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f01369e0 con 0x7f09f00a5bb0 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5f74640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f09f0138a50 0x7f09f0135700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5773640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a5bb0 0x7f09f00a4990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00d5d0 tx=0x7f09ec00d9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 client.?.objecter ms_handle_connect 0x7f09f00a5bb0 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:17.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09ec012a60 con 0x7f09f00a5bb0 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: dump: 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.886 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09ec019070 con 0x7f09f00a5bb0 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 5 monclient: authenticate success, global_id 144298 2024-04-19T01:01:17.887 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 monclient: _renew_subs 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f009fc90 con 0x7f09f00a5bb0 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09ec012c40 con 0x7f09f00a5bb0 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e77fe640 4 set_mon_vals no callback set 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: dump: 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.888 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 client.144298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 monclient: _renew_subs 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f01d1220 con 0x7f09f00a5bb0 2024-04-19T01:01:17.889 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 1 librados: init done 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 0x7f09f009e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f09f006d140 con 0x7f09f009c3a0 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 0x7f09f009e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient: get_auth_request con 0x7f09f009c3a0 auth_method 0 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.890 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 10 monclient(hunting): handle_auth_done global_id 144285 payload 824 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09f0068750 con 0x7f09f009c3a0 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.1 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f09ec0125f0 con 0x7f09f00a5bb0 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f4f72640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 0x7f09f009e470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097c0 tx=0x7f09e001abb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f09d4038e10 0x7f09d403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 client.144298.objecter ms_handle_connect 0x7f09f009c3a0 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 client.144298.objecter resend_mon_ops 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.1 v2:172.21.15.161:3300/0 5 ==== osd_map(618..618 src has 501..618) ==== 6757+0+0 (secure 0 0 0) 0x7f09ec00f040 con 0x7f09f00a5bb0 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 client.144298.objecter ms_dispatch 0x7f09f0000ba0 osd_map(618..618 src has 501..618) 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 3 client.144298.objecter handle_osd_map got epochs [618,618] > 0 2024-04-19T01:01:17.891 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 3 client.144298.objecter handle_osd_map decoding full epoch 618 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 20 client.144298.objecter dump_active .. 0 homeless 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09e00b9030 con 0x7f09f009c3a0 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: dump: 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.892 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5f74640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f09d4038e10 0x7f09d403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09f5f74640 10 monclient: get_auth_request con 0x7f09d4038e10 auth_method 0 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09e00b1040 con 0x7f09f009c3a0 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 msgr2=0x7f09f009e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 0x7f09f009e470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097c0 tx=0x7f09e001abb0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 msgr2=0x7f09f009e470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.893 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.865+0000 7f09e5ffb640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f009c3a0 0x7f09f009e470 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f09e00097c0 tx=0x7f09e001abb0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.869+0000 7f09e5ffb640 10 client.144298.objecter ms_handle_connect 0x7f09d4038e10 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.869+0000 7f09f5f74640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f09d4038e10 0x7f09d403b2d0 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009c90 tx=0x7f09e800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 0x7f09f0136020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f09f009e9b0 con 0x7f09f0135c40 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 0x7f09f0136020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient: get_auth_request con 0x7f09f0135c40 auth_method 0 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:17.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 10 monclient(hunting): handle_auth_done global_id 144294 payload 824 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 1 -- 172.21.15.43:0/2867515957 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09f0136560 con 0x7f09f0135c40 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 client.144298.objecter ms_handle_connect 0x7f09f0135c40 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f4f72640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 0x7f09f0136020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09e001c1a0 tx=0x7f09e0019110 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 client.144298.objecter resend_mon_ops 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f09e00b9070 con 0x7f09f0135c40 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 monclient: dump: 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:17.895 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:17.896 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7f09e0003620 con 0x7f09f0135c40 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 msgr2=0x7f09f0136020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 0x7f09f0136020 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09e001c1a0 tx=0x7f09e0019110 comp rx=0 tx=0).stop 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 msgr2=0x7f09f0136020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09e5ffb640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f09f0135c40 0x7f09f0136020 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f09e001c1a0 tx=0x7f09e0019110 comp rx=0 tx=0).stop 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:17.897 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 42503, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:17.898 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.899 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.900 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:17.901 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:17.902 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:17.903 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:17.904 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.905 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.906 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:17.907 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.908 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:17.909 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.910 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.911 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.912 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.913 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.914 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.915 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.916 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.917 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.918 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.919 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:17.920 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:17.921 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:17.921 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 10 librados: watch_flush enter 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 10 librados: watch_flush exit 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 20 client.144298.objecter shutdown clearing up homeless session... 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 10 client.144298.objecter successfully canceled tick 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f09d4038e10 msgr2=0x7f09d403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f09d4038e10 0x7f09d403b2d0 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009c90 tx=0x7f09e800b040 comp rx=0 tx=0).stop 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 10 monclient: shutdown 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a5bb0 msgr2=0x7f09f00a4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:17.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 --2- 172.21.15.43:0/2867515957 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09f00a5bb0 0x7f09f00a4990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00d5d0 tx=0x7f09ec00d9d0 comp rx=0 tx=0).stop 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f5f74640 1 -- 172.21.15.43:0/2867515957 reap_dead start 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f5f74640 1 -- 172.21.15.43:0/2867515957 reap_dead start 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 shutdown_connections 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 >> 172.21.15.43:0/2867515957 conn(0x7f09f00a7cb0 msgr2=0x7f09f006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 shutdown_connections 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 -- 172.21.15.43:0/2867515957 wait complete. 2024-04-19T01:01:17.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:17.873+0000 7f09f79d7640 1 librados: shutdown 2024-04-19T01:01:17.924 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:17.924 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon e 2024-04-19T01:01:17.925 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient: get_monmap_and_config 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient: build_initial_monmap 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient: monmap: 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:18.043159+0000 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:18.043159+0000 2024-04-19T01:01:18.040 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:18.041 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 Processor -- start 2024-04-19T01:01:18.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 -- start start 2024-04-19T01:01:18.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient: init 2024-04-19T01:01:18.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:18.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient: _add_conns ranks=[7,5,2,0,4,6,3,1,8,9,10] 2024-04-19T01:01:18.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3dc0135d20 0x7f3dc0136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): picked mon.noname-i con 0x7f3dc0135d20 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc0135860 0x7f3dc00a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): picked mon.noname-h con 0x7f3dc0135860 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 0x7f3dc00a4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): picked mon.noname-b con 0x7f3dc00a3e80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc00a1780 con 0x7f3dc00a3e80 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f3dc00a2400 con 0x7f3dc0135860 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f3dc00a1130 con 0x7f3dc0135d20 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:18.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dc59c5640 10 monclient(hunting): authenticate will time out at 1948.170563s 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbeffd640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3dc0135d20 msgr2=0x7f3dc0136100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbeffd640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3dc0135d20 0x7f3dc0136100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 0x7f3dc00a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 0x7f3dc00a4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:42824/0 (socket says 172.21.15.43:42824) 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/3405560485 learned_addr learned my addr 172.21.15.43:0/3405560485 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): get_auth_request con 0x7f3dc00a3e80 auth_method 0 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbe7fc640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc0135860 0x7f3dc00a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbe7fc640 10 monclient(hunting): get_auth_request con 0x7f3dc0135860 auth_method 0 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbe7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_done global_id 144290 payload 274 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3dc0135d20 msgr2=0x7f3dc0136100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3dc0135d20 0x7f3dc0136100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc0135860 msgr2=0x7f3dc00a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc0135860 0x7f3dc00a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 monclient: found mon.noname-b 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.041+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/3405560485 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc00a1410 con 0x7f3dc00a3e80 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbf7fe640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 0x7f3dc00a4300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009e80 tx=0x7f3db801c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88910d7f33d83b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 1 -- 172.21.15.43:0/3405560485 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3db80090d0 con 0x7f3dc00a3e80 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: dump: 2024-04-19T01:01:18.050 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.051 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:18.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 1 -- 172.21.15.43:0/3405560485 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db80a5b20 con 0x7f3dc00a3e80 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 5 monclient: authenticate success, global_id 144290 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 1 -- 172.21.15.43:0/3405560485 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3db80a5d10 con 0x7f3dc00a3e80 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 4 set_mon_vals no callback set 2024-04-19T01:01:18.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dbdffb640 10 monclient: dump: 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.054 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.055 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: shutdown 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- 172.21.15.43:0/3405560485 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 msgr2=0x7f3dc00a4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 0x7f3dc00a4300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3db8009e80 tx=0x7f3db801c9c0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- 172.21.15.43:0/3405560485 shutdown_connections 2024-04-19T01:01:18.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3dc00a3e80 0x7f3dc00a4300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc0135860 0x7f3dc00a3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/3405560485 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3dc0135d20 0x7f3dc0136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- 172.21.15.43:0/3405560485 >> 172.21.15.43:0/3405560485 conn(0x7f3dc00a85c0 msgr2=0x7f3dc00a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- 172.21.15.43:0/3405560485 shutdown_connections 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- 172.21.15.43:0/3405560485 wait complete. 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: build_initial_monmap 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: monmap: 2024-04-19T01:01:18.057 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:18.058 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 librados: starting msgr at 2024-04-19T01:01:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 librados: starting objecter 2024-04-19T01:01:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 Processor -- start 2024-04-19T01:01:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- start start 2024-04-19T01:01:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 librados: setting wanted keys 2024-04-19T01:01:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 librados: calling monclient init 2024-04-19T01:01:18.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: init 2024-04-19T01:01:18.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:18.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient: _add_conns ranks=[5,3,4,6,0,7,1,8,2] 2024-04-19T01:01:18.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc00a3e80 0x7f3dc013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): picked mon.noname-h con 0x7f3dc00a3e80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:18.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3dc0135860 0x7f3dc01345f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): picked mon.noname-g con 0x7f3dc0135860 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 0x7f3dc011e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): picked mon.noname-c con 0x7f3dc0135d20 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc013f7c0 con 0x7f3dc0135860 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f3dc0073460 con 0x7f3dc0135d20 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f3dc013f940 con 0x7f3dc00a3e80 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.045+0000 7f3dc59c5640 10 monclient(hunting): authenticate will time out at 1948.175945s 2024-04-19T01:01:18.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbe7fc640 1 -- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3dc0135860 msgr2=0x7f3dc01345f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3301/0 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 0x7f3dc011e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbe7fc640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3dc0135860 0x7f3dc01345f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 0x7f3dc011e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47084/0 (socket says 172.21.15.43:47084) 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/556907742 learned_addr learned my addr 172.21.15.43:0/556907742 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): get_auth_request con 0x7f3dc0135d20 auth_method 0 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc00a3e80 0x7f3dc013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): get_auth_request con 0x7f3dc00a3e80 auth_method 0 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient(hunting): handle_auth_done global_id 134464 payload 824 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc00a3e80 msgr2=0x7f3dc013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f3dc00a3e80 0x7f3dc013b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3dc0135860 msgr2=0x7f3dc01345f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f3dc0135860 0x7f3dc01345f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 monclient: found mon.noname-c 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc01cf6b0 con 0x7f3dc0135d20 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbf7fe640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 0x7f3dc011e580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3db801c7f0 tx=0x7f3db8019760 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 client.?.objecter ms_handle_connect 0x7f3dc0135d20 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3db80bc020 con 0x7f3dc0135d20 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: dump: 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.074 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: _finish_auth 0 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: _check_auth_tickets 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db8003e10 con 0x7f3dc0135d20 2024-04-19T01:01:18.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 5 monclient: authenticate success, global_id 134464 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 monclient: _renew_subs 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc0120fa0 con 0x7f3dc0135d20 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3db80044b0 con 0x7f3dc0135d20 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: dump: 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.076 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.077 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 client.134464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 monclient: _renew_subs 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbd7fa640 4 set_mon_vals no callback set 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc00736a0 con 0x7f3dc0135d20 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 1 librados: init done 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 0x7f3dc002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f3dc006d140 con 0x7f3dc011f280 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 0x7f3dc002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient: get_auth_request con 0x7f3dc011f280 auth_method 0 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f3db80bf020 con 0x7f3dc0135d20 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3dac038e10 0x7f3dac03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_done global_id 144287 payload 824 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(618..618 src has 501..618) ==== 6757+0+0 (secure 0 0 0) 0x7f3db80e42d0 con 0x7f3dc0135d20 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 client.134464.objecter ms_dispatch 0x7f3dc0000ba0 osd_map(618..618 src has 501..618) 2024-04-19T01:01:18.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 3 client.134464.objecter handle_osd_map got epochs [618,618] > 0 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 3 client.134464.objecter handle_osd_map decoding full epoch 618 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dc0137f60 con 0x7f3dc011f280 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 20 client.134464.objecter dump_active .. 0 homeless 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbe7fc640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3dac038e10 0x7f3dac03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 client.134464.objecter ms_handle_connect 0x7f3dc011f280 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 client.134464.objecter resend_mon_ops 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 0x7f3dc002c450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3db000d700 tx=0x7f3db000dbc0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbe7fc640 10 monclient: get_auth_request con 0x7f3dac038e10 auth_method 0 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3db0003e30 con 0x7f3dc011f280 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: dump: 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.081 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3db0017070 con 0x7f3dc011f280 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 msgr2=0x7f3dc002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 0x7f3dc002c450 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3db000d700 tx=0x7f3db000dbc0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 msgr2=0x7f3dc002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc011f280 0x7f3dc002c450 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3db000d700 tx=0x7f3db000dbc0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3da7fff640 10 client.134464.objecter ms_handle_connect 0x7f3dac038e10 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.049+0000 7f3dbe7fc640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3dac038e10 0x7f3dac03b2d0 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f3dc013cac0 tx=0x7f3db40092e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f3dc0076500 con 0x7f3dc002c990 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient: get_auth_request con 0x7f3dc002c990 auth_method 0 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_done global_id 144296 payload 824 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3dc0120c20 con 0x7f3dc002c990 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 client.134464.objecter ms_handle_connect 0x7f3dc002c990 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3db0016000 tx=0x7f3db00059e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 client.134464.objecter resend_mon_ops 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 msgr2=0x7f3dc002cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 msgr2=0x7f3dc002cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3db0016000 tx=0x7f3db00059e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3db0016000 tx=0x7f3db00059e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 msgr2=0x7f3dc002cda0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 0x7f3dac0447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 msgr2=0x7f3dc002cda0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dc002c990 0x7f3dc002cda0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f3dac041e30 con 0x7f3dac0423c0 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 0x7f3dac0447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient: get_auth_request con 0x7f3dac0423c0 auth_method 0 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 10 monclient(hunting): handle_auth_done global_id 144305 payload 824 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 -- 172.21.15.43:0/556907742 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3db0017070 con 0x7f3dac0423c0 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 client.134464.objecter ms_handle_connect 0x7f3dac0423c0 2024-04-19T01:01:18.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 client.134464.objecter resend_mon_ops 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbeffd640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 0x7f3dac0447b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3db000dc00 tx=0x7f3db0008040 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3db0033070 con 0x7f3dac0423c0 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 monclient: dump: 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.086 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7f3db0006670 con 0x7f3dac0423c0 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 msgr2=0x7f3dac0447b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 0x7f3dac0447b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3db000dc00 tx=0x7f3db0008040 comp rx=0 tx=0).stop 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 msgr2=0x7f3dac0447b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3da7fff640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3dac0423c0 0x7f3dac0447b0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3db000dc00 tx=0x7f3db0008040 comp rx=0 tx=0).stop 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dbf7fe640 1 -- 172.21.15.43:0/556907742 reap_dead start 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:18.088 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 42687, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.089 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.090 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.091 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:18.092 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:18.093 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.094 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.095 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.096 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.097 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.098 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:18.099 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:18.100 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.101 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.102 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:18.103 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:18.104 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.105 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:18.106 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.109 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:18.112 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:01:18.113 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.113 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.113 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:18.113 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 10 librados: watch_flush enter 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 10 librados: watch_flush exit 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 20 client.134464.objecter shutdown clearing up homeless session... 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 10 client.134464.objecter successfully canceled tick 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3dac038e10 msgr2=0x7f3dac03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3dac038e10 0x7f3dac03b2d0 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f3dc013cac0 tx=0x7f3db40092e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 10 monclient: shutdown 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:18.115 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 msgr2=0x7f3dc011e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 0x7f3dc011e580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3db801c7f0 tx=0x7f3db8019760 comp rx=0 tx=0).stop 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 shutdown_connections 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3dac038e10 0x7f3dac03b2d0 unknown :-1 s=CLOSED pgs=3619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 --2- 172.21.15.43:0/556907742 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f3dc0135d20 0x7f3dc011e580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 >> 172.21.15.43:0/556907742 conn(0x7f3dc00a85c0 msgr2=0x7f3dc00a6760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 shutdown_connections 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 -- 172.21.15.43:0/556907742 wait complete. 2024-04-19T01:01:18.116 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.057+0000 7f3dc59c5640 1 librados: shutdown 2024-04-19T01:01:18.117 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:18.117 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon f 2024-04-19T01:01:18.117 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:01:18.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.233+0000 7f79bd79e640 10 monclient: get_monmap_and_config 2024-04-19T01:01:18.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.233+0000 7f79bd79e640 10 monclient: build_initial_monmap 2024-04-19T01:01:18.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.233+0000 7f79bd79e640 10 monclient: monmap: 2024-04-19T01:01:18.236 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:18.236 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.237 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:18.238359+0000 2024-04-19T01:01:18.237 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:18.238359+0000 2024-04-19T01:01:18.237 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:18.237 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.237 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:18.238 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:18.238 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:18.238 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:18.238 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:18.238 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:18.238 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:18.239 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:18.239 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:18.239 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:18.239 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:18.239 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 Processor -- start 2024-04-19T01:01:18.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 -- start start 2024-04-19T01:01:18.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient: init 2024-04-19T01:01:18.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient: _add_conns ranks=[4,6,1,2,8,3,9,0,5,10,7] 2024-04-19T01:01:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 0x7f79b81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): picked mon.noname-c con 0x7f79b8138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:01:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f79b80a5b80 0x7f79b81395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): picked mon.noname-d con 0x7f79b80a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f79b81358b0 addr v1:172.21.15.161:6789/0 2024-04-19T01:01:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b80a3dc0 con 0x7f79b81358b0 2024-04-19T01:01:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f79b80a4410 con 0x7f79b8138cd0 2024-04-19T01:01:18.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f79b8139b30 con 0x7f79b80a5b80 2024-04-19T01:01:18.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:18.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79bd79e640 10 monclient(hunting): authenticate will time out at 1948.365493s 2024-04-19T01:01:18.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 0x7f79b81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f79b80a5b80 0x7f79b81395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 0x7f79b81390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47090/0 (socket says 172.21.15.43:47090) 2024-04-19T01:01:18.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 -- 172.21.15.43:0/2304333804 learned_addr learned my addr 172.21.15.43:0/2304333804 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:18.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): get_auth_request con 0x7f79b80a5b80 auth_method 0 2024-04-19T01:01:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): get_auth_request con 0x7f79b8138cd0 auth_method 0 2024-04-19T01:01:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_done global_id 134473 payload 274 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 -- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f79b80a5b80 msgr2=0x7f79b81395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 --2- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f79b80a5b80 0x7f79b81395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 -- 172.21.15.43:0/2304333804 >> v1:172.21.15.161:6789/0 conn(0x7f79b81358b0 legacy=0x7f79b8135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 monclient: found mon.noname-c 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:01:18.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b6ffd640 1 -- 172.21.15.43:0/2304333804 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b80a40a0 con 0x7f79b8138cd0 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.237+0000 7f79b67fc640 1 --2- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f79b80a5b80 0x7f79b81395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b6ffd640 1 --2- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 0x7f79b81390b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f79a00098a0 tx=0x7f79a001c3e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=70054b64971df3b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 1 -- 172.21.15.43:0/2304333804 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79a00b0070 con 0x7f79b8138cd0 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: dump: 2024-04-19T01:01:18.270 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.271 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: _finish_auth 0 2024-04-19T01:01:18.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 1 -- 172.21.15.43:0/2304333804 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a00aa810 con 0x7f79b8138cd0 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 1 -- 172.21.15.43:0/2304333804 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79a00a9300 con 0x7f79b8138cd0 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 5 monclient: authenticate success, global_id 134473 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 4 set_mon_vals no callback set 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79b5ffb640 10 monclient: dump: 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.273 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.274 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: shutdown 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 msgr2=0x7f79b81390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 --2- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 0x7f79b81390b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f79a00098a0 tx=0x7f79a001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- 172.21.15.43:0/2304333804 shutdown_connections 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 --2- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f79b80a5b80 0x7f79b81395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 --2- 172.21.15.43:0/2304333804 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f79b8138cd0 0x7f79b81390b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- 172.21.15.43:0/2304333804 >> 172.21.15.43:0/2304333804 conn(0x7f79b80ab2f0 msgr2=0x7f79b80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- 172.21.15.43:0/2304333804 shutdown_connections 2024-04-19T01:01:18.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- 172.21.15.43:0/2304333804 wait complete. 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: build_initial_monmap 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: monmap: 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:18.276 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 librados: starting msgr at 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 librados: starting objecter 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 Processor -- start 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- start start 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 librados: setting wanted keys 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 librados: calling monclient init 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: init 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:18.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient: _add_conns ranks=[0,8,2,1,3,7,4,6,5] 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 0x7f79b812aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): picked mon.noname-a con 0x7f79b80a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f79b81358b0 0x7f79b812afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): picked mon.noname-e con 0x7f79b81358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b8138cd0 0x7f79b81325b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): picked mon.noname-b con 0x7f79b8138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f79b8143ab0 con 0x7f79b80a5b80 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f79b80a20a0 con 0x7f79b8138cd0 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f79b8143c30 con 0x7f79b81358b0 2024-04-19T01:01:18.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.241+0000 7f79bd79e640 10 monclient(hunting): authenticate will time out at 1948.371937s 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 0x7f79b812aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b8138cd0 0x7f79b81325b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 0x7f79b812aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46642/0 (socket says 172.21.15.43:46642) 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 -- 172.21.15.43:0/1943721604 learned_addr learned my addr 172.21.15.43:0/1943721604 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b8138cd0 0x7f79b81325b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:42830/0 (socket says 172.21.15.43:42830) 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): get_auth_request con 0x7f79b80a5b80 auth_method 0 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): get_auth_request con 0x7f79b8138cd0 auth_method 0 2024-04-19T01:01:18.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f79b81358b0 0x7f79b812afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): get_auth_request con 0x7f79b81358b0 auth_method 0 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.280 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient(hunting): handle_auth_done global_id 65565 payload 824 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f79b81358b0 msgr2=0x7f79b812afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f79b81358b0 0x7f79b812afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b8138cd0 msgr2=0x7f79b81325b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f79b8138cd0 0x7f79b81325b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:01:18.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 monclient: found mon.noname-a 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b80a3590 con 0x7f79b80a5b80 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f79b81358b0 0x7f79b812afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b6ffd640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 0x7f79b812aa90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f79a00ae040 tx=0x7f79a00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 client.?.objecter ms_handle_connect 0x7f79b80a5b80 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79a00ba020 con 0x7f79b80a5b80 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.282 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: dump: 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.283 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: _finish_auth 0 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a00044b0 con 0x7f79b80a5b80 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 5 monclient: authenticate success, global_id 65565 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 monclient: _renew_subs 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b80a3040 con 0x7f79b80a5b80 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79a00b0040 con 0x7f79b80a5b80 2024-04-19T01:01:18.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b57fa640 4 set_mon_vals no callback set 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 client.65565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient: dump: 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.285 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 monclient: _renew_subs 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b809bbc0 con 0x7f79b80a5b80 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 1 librados: init done 2024-04-19T01:01:18.286 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79b806d140 con 0x7f79b8132af0 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f79a00aac80 con 0x7f79b80a5b80 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7998038e10 0x7f799803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(618..618 src has 501..618) ==== 6757+0+0 (secure 0 0 0) 0x7f79a00e34f0 con 0x7f79b80a5b80 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 client.65565.objecter ms_dispatch 0x7f79b8000ba0 osd_map(618..618 src has 501..618) 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 3 client.65565.objecter handle_osd_map got epochs [618,618] > 0 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 3 client.65565.objecter handle_osd_map decoding full epoch 618 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient: get_auth_request con 0x7f79b8132af0 auth_method 0 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.287 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 20 client.65565.objecter dump_active .. 0 homeless 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7998038e10 0x7f799803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 10 monclient: get_auth_request con 0x7f7998038e10 auth_method 0 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_done global_id 144307 payload 824 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79b80693b0 con 0x7f79b8132af0 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009fd0 tx=0x7f79a80079f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 client.65565.objecter ms_handle_connect 0x7f79b8132af0 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 client.65565.objecter resend_mon_ops 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 client.65565.objecter ms_handle_connect 0x7f7998038e10 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b77fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7998038e10 0x7f799803b2d0 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7f79b80a1710 tx=0x7f79b0007dc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.288 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 msgr2=0x7f79b8132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 msgr2=0x7f79b8132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009fd0 tx=0x7f79a80079f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009fd0 tx=0x7f79a80079f0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 msgr2=0x7f79b8132ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 0x7f7998043e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 msgr2=0x7f79b8132ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8132af0 0x7f79b8132ed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79980444a0 con 0x7f7998041a00 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 0x7f7998043e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient: get_auth_request con 0x7f7998041a00 auth_method 0 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.289 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.245+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79b67fc640 10 monclient(hunting): handle_auth_done global_id 144316 payload 824 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79b67fc640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f79a00c2ec0 con 0x7f7998041a00 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 0x7f7998043e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79a800efd0 tx=0x7f79a8004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 client.65565.objecter ms_handle_connect 0x7f7998041a00 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 client.65565.objecter resend_mon_ops 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79a802f070 con 0x7f7998041a00 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.290 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 monclient: dump: 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.291 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f79a8015410 con 0x7f7998041a00 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 msgr2=0x7f7998043e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 0x7f7998043e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79a800efd0 tx=0x7f79a8004ac0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 msgr2=0x7f7998043e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.249+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7998041a00 0x7f7998043e40 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79a800efd0 tx=0x7f79a8004ac0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.253+0000 7f79bd79e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.253+0000 7f79bd79e640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 0x7f79b8137a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.253+0000 7f79bd79e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.292 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.253+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f79b8068240 con 0x7f79b8133410 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 0x7f79b8137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient: get_auth_request con 0x7f79b8133410 auth_method 0 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 10 monclient(hunting): handle_auth_done global_id 144325 payload 824 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 1 -- 172.21.15.43:0/1943721604 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f79b8076190 con 0x7f79b8133410 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b67fc640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 0x7f79b8137a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79a8021000 tx=0x7f79a801e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 client.65565.objecter ms_handle_connect 0x7f79b8133410 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 client.65565.objecter resend_mon_ops 2024-04-19T01:01:18.293 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f79a8037070 con 0x7f79b8133410 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 monclient: dump: 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.294 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.295 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9251 (secure 0 0 0) 0x7f79a8031620 con 0x7f79b8133410 2024-04-19T01:01:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 msgr2=0x7f79b8137a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 0x7f79b8137a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79a8021000 tx=0x7f79a801e000 comp rx=0 tx=0).stop 2024-04-19T01:01:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 msgr2=0x7f79b8137a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.296 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79af7fe640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f79b8133410 0x7f79b8137a20 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79a8021000 tx=0x7f79a801e000 comp rx=0 tx=0).stop 2024-04-19T01:01:18.297 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79b77fe640 1 -- 172.21.15.43:0/1943721604 reap_dead start 2024-04-19T01:01:18.297 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:18.297 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:18.298 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:18.298 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:18.298 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:18.298 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 42887, 2024-04-19T01:01:18.298 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:18.298 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:18.299 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.300 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:01:18.300 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:18.300 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:18.300 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:18.300 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.300 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.301 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.301 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.301 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.301 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.301 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.301 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.302 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.302 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.302 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.302 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.302 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:18.302 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:18.303 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.303 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.303 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.303 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.303 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.303 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.304 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.305 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:18.305 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:18.305 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:18.305 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:18.305 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:18.305 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:18.306 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:18.307 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:18.307 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:18.307 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:18.307 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:18.307 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:18.307 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.308 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.308 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.308 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.308 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.308 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.308 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.309 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.309 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.309 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.309 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.309 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.310 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:18.310 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.310 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:18.310 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.310 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:18.310 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:18.311 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.311 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.311 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.311 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.311 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:18.311 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.312 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.312 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.312 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.312 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:18.312 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.312 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.313 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.313 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.313 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:18.313 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:18.313 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.313 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.314 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.314 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.314 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.314 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:18.314 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:18.314 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.315 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.315 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.315 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.315 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:18.315 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.315 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.316 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.317 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.318 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.319 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.320 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:18.321 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.322 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.323 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.324 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.325 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.326 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:18.327 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.328 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:18.329 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:18.330 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:18.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 10 librados: watch_flush enter 2024-04-19T01:01:18.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 10 librados: watch_flush exit 2024-04-19T01:01:18.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 20 client.65565.objecter shutdown clearing up homeless session... 2024-04-19T01:01:18.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 10 client.65565.objecter successfully canceled tick 2024-04-19T01:01:18.332 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7998038e10 msgr2=0x7f799803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7998038e10 0x7f799803b2d0 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7f79b80a1710 tx=0x7f79b0007dc0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 10 monclient: shutdown 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 msgr2=0x7f79b812aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 0x7f79b812aa90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f79a00ae040 tx=0x7f79a00191e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 shutdown_connections 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7998038e10 0x7f799803b2d0 unknown :-1 s=CLOSED pgs=3620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 --2- 172.21.15.43:0/1943721604 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f79b80a5b80 0x7f79b812aa90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 >> 172.21.15.43:0/1943721604 conn(0x7f79b80ab2f0 msgr2=0x7f79b80a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 shutdown_connections 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 -- 172.21.15.43:0/1943721604 wait complete. 2024-04-19T01:01:18.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.257+0000 7f79bd79e640 1 librados: shutdown 2024-04-19T01:01:18.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:18.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 7 for mon h 2024-04-19T01:01:18.334 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:01:18.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.461+0000 7ffaccb0f640 10 monclient: get_monmap_and_config 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.461+0000 7ffaccb0f640 10 monclient: build_initial_monmap 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.461+0000 7ffaccb0f640 10 monclient: monmap: 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:01:18.465904+0000 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:01:18.465904+0000 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:01:18.463 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:01:18.464 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 Processor -- start 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 -- start start 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient: init 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient: _add_conns ranks=[7,6,2,5,3,10,9,8,1,4,0] 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffac8138a50 0x7ffac8138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): picked mon.noname-i con 0x7ffac8138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 0x7ffac8139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): picked mon.noname-d con 0x7ffac80a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffac8137960 0x7ffac8137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): picked mon.noname-b con 0x7ffac8137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffac80a1020 con 0x7ffac8137960 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ffac81398b0 con 0x7ffac80a5bb0 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ffac80a0780 con 0x7ffac8138a50 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffaccb0f640 10 monclient(hunting): authenticate will time out at 1948.593181s 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac659c640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffac8138a50 msgr2=0x7ffac8138e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:01:18.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac659c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffac8138a50 0x7ffac8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 0x7ffac8139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffac8137960 0x7ffac8137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 0x7ffac8139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59194/0 (socket says 172.21.15.43:59194) 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): get_auth_request con 0x7ffac8137960 auth_method 0 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 1 -- 172.21.15.43:0/1012929549 learned_addr learned my addr 172.21.15.43:0/1012929549 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): get_auth_request con 0x7ffac80a5bb0 auth_method 0 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_done global_id 144308 payload 274 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 -- 172.21.15.43:0/1012929549 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffac8138a50 msgr2=0x7ffac8138e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffac8138a50 0x7ffac8138e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 -- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 msgr2=0x7ffac8139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 0x7ffac8139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 monclient: found mon.noname-b 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:01:18.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 -- 172.21.15.43:0/1012929549 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac80a0d40 con 0x7ffac8137960 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac5d9b640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 0x7ffac8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac6d9d640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffac8137960 0x7ffac8137dc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009dc0 tx=0x7ffac001c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29208701887d786b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac559a640 1 -- 172.21.15.43:0/1012929549 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffac00b2070 con 0x7ffac8137960 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.465+0000 7ffac559a640 10 monclient: dump: 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.478 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 10 monclient: _finish_auth 0 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 10 monclient: _check_auth_tickets 2024-04-19T01:01:18.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 1 -- 172.21.15.43:0/1012929549 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac00a5ab0 con 0x7ffac8137960 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 5 monclient: authenticate success, global_id 144308 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: get_monmap_and_config success 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 1 -- 172.21.15.43:0/1012929549 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffac00a5ce0 con 0x7ffac8137960 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 4 set_mon_vals no callback set 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac559a640 10 monclient: dump: 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.480 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.481 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: shutdown 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffac8137960 msgr2=0x7ffac8137dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffac8137960 0x7ffac8137dc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffac0009dc0 tx=0x7ffac001c840 comp rx=0 tx=0).stop 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/1012929549 shutdown_connections 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffac8137960 0x7ffac8137dc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac80a5bb0 0x7ffac8139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/1012929549 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ffac8138a50 0x7ffac8138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/1012929549 >> 172.21.15.43:0/1012929549 conn(0x7ffac80a7cb0 msgr2=0x7ffac80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/1012929549 shutdown_connections 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/1012929549 wait complete. 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: build_initial_monmap 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: monmap: 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:01:18.483 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 librados: starting msgr at 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 librados: starting objecter 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 Processor -- start 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- start start 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 librados: setting wanted keys 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 librados: calling monclient init 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: init 2024-04-19T01:01:18.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: _reopen_session rank -1 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: _add_conns ranks=[5,8,6,4,3,1,0,2,7] 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac80a5bb0 0x7ffac81d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): picked mon.noname-h con 0x7ffac80a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffac8137960 0x7ffac8136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): picked mon.noname-e con 0x7ffac8137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac8138a50 0x7ffac813c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): picked mon.noname-d con 0x7ffac8138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ffac80a2100 con 0x7ffac80a5bb0 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ffac80a1f80 con 0x7ffac8138a50 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ffac8142810 con 0x7ffac8137960 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): _renew_subs 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): authenticate will time out at 1948.598110s 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac8138a50 0x7ffac813c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffac8137960 0x7ffac8136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac8138a50 0x7ffac813c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:59198/0 (socket says 172.21.15.43:59198) 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 -- 172.21.15.43:0/4077931571 learned_addr learned my addr 172.21.15.43:0/4077931571 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): get_auth_request con 0x7ffac8138a50 auth_method 0 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): get_auth_request con 0x7ffac8137960 auth_method 0 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac659c640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac80a5bb0 0x7ffac81d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient(hunting): handle_auth_done global_id 144321 payload 824 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffac8137960 msgr2=0x7ffac8136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffac8137960 0x7ffac8136850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac80a5bb0 msgr2=0x7ffac81d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac80a5bb0 0x7ffac81d07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 monclient: found mon.noname-d 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac809fc90 con 0x7ffac8138a50 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac6d9d640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac8138a50 0x7ffac813c660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffac001c670 tx=0x7ffac00195e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 client.?.objecter ms_handle_connect 0x7ffac8138a50 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffac00b2050 con 0x7ffac8138a50 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:01:18.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: dump: 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.489 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 5 monclient: authenticate success, global_id 144321 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: _renew_subs 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac00be070 con 0x7ffac8138a50 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac8121f30 con 0x7ffac8138a50 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffac00a6e10 con 0x7ffac8138a50 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffac4d99640 4 set_mon_vals no callback set 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 client.144321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 10 monclient: dump: 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.491 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: _renew_subs 2024-04-19T01:01:18.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac809ee60 con 0x7ffac8138a50 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 librados: init done 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 0x7ffac813cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ffac8121d90 con 0x7ffac813cba0 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.469+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ffac0003bb0 con 0x7ffac8138a50 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaa8038e10 0x7ffaa803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(618..618 src has 501..618) ==== 6757+0+0 (secure 0 0 0) 0x7ffac00e4640 con 0x7ffac8138a50 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 client.144321.objecter ms_dispatch 0x7ffac8000ba0 osd_map(618..618 src has 501..618) 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 3 client.144321.objecter handle_osd_map got epochs [618,618] > 0 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 3 client.144321.objecter handle_osd_map decoding full epoch 618 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 0x7ffac813cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient: get_auth_request con 0x7ffac813cba0 auth_method 0 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 20 client.144321.objecter dump_active .. 0 homeless 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac5d9b640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaa8038e10 0x7ffaa803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac5d9b640 10 monclient: get_auth_request con 0x7ffaa8038e10 auth_method 0 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 10 monclient(hunting): handle_auth_done global_id 74930 payload 824 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffac806d140 con 0x7ffac813cba0 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac659c640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 0x7ffac813cf80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c4c0 tx=0x7ffab800c980 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 client.144321.objecter ms_handle_connect 0x7ffac813cba0 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 client.144321.objecter resend_mon_ops 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 client.144321.objecter ms_handle_connect 0x7ffaa8038e10 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffac5d9b640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaa8038e10 0x7ffaa803b2d0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7ffac813ddc0 tx=0x7ffabc009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffab8003bd0 con 0x7ffac813cba0 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 monclient: dump: 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.495 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffab8016070 con 0x7ffac813cba0 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 msgr2=0x7ffac813cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 0x7ffac813cf80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c4c0 tx=0x7ffab800c980 comp rx=0 tx=0).stop 2024-04-19T01:01:18.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 msgr2=0x7ffac813cf80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.473+0000 7ffab6ffd640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813cba0 0x7ffac813cf80 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c4c0 tx=0x7ffab800c980 comp rx=0 tx=0).stop 2024-04-19T01:01:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:01:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 0x7ffac80a2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:01:18.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7ffac813e2e0 con 0x7ffac813d4c0 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 0x7ffac80a2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient: get_auth_request con 0x7ffac813d4c0 auth_method 0 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 10 monclient(hunting): handle_auth_done global_id 74939 payload 824 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 1 -- 172.21.15.43:0/4077931571 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffac8076190 con 0x7ffac813d4c0 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac659c640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 0x7ffac80a2f70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffab8000eb0 tx=0x7ffab80059e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 client.144321.objecter ms_handle_connect 0x7ffac813d4c0 2024-04-19T01:01:18.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 client.144321.objecter resend_mon_ops 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffab8005830 con 0x7ffac813d4c0 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 monclient: dump: 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:01:18.500 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9252 (secure 0 0 0) 0x7ffab80053c0 con 0x7ffac813d4c0 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 msgr2=0x7ffac80a2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 0x7ffac80a2f70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffab8000eb0 tx=0x7ffab80059e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 msgr2=0x7ffac80a2f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffab6ffd640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7ffac813d4c0 0x7ffac80a2f70 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffab8000eb0 tx=0x7ffab80059e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 112, 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 458095, 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:01:18.502 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.503 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.504 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:01:18.505 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:01:18.506 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:01:18.507 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:01:18.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:01:18.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:01:18.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:01:18.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.514 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:01:18.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:01:18.521 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:01:18.523 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:01:18.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:01:18.525 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 10 librados: watch_flush enter 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 10 librados: watch_flush exit 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 20 client.144321.objecter shutdown clearing up homeless session... 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 10 client.144321.objecter successfully canceled tick 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaa8038e10 msgr2=0x7ffaa803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaa8038e10 0x7ffaa803b2d0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7ffac813ddc0 tx=0x7ffabc009240 comp rx=0 tx=0).stop 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 10 monclient: shutdown 2024-04-19T01:01:18.527 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac8138a50 msgr2=0x7ffac813c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 --2- 172.21.15.43:0/4077931571 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffac8138a50 0x7ffac813c660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffac001c670 tx=0x7ffac00195e0 comp rx=0 tx=0).stop 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac6d9d640 1 -- 172.21.15.43:0/4077931571 reap_dead start 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffac6d9d640 1 -- 172.21.15.43:0/4077931571 reap_dead start 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 shutdown_connections 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 >> 172.21.15.43:0/4077931571 conn(0x7ffac80a7cb0 msgr2=0x7ffac80a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 shutdown_connections 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 -- 172.21.15.43:0/4077931571 wait complete. 2024-04-19T01:01:18.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:01:18.481+0000 7ffaccb0f640 1 librados: shutdown 2024-04-19T01:01:18.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 7 2024-04-19T01:01:18.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T01:02:48.529 INFO:tasks.mon_thrash.mon_thrasher:killing mon.g 2024-04-19T01:02:48.530 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.g 2024-04-19T01:02:48.530 INFO:tasks.ceph.mon.g:Restarting daemon 2024-04-19T01:02:48.530 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i g 2024-04-19T01:02:48.534 INFO:tasks.ceph.mon.g:Started 2024-04-19T01:02:48.534 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-04-19T01:02:48.534 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-04-19T01:02:48.534 INFO:tasks.ceph.mon.i:Restarting daemon 2024-04-19T01:02:48.534 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-04-19T01:02:48.537 INFO:tasks.ceph.mon.i:Started 2024-04-19T01:02:48.537 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T01:02:48.537 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.668+0000 7f476d310640 10 monclient: get_monmap_and_config 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.668+0000 7f476d310640 10 monclient: build_initial_monmap 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.668+0000 7f476d310640 10 monclient: monmap: 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:48.673043+0000 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:48.673043+0000 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:48.670 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:48.671 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:48.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.668+0000 7f476d310640 1 Processor -- start 2024-04-19T01:02:48.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.668+0000 7f476d310640 1 -- start start 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.668+0000 7f476d310640 10 monclient: init 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient: _add_conns ranks=[4,10,5,2,8,6,7,3,1,0,9] 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 0x7f4768138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): picked mon.noname-c con 0x7f4768138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47680a5bb0 0x7f4768139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): picked mon.noname-f con 0x7f47680a5bb0 addr v2:172.21.15.161:3300/0 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 0x7f4768137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): picked mon.noname-h con 0x7f4768137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f47680a1020 con 0x7f47680a5bb0 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:48.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f47681398b0 con 0x7f4768138a50 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f47680a0780 con 0x7f4768137960 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient(hunting): authenticate will time out at 2038.800187s 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47680a5bb0 0x7f4768139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 0x7f4768138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 0x7f4768137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 0x7f4768137de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:33334/0 (socket says 172.21.15.43:33334) 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 -- 172.21.15.43:0/2579935513 learned_addr learned my addr 172.21.15.43:0/2579935513 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 0x7f4768138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:60566/0 (socket says 172.21.15.43:60566) 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): get_auth_request con 0x7f4768137960 auth_method 0 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47680a5bb0 0x7f4768139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:45570/0 (socket says 172.21.15.43:45570) 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:48.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): get_auth_request con 0x7f4768138a50 auth_method 0 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): get_auth_request con 0x7f47680a5bb0 auth_method 0 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:48.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f475ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient(hunting): handle_auth_done global_id 74948 payload 274 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 -- 172.21.15.43:0/2579935513 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 msgr2=0x7f4768138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 --2- 172.21.15.43:0/2579935513 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 0x7f4768138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 -- 172.21.15.43:0/2579935513 >> v2:172.21.15.161:3300/0 conn(0x7f47680a5bb0 msgr2=0x7f4768139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:48.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 --2- 172.21.15.43:0/2579935513 >> v2:172.21.15.161:3300/0 conn(0x7f47680a5bb0 0x7f4768139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient: _finish_hunting 0 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 monclient: found mon.noname-h 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 -- 172.21.15.43:0/2579935513 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680a0d40 con 0x7f4768137960 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f4766ffd640 1 --2- 172.21.15.43:0/2579935513 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 0x7f4768138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47677fe640 1 --2- 172.21.15.43:0/2579935513 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 0x7f4768137de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4760009e50 tx=0x7f476001c8c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=59feac0cd5a9c5ab server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 1 -- 172.21.15.43:0/2579935513 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47600090d0 con 0x7f4768137960 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: dump: 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:48.680 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:48.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: _finish_auth 0 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: _check_auth_tickets 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 1 -- 172.21.15.43:0/2579935513 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47600a5b70 con 0x7f4768137960 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 1 -- 172.21.15.43:0/2579935513 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47600a5d60 con 0x7f4768137960 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 5 monclient: authenticate success, global_id 74948 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 4 set_mon_vals no callback set 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f47667fc640 10 monclient: dump: 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:48.682 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:48.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 10 monclient: shutdown 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 -- 172.21.15.43:0/2579935513 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 msgr2=0x7f4768137de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- 172.21.15.43:0/2579935513 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 0x7f4768137de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4760009e50 tx=0x7f476001c8c0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 -- 172.21.15.43:0/2579935513 shutdown_connections 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- 172.21.15.43:0/2579935513 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4768137960 0x7f4768137de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- 172.21.15.43:0/2579935513 >> v2:172.21.15.161:3300/0 conn(0x7f47680a5bb0 0x7f4768139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 --2- 172.21.15.43:0/2579935513 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4768138a50 0x7f4768138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.672+0000 7f476d310640 1 -- 172.21.15.43:0/2579935513 >> 172.21.15.43:0/2579935513 conn(0x7f47680a7cb0 msgr2=0x7f47680a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- 172.21.15.43:0/2579935513 shutdown_connections 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- 172.21.15.43:0/2579935513 wait complete. 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: build_initial_monmap 2024-04-19T01:02:48.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: monmap: 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:48.685 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 librados: starting msgr at 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 librados: starting objecter 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 Processor -- start 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- start start 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 librados: setting wanted keys 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 librados: calling monclient init 2024-04-19T01:02:48.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: init 2024-04-19T01:02:48.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:48.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _add_conns ranks=[1,6,8,4,7,3,2,5,0] 2024-04-19T01:02:48.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47680a5bb0 0x7f47681d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): picked mon.noname-f con 0x7f47680a5bb0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 0x7f4768136810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): picked mon.noname-d con 0x7f4768137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4768138a50 0x7f476813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): picked mon.noname-e con 0x7f4768138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47680a2100 con 0x7f47680a5bb0 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:48.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f47680a1f80 con 0x7f4768137960 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4768142810 con 0x7f4768138a50 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient(hunting): authenticate will time out at 2038.805505s 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4768138a50 0x7f476813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 0x7f4768136810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4768138a50 0x7f476813c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:56594/0 (socket says 172.21.15.43:56594) 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 1 -- 172.21.15.43:0/4218299618 learned_addr learned my addr 172.21.15.43:0/4218299618 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): get_auth_request con 0x7f4768138a50 auth_method 0 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 0x7f4768136810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:33188/0 (socket says 172.21.15.43:33188) 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): get_auth_request con 0x7f4768137960 auth_method 0 2024-04-19T01:02:48.693 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47680a5bb0 0x7f47681d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): get_auth_request con 0x7f47680a5bb0 auth_method 0 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:48.694 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient(hunting): handle_auth_done global_id 144330 payload 824 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 -- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4768138a50 msgr2=0x7f476813c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4768138a50 0x7f476813c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 -- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47680a5bb0 msgr2=0x7f47681d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47680a5bb0 0x7f47681d07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 monclient: found mon.noname-d 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 -- 172.21.15.43:0/4218299618 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476813e330 con 0x7f4768137960 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475ffff640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 0x7f4768136810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f475000e840 tx=0x7f475000ed00 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:48.695 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 client.?.objecter ms_handle_connect 0x7f4768137960 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4750017070 con 0x7f4768137960 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: dump: 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.696 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:48.697 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 5 monclient: authenticate success, global_id 144330 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475000cdb0 con 0x7f4768137960 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47500138e0 con 0x7f4768137960 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f47657fa640 4 set_mon_vals no callback set 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 monclient: dump: 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:48.698 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:48.699 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _renew_subs 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476809ee60 con 0x7f4768137960 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 client.144330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _renew_subs 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f47681221d0 con 0x7f4768137960 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 librados: init done 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:48.700 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f475001a070 con 0x7f4768137960 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47681d1620 con 0x7f4768137960 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4748038e40 0x7f474803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(686..686 src has 501..686) ==== 6763+0+0 (secure 0 0 0) 0x7f475000f040 con 0x7f4768137960 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 10 client.144330.objecter ms_dispatch 0x7f4768000ba0 osd_map(686..686 src has 501..686) 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 3 client.144330.objecter handle_osd_map got epochs [686,686] > 0 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 3 client.144330.objecter handle_osd_map decoding full epoch 686 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f475f7fe640 20 client.144330.objecter dump_active .. 0 homeless 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4748038e40 0x7f474803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.676+0000 7f4766ffd640 10 monclient: get_auth_request con 0x7f4748038e40 auth_method 0 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.680+0000 7f475f7fe640 10 client.144330.objecter ms_handle_connect 0x7f4748038e40 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.680+0000 7f4766ffd640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4748038e40 0x7f474803b300 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7f4758009580 tx=0x7f4758009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:48.701 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.684+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4750011e20 con 0x7f4768137960 2024-04-19T01:02:48.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.684+0000 7f475f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:48.702 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.684+0000 7f475f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:48.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f476d310640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:02:48.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f476d310640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:48.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f476d310640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:48.912 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4768076190 con 0x7f4768137960 2024-04-19T01:02:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f475f7fe640 1 -- 172.21.15.43:0/4218299618 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3182 (secure 0 0 0) 0x7f4750012dc0 con 0x7f4768137960 2024-04-19T01:02:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f475f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:48.913 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.912+0000 7f475f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:48.913 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":112,"quorum":[0,1,2,4,5,6,8],"quorum_names":["a","f","b","c","h","d","e"],"quorum_leader_name":"b","quorum_age":93,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 10 librados: watch_flush enter 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 10 librados: watch_flush exit 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 20 client.144330.objecter shutdown clearing up homeless session... 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 10 client.144330.objecter successfully canceled tick 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4748038e40 msgr2=0x7f474803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4748038e40 0x7f474803b300 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7f4758009580 tx=0x7f4758009190 comp rx=0 tx=0).stop 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 10 monclient: shutdown 2024-04-19T01:02:48.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 msgr2=0x7f4768136810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 0x7f4768136810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f475000e840 tx=0x7f475000ed00 comp rx=0 tx=0).stop 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 shutdown_connections 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4748038e40 0x7f474803b300 unknown :-1 s=CLOSED pgs=3624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4768138a50 0x7f476813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4768137960 0x7f4768136810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 --2- 172.21.15.43:0/4218299618 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47680a5bb0 0x7f47681d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 >> 172.21.15.43:0/4218299618 conn(0x7f47680a7cb0 msgr2=0x7f476806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 shutdown_connections 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 -- 172.21.15.43:0/4218299618 wait complete. 2024-04-19T01:02:48.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:48.916+0000 7f476d310640 1 librados: shutdown 2024-04-19T01:02:51.933 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient: get_monmap_and_config 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient: build_initial_monmap 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient: monmap: 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:52.068494+0000 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:52.068494+0000 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:52.065 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:52.066 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 Processor -- start 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 -- start start 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient: init 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient: _add_conns ranks=[2,8,6,10,9,1,5,3,0,7,4] 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 0x7ffb00072b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): picked mon.noname-b con 0x7ffb00072750 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 0x7ffb000a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): picked mon.noname-e con 0x7ffb00073500 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb000a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): picked mon.noname-d con 0x7ffb00073070 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ffb0006e630 con 0x7ffb00072750 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ffb0006f210 con 0x7ffb00073070 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ffb0006f4f0 con 0x7ffb00073500 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb0704c640 10 monclient(hunting): authenticate will time out at 2042.195600s 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 0x7ffb00072b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffafffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 0x7ffb000a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 0x7ffb00072b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44424/0 (socket says 172.21.15.43:44424) 2024-04-19T01:02:52.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 1 -- 172.21.15.43:0/2792846363 learned_addr learned my addr 172.21.15.43:0/2792846363 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 10 monclient(hunting): get_auth_request con 0x7ffb00072750 auth_method 0 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffafffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 0x7ffb000a4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59798/0 (socket says 172.21.15.43:59798) 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb04de8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffafffff640 10 monclient(hunting): get_auth_request con 0x7ffb00073500 auth_method 0 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffafffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb055e9640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb000a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb055e9640 10 monclient(hunting): get_auth_request con 0x7ffb00073070 auth_method 0 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb055e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb055e9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:52.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.064+0000 7ffb055e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb055e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:52.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb055e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb055e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 10 monclient(hunting): handle_auth_done global_id 144317 payload 274 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 -- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 msgr2=0x7ffb000a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 0x7ffb000a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 -- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 msgr2=0x7ffb000a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb000a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 10 monclient: _finish_hunting 0 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 monclient: found mon.noname-b 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 -- 172.21.15.43:0/2792846363 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0006c450 con 0x7ffb00072750 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffafffff640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 0x7ffb000a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:52.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb04de8640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 0x7ffb00072b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00098a0 tx=0x7ffaf001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5010d27bc0cc0f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 1 -- 172.21.15.43:0/2792846363 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffaf00b2070 con 0x7ffb00072750 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: dump: 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:52.081 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: _finish_auth 0 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 1 -- 172.21.15.43:0/2792846363 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf00ab7f0 con 0x7ffb00072750 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 5 monclient: authenticate success, global_id 144317 2024-04-19T01:02:52.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 1 -- 172.21.15.43:0/2792846363 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffaf00abe50 con 0x7ffb00072750 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 4 set_mon_vals no callback set 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffaff7fe640 10 monclient: dump: 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:52.083 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 10 monclient: shutdown 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 -- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 msgr2=0x7ffb00072b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 0x7ffb00072b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00098a0 tx=0x7ffaf001c440 comp rx=0 tx=0).stop 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 -- 172.21.15.43:0/2792846363 shutdown_connections 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb000a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ffb00073500 0x7ffb000a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 --2- 172.21.15.43:0/2792846363 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ffb00072750 0x7ffb00072b30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 -- 172.21.15.43:0/2792846363 >> 172.21.15.43:0/2792846363 conn(0x7ffb000a6350 msgr2=0x7ffb0006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 -- 172.21.15.43:0/2792846363 shutdown_connections 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.068+0000 7ffb0704c640 1 -- 172.21.15.43:0/2792846363 wait complete. 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: build_initial_monmap 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: monmap: 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:52.085 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 librados: starting msgr at 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 librados: starting objecter 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 Processor -- start 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- start start 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 librados: setting wanted keys 2024-04-19T01:02:52.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 librados: calling monclient init 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: init 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _add_conns ranks=[1,6,4,0,7,8,5,3,2] 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb00072750 0x7ffb001d0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): picked mon.noname-f con 0x7ffb00072750 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb00135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): picked mon.noname-d con 0x7ffb00073070 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb00073500 0x7ffb0013de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): picked mon.noname-c con 0x7ffb00073500 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffb001429d0 con 0x7ffb00072750 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ffb00142850 con 0x7ffb00073500 2024-04-19T01:02:52.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ffb00142b50 con 0x7ffb00073070 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient(hunting): authenticate will time out at 2042.201031s 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb00135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb00135fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53484/0 (socket says 172.21.15.43:53484) 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb00073500 0x7ffb0013de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 -- 172.21.15.43:0/3308508973 learned_addr learned my addr 172.21.15.43:0/3308508973 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): get_auth_request con 0x7ffb00073070 auth_method 0 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): get_auth_request con 0x7ffb00073500 auth_method 0 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:52.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb055e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb04de8640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb00072750 0x7ffb001d0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient(hunting): handle_auth_done global_id 144348 payload 824 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 -- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb00073500 msgr2=0x7ffb0013de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb00073500 0x7ffb0013de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 -- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb00072750 msgr2=0x7ffb001d0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:52.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb00072750 0x7ffb001d0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient: _finish_hunting 0 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 monclient: found mon.noname-d 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 -- 172.21.15.43:0/3308508973 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0006e870 con 0x7ffb00073070 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb04de8640 10 monclient: get_auth_request con 0x7ffb00072750 auth_method 0 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb04de8640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb00072750 0x7ffb001d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafffff640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb00135fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400e900 tx=0x7ffaf400edc0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 client.?.objecter ms_handle_connect 0x7ffb00073070 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffaf4017070 con 0x7ffb00073070 2024-04-19T01:02:52.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: dump: 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:52.091 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: _finish_auth 0 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf4013470 con 0x7ffb00073070 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:52.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ffaf4013a90 con 0x7ffb00073070 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 5 monclient: authenticate success, global_id 144348 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _renew_subs 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0006e420 con 0x7ffb00073070 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 monclient: dump: 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:52.093 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafeffd640 4 set_mon_vals no callback set 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 client.144348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _renew_subs 2024-04-19T01:02:52.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0011ff30 con 0x7ffb00073070 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 librados: init done 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ffaf401a070 con 0x7ffb00073070 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb000693b0 con 0x7ffb00073070 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaec038e40 0x7ffaec03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(687..687 src has 501..687) ==== 6763+0+0 (secure 0 0 0) 0x7ffaf400f040 con 0x7ffb00073070 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 10 client.144348.objecter ms_dispatch 0x7ffb00000ba0 osd_map(687..687 src has 501..687) 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 3 client.144348.objecter handle_osd_map got epochs [687,687] > 0 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 3 client.144348.objecter handle_osd_map decoding full epoch 687 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffafd7fa640 20 client.144348.objecter dump_active .. 0 homeless 2024-04-19T01:02:52.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb04de8640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaec038e40 0x7ffaec03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:52.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.072+0000 7ffb04de8640 10 monclient: get_auth_request con 0x7ffaec038e40 auth_method 0 2024-04-19T01:02:52.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.076+0000 7ffb04de8640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaec038e40 0x7ffaec03b300 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00af000 tx=0x7ffaf00a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:52.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.076+0000 7ffafd7fa640 10 client.144348.objecter ms_handle_connect 0x7ffaec038e40 2024-04-19T01:02:52.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.076+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffaf4012dc0 con 0x7ffb00073070 2024-04-19T01:02:52.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.076+0000 7ffafd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:52.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.076+0000 7ffafd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:52.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffb0704c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:02:52.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffb0704c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:52.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffb0704c640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:02:52.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffb001218d0 con 0x7ffb00073070 2024-04-19T01:02:52.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffafd7fa640 1 -- 172.21.15.43:0/3308508973 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3182 (secure 0 0 0) 0x7ffaf4013c30 con 0x7ffb00073070 2024-04-19T01:02:52.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffafd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:52.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.304+0000 7ffafd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:52.306 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":112,"quorum":[0,1,2,4,5,6,8],"quorum_names":["a","f","b","c","h","d","e"],"quorum_leader_name":"b","quorum_age":96,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:02:52.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 10 librados: watch_flush enter 2024-04-19T01:02:52.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 10 librados: watch_flush exit 2024-04-19T01:02:52.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 20 client.144348.objecter shutdown clearing up homeless session... 2024-04-19T01:02:52.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 10 client.144348.objecter successfully canceled tick 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaec038e40 msgr2=0x7ffaec03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaec038e40 0x7ffaec03b300 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7ffaf00af000 tx=0x7ffaf00a7000 comp rx=0 tx=0).stop 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 10 monclient: shutdown 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 msgr2=0x7ffb00135fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb00135fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffaf400e900 tx=0x7ffaf400edc0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 shutdown_connections 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ffaec038e40 0x7ffaec03b300 unknown :-1 s=CLOSED pgs=3625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ffb00073500 0x7ffb0013de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ffb00073070 0x7ffb00135fb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 --2- 172.21.15.43:0/3308508973 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffb00072750 0x7ffb001d0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 >> 172.21.15.43:0/3308508973 conn(0x7ffb000a6350 msgr2=0x7ffb00137340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 shutdown_connections 2024-04-19T01:02:52.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 -- 172.21.15.43:0/3308508973 wait complete. 2024-04-19T01:02:52.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:52.308+0000 7ffb0704c640 1 librados: shutdown 2024-04-19T01:02:55.326 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:02:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient: get_monmap_and_config 2024-04-19T01:02:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient: build_initial_monmap 2024-04-19T01:02:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient: monmap: 2024-04-19T01:02:55.456 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:55.459760+0000 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:55.459760+0000 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:55.457 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:55.458 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:55.458 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 Processor -- start 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 -- start start 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient: init 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient: _add_conns ranks=[6,9,2,4,8,3,5,0,10,1,7] 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd16c138cd0 0x7fd16c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): picked mon.noname-d con 0x7fd16c138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd16c0a5b80 0x7fd16c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): picked mon.noname-a con 0x7fd16c0a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 0x7fd16c135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c1358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fd16c0a4410 con 0x7fd16c0a5b80 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd16c139b30 con 0x7fd16c1358b0 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd16c0a3dc0 con 0x7fd16c138cd0 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd172d20640 10 monclient(hunting): authenticate will time out at 2045.587140s 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 0x7fd16c135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd16c0a5b80 0x7fd16c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd16c138cd0 0x7fd16c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 0x7fd16c135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44428/0 (socket says 172.21.15.43:44428) 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fd16c0a5b80 0x7fd16c1395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46744/0 (socket says 172.21.15.43:46744) 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 -- 172.21.15.43:0/2713319733 learned_addr learned my addr 172.21.15.43:0/2713319733 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): get_auth_request con 0x7fd16c1358b0 auth_method 0 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): get_auth_request con 0x7fd16c0a5b80 auth_method 0 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): get_auth_request con 0x7fd16c138cd0 auth_method 0 2024-04-19T01:02:55.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd16bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient(hunting): handle_auth_done global_id 144326 payload 274 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 -- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd16c138cd0 msgr2=0x7fd16c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:55.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 --2- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd16c138cd0 0x7fd16c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 -- 172.21.15.43:0/2713319733 >> v2:172.21.15.43:3300/0 conn(0x7fd16c0a5b80 msgr2=0x7fd16c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 --2- 172.21.15.43:0/2713319733 >> v2:172.21.15.43:3300/0 conn(0x7fd16c0a5b80 0x7fd16c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient: _finish_hunting 0 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 monclient: found mon.noname-b 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd1712bd640 1 -- 172.21.15.43:0/2713319733 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c0a40a0 con 0x7fd16c1358b0 2024-04-19T01:02:55.464 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.456+0000 7fd170abc640 1 --2- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd16c138cd0 0x7fd16c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd1712bd640 1 --2- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 0x7fd16c135d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd164009870 tx=0x7fd16401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53b5564f7d96fdf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:55.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 1 -- 172.21.15.43:0/2713319733 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1640b0070 con 0x7fd16c1358b0 2024-04-19T01:02:55.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:55.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:02:55.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: dump: 2024-04-19T01:02:55.473 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:55.474 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 1 -- 172.21.15.43:0/2713319733 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1640aa810 con 0x7fd16c1358b0 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 1 -- 172.21.15.43:0/2713319733 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1640a9300 con 0x7fd16c1358b0 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 5 monclient: authenticate success, global_id 144326 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:55.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 4 set_mon_vals no callback set 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd16b7fe640 10 monclient: dump: 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:55.476 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 10 monclient: shutdown 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 -- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 msgr2=0x7fd16c135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 --2- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 0x7fd16c135d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd164009870 tx=0x7fd16401c3c0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 -- 172.21.15.43:0/2713319733 shutdown_connections 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 --2- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd16c1358b0 0x7fd16c135d40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 --2- 172.21.15.43:0/2713319733 >> v2:172.21.15.43:3300/0 conn(0x7fd16c0a5b80 0x7fd16c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 --2- 172.21.15.43:0/2713319733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd16c138cd0 0x7fd16c1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 -- 172.21.15.43:0/2713319733 >> 172.21.15.43:0/2713319733 conn(0x7fd16c0ab2f0 msgr2=0x7fd16c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 -- 172.21.15.43:0/2713319733 shutdown_connections 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 1 -- 172.21.15.43:0/2713319733 wait complete. 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 10 monclient: build_initial_monmap 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.460+0000 7fd172d20640 10 monclient: monmap: 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:55.478 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 librados: starting msgr at 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 librados: starting objecter 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 Processor -- start 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- start start 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 librados: setting wanted keys 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 librados: calling monclient init 2024-04-19T01:02:55.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: init 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _add_conns ranks=[7,4,0,3,5,8,6,1,2] 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd16c0a5b80 0x7fd16c133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): picked mon.noname-i con 0x7fd16c0a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 0x7fd16c133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): picked mon.noname-c con 0x7fd16c1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd16c138cd0 0x7fd16c134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): picked mon.noname-a con 0x7fd16c138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd16c0a2010 con 0x7fd16c138cd0 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd16c0a1e90 con 0x7fd16c1358b0 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:55.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fd16c142aa0 con 0x7fd16c0a5b80 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient(hunting): authenticate will time out at 2045.592780s 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 0x7fd16c133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd16c138cd0 0x7fd16c134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 0x7fd16c133ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:57404/0 (socket says 172.21.15.43:57404) 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 1 -- 172.21.15.43:0/278786419 learned_addr learned my addr 172.21.15.43:0/278786419 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): get_auth_request con 0x7fd16c1358b0 auth_method 0 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:55.481 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): get_auth_request con 0x7fd16c138cd0 auth_method 0 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd16c0a5b80 0x7fd16c133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 10 monclient(hunting): get_auth_request con 0x7fd16c0a5b80 auth_method 0 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:55.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient(hunting): handle_auth_done global_id 65583 payload 824 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 -- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd16c0a5b80 msgr2=0x7fd16c133980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd16c0a5b80 0x7fd16c133980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 -- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 msgr2=0x7fd16c133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 0x7fd16c133ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient: _finish_hunting 0 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 monclient: found mon.noname-a 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 -- 172.21.15.43:0/278786419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16c135490 con 0x7fd16c138cd0 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16bfff640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 0x7fd16c133ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1712bd640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd16c138cd0 0x7fd16c134400 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd1640ab000 tx=0x7fd1640191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 client.?.objecter ms_handle_connect 0x7fd16c138cd0 2024-04-19T01:02:55.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1640b9070 con 0x7fd16c138cd0 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: dump: 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:55.484 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: _finish_auth 0 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: _check_auth_tickets 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 5 monclient: authenticate success, global_id 65583 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1640044b0 con 0x7fd16c138cd0 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _renew_subs 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd16c13fbd0 con 0x7fd16c138cd0 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:55.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1640b0040 con 0x7fd16c138cd0 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd16affd640 4 set_mon_vals no callback set 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 client.65583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 monclient: dump: 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:55.486 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _renew_subs 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd16c13ba30 con 0x7fd16c138cd0 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 librados: init done 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd1640aabe0 con 0x7fd16c138cd0 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:55.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16c068240 con 0x7fd16c138cd0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd14c038e00 0x7fd14c03b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(687..687 src has 501..687) ==== 6763+0+0 (secure 0 0 0) 0x7fd1640e2700 con 0x7fd16c138cd0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 10 client.65583.objecter ms_dispatch 0x7fd16c000ba0 osd_map(687..687 src has 501..687) 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 3 client.65583.objecter handle_osd_map got epochs [687,687] > 0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 3 client.65583.objecter handle_osd_map decoding full epoch 687 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd1697fa640 20 client.65583.objecter dump_active .. 0 homeless 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd14c038e00 0x7fd14c03b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.464+0000 7fd170abc640 10 monclient: get_auth_request con 0x7fd14c038e00 auth_method 0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.468+0000 7fd170abc640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd14c038e00 0x7fd14c03b2c0 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7fd15c0099d0 tx=0x7fd15c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:55.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.468+0000 7fd1697fa640 10 client.65583.objecter ms_handle_connect 0x7fd14c038e00 2024-04-19T01:02:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.472+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd164016130 con 0x7fd16c138cd0 2024-04-19T01:02:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.472+0000 7fd1697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:55.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.472+0000 7fd1697fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.700+0000 7fd172d20640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:02:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.700+0000 7fd172d20640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.700+0000 7fd172d20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:02:55.703 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.700+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd16c0693b0 con 0x7fd16c138cd0 2024-04-19T01:02:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd1697fa640 1 -- 172.21.15.43:0/278786419 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3183 (secure 0 0 0) 0x7fd1640bc360 con 0x7fd16c138cd0 2024-04-19T01:02:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd1697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:55.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd1697fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:55.704 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":112,"quorum":[0,1,2,4,5,6,8],"quorum_names":["a","f","b","c","h","d","e"],"quorum_leader_name":"b","quorum_age":100,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 10 librados: watch_flush enter 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 10 librados: watch_flush exit 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 20 client.65583.objecter shutdown clearing up homeless session... 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 10 client.65583.objecter successfully canceled tick 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd14c038e00 msgr2=0x7fd14c03b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd14c038e00 0x7fd14c03b2c0 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7fd15c0099d0 tx=0x7fd15c008040 comp rx=0 tx=0).stop 2024-04-19T01:02:55.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 10 monclient: shutdown 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd16c138cd0 msgr2=0x7fd16c134400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd16c138cd0 0x7fd16c134400 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd1640ab000 tx=0x7fd1640191c0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 shutdown_connections 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd14c038e00 0x7fd14c03b2c0 unknown :-1 s=CLOSED pgs=3626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd16c138cd0 0x7fd16c134400 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd16c1358b0 0x7fd16c133ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 --2- 172.21.15.43:0/278786419 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fd16c0a5b80 0x7fd16c133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 >> 172.21.15.43:0/278786419 conn(0x7fd16c0ab2f0 msgr2=0x7fd16c0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 shutdown_connections 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 -- 172.21.15.43:0/278786419 wait complete. 2024-04-19T01:02:55.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:55.704+0000 7fd172d20640 1 librados: shutdown 2024-04-19T01:02:58.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient: get_monmap_and_config 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient: build_initial_monmap 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient: monmap: 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:58.859068+0000 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:58.859068+0000 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:58.856 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:58.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 Processor -- start 2024-04-19T01:02:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 -- start start 2024-04-19T01:02:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient: init 2024-04-19T01:02:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient: _add_conns ranks=[4,7,6,10,5,9,3,2,1,8,0] 2024-04-19T01:02:58.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): picked mon.noname-c con 0x7f47a8138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): picked mon.noname-i con 0x7f47a80a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): picked mon.noname-d con 0x7f47a81358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:58.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f47a80a4410 con 0x7f47a8138cd0 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f47a8139b30 con 0x7f47a81358b0 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f47a80a3dc0 con 0x7f47a80a5b80 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47ae48f640 10 monclient(hunting): authenticate will time out at 2048.986421s 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:57420/0 (socket says 172.21.15.43:57420) 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 -- 172.21.15.43:0/3741233815 learned_addr learned my addr 172.21.15.43:0/3741233815 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 10 monclient(hunting): get_auth_request con 0x7f47a8138cd0 auth_method 0 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:58.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 10 monclient(hunting): get_auth_request con 0x7f47a81358b0 auth_method 0 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 msgr2=0x7f47a8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 msgr2=0x7f47a8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 msgr2=0x7f47a81390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 msgr2=0x7f47a81390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47aca2c640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a7fff640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:02:58.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 10 monclient(hunting): get_auth_request con 0x7f47a80a5b80 auth_method 0 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 msgr2=0x7f47a81395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 msgr2=0x7f47a81395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:02:58.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:58.856+0000 7f47a77fe640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:02:59.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): get_auth_request con 0x7f47a8138cd0 auth_method 0 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): get_auth_request con 0x7f47a81358b0 auth_method 0 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a77fe640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a77fe640 10 monclient(hunting): get_auth_request con 0x7f47a80a5b80 auth_method 0 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47aca2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient(hunting): handle_auth_done global_id 134509 payload 274 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 msgr2=0x7f47a81395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 msgr2=0x7f47a8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 monclient: found mon.noname-c 2024-04-19T01:02:59.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.060+0000 7f47a7fff640 1 -- 172.21.15.43:0/3741233815 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a80a40a0 con 0x7f47a8138cd0 2024-04-19T01:02:59.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a7fff640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f47980098d0 tx=0x7f479801c670 comp rx=0 tx=0).ready entity=mon.4 client_cookie=d3cc78965ad348bf server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 1 -- 172.21.15.43:0/3741233815 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f47980ad070 con 0x7f47a8138cd0 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: dump: 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.064 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 1 -- 172.21.15.43:0/3741233815 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47980a98e0 con 0x7f47a8138cd0 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 1 -- 172.21.15.43:0/3741233815 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f479800f040 con 0x7f47a8138cd0 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 5 monclient: authenticate success, global_id 134509 2024-04-19T01:02:59.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 4 set_mon_vals no callback set 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47a6ffd640 10 monclient: dump: 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.066 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.067 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.067 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: shutdown 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 msgr2=0x7f47a81390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f47980098d0 tx=0x7f479801c670 comp rx=0 tx=0).stop 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- 172.21.15.43:0/3741233815 shutdown_connections 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a81358b0 0x7f47a8135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f47a80a5b80 0x7f47a81395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- 172.21.15.43:0/3741233815 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a8138cd0 0x7f47a81390b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- 172.21.15.43:0/3741233815 >> 172.21.15.43:0/3741233815 conn(0x7f47a80ab2f0 msgr2=0x7f47a80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- 172.21.15.43:0/3741233815 shutdown_connections 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- 172.21.15.43:0/3741233815 wait complete. 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: monmap: 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.073 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 librados: starting msgr at 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 librados: starting objecter 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 Processor -- start 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- start start 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 librados: setting wanted keys 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 librados: calling monclient init 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: init 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient: _add_conns ranks=[2,4,6,8,5,0,1,7,3] 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47a80a5b80 0x7f47a8134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): picked mon.noname-b con 0x7f47a80a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a81358b0 0x7f47a8135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): picked mon.noname-c con 0x7f47a81358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a8138cd0 0x7f47a81d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): picked mon.noname-d con 0x7f47a8138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f47a80a2010 con 0x7f47a80a5b80 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f47a80a1e90 con 0x7f47a81358b0 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f47a8142aa0 con 0x7f47a8138cd0 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.064+0000 7f47ae48f640 10 monclient(hunting): authenticate will time out at 2049.195924s 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47a80a5b80 0x7f47a8134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47a80a5b80 0x7f47a8134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44438/0 (socket says 172.21.15.43:44438) 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a81358b0 0x7f47a8135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 1 -- 172.21.15.43:0/311784040 learned_addr learned my addr 172.21.15.43:0/311784040 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): get_auth_request con 0x7f47a80a5b80 auth_method 0 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): get_auth_request con 0x7f47a81358b0 auth_method 0 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient(hunting): handle_auth_done global_id 134518 payload 824 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 -- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a8138cd0 msgr2=0x7f47a81d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a8138cd0 0x7f47a81d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 -- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47a80a5b80 msgr2=0x7f47a8134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47a80a5b80 0x7f47a8134bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 monclient: found mon.noname-c 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 -- 172.21.15.43:0/311784040 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a80a0310 con 0x7f47a81358b0 2024-04-19T01:02:59.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a77fe640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a81358b0 0x7f47a8135130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f479c00f580 tx=0x7f479c00fa40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 client.?.objecter ms_handle_connect 0x7f47a81358b0 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f479c019020 con 0x7f47a81358b0 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: dump: 2024-04-19T01:02:59.078 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.079 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f479c014ae0 con 0x7f47a81358b0 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f479c014c90 con 0x7f47a81358b0 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 10 monclient: dump: 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.081 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a5ffb640 4 set_mon_vals no callback set 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 5 monclient: authenticate success, global_id 134518 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 10 monclient: _renew_subs 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a80a3040 con 0x7f47a81358b0 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 10 client.134518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 10 monclient: _renew_subs 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a809e290 con 0x7f47a81358b0 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47ae48f640 1 librados: init done 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.068+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f479c01c020 con 0x7f47a81358b0 2024-04-19T01:02:59.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4794038e10 0x7f479403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47ae48f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(689..689 src has 501..689) ==== 6729+0+0 (secure 0 0 0) 0x7f479c049d40 con 0x7f47a81358b0 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 10 client.134518.objecter ms_dispatch 0x7f47a8000ba0 osd_map(689..689 src has 501..689) 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 3 client.134518.objecter handle_osd_map got epochs [689,689] > 0 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47ae48f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 3 client.134518.objecter handle_osd_map decoding full epoch 689 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47ae48f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a80693b0 con 0x7f47a81358b0 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 20 client.134518.objecter dump_active .. 0 homeless 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a7fff640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4794038e10 0x7f479403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a7fff640 10 monclient: get_auth_request con 0x7f4794038e10 auth_method 0 2024-04-19T01:02:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a7fff640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4794038e10 0x7f479403b2d0 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f47980098d0 tx=0x7f47980b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.072+0000 7f47a4ff9640 10 client.134518.objecter ms_handle_connect 0x7f4794038e10 2024-04-19T01:02:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.076+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f479c023020 con 0x7f47a81358b0 2024-04-19T01:02:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.076+0000 7f47a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.076+0000 7f47a4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:59.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:02:59.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:59.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47a8135670 con 0x7f47a81358b0 2024-04-19T01:02:59.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47a4ff9640 1 -- 172.21.15.43:0/311784040 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f479c013c00 con 0x7f47a81358b0 2024-04-19T01:02:59.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:02:59.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47a4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:59.313 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":118,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 librados: watch_flush enter 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 librados: watch_flush exit 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 20 client.134518.objecter shutdown clearing up homeless session... 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 client.134518.objecter successfully canceled tick 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4794038e10 msgr2=0x7f479403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4794038e10 0x7f479403b2d0 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f47980098d0 tx=0x7f47980b5040 comp rx=0 tx=0).stop 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 10 monclient: shutdown 2024-04-19T01:02:59.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a81358b0 msgr2=0x7f47a8135130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a81358b0 0x7f47a8135130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f479c00f580 tx=0x7f479c00fa40 comp rx=0 tx=0).stop 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 shutdown_connections 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.312+0000 7f47ae48f640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4794038e10 0x7f479403b2d0 unknown :-1 s=CLOSED pgs=3627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f47a8138cd0 0x7f47a81d0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f47a81358b0 0x7f47a8135130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 --2- 172.21.15.43:0/311784040 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f47a80a5b80 0x7f47a8134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 >> 172.21.15.43:0/311784040 conn(0x7f47a80ab2f0 msgr2=0x7f47a81368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 shutdown_connections 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 -- 172.21.15.43:0/311784040 wait complete. 2024-04-19T01:02:59.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.316+0000 7f47ae48f640 1 librados: shutdown 2024-04-19T01:02:59.332 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T01:02:59.332 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T01:02:59.332 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:02:59.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient: get_monmap_and_config 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient: monmap: 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:59.463104+0000 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:59.463104+0000 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.460 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:59.461 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 1 Processor -- start 2024-04-19T01:02:59.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 1 -- start start 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient: init 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient: _add_conns ranks=[1,0,10,3,6,9,2,7,8,5,4] 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f9b8c0a2b80 addr v1:172.21.15.161:6789/0 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b8c136f40 addr v1:172.21.15.43:6789/0 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b8c137320 0x7f9b8c13f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient(hunting): picked mon.noname-f con 0x7f9b8c137320 addr v2:172.21.15.161:3300/0 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b8c073690 con 0x7f9b8c136f40 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b8c0730d0 con 0x7f9b8c0a2b80 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9b8c0a07a0 con 0x7f9b8c137320 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 10 monclient(hunting): authenticate will time out at 2049.590217s 2024-04-19T01:02:59.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b90862640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9b8c136f40 0x7f9b8c13bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:54866/0 (socket says 172.21.15.43:54866) 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b90862640 1 -- 172.21.15.43:0/994143420 learned_addr learned my addr 172.21.15.43:0/994143420 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 --2- 172.21.15.43:0/994143420 >> v2:172.21.15.161:3300/0 conn(0x7f9b8c137320 0x7f9b8c13f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017391108 0 0) 0x7f9b7c001120 con 0x7f9b8c136f40 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): _negotiate 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): my global_id is 65592 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b78003530 con 0x7f9b8c136f40 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): get_auth_request con 0x7f9b8c137320 auth_method 0 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 575541893 0 0) 0x7f9b78003530 con 0x7f9b8c136f40 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 >> v2:172.21.15.161:3300/0 conn(0x7f9b8c137320 msgr2=0x7f9b8c13f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 --2- 172.21.15.43:0/994143420 >> v2:172.21.15.161:3300/0 conn(0x7f9b8c137320 0x7f9b8c13f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 >> v1:172.21.15.161:6789/0 conn(0x7f9b8c0a2b80 legacy=0x7f9b8c0a2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8c0733b0 con 0x7f9b8c136f40 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f9b7c001b40 con 0x7f9b8c136f40 2024-04-19T01:02:59.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: dump: 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.470 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 >> v1:172.21.15.43:6789/0 conn(0x7f9b8c136f40 legacy=0x7f9b8c13bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient: _add_conns ranks=[1,3,2,7,4,8,0,5,6] 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b78004210 0x7f9b78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): picked mon.f con 0x7f9b78004210 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b78004da0 0x7f9b78005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): picked mon.g con 0x7f9b78004da0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b7800c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): picked mon.b con 0x7f9b78009c40 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7800c980 con 0x7f9b78004210 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7800cb30 con 0x7f9b78009c40 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7800cce0 con 0x7f9b78004da0 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b83fff640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b7800c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): get_auth_request con 0x7f9b78009c40 auth_method 0 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91063640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b78004210 0x7f9b78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91063640 10 monclient(hunting): get_auth_request con 0x7f9b78004210 auth_method 0 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91063640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91063640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91063640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b90862640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b78004da0 0x7f9b78005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient(hunting): handle_auth_done global_id 65592 payload 293 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 -- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b78004da0 msgr2=0x7f9b78005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b78004da0 0x7f9b78005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 -- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b78004210 msgr2=0x7f9b78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b78004210 0x7f9b78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 monclient: found mon.b 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:02:59.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 1 -- 172.21.15.43:0/994143420 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b78003530 con 0x7f9b78009c40 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b91864640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 5 monclient: authenticate success, global_id 65592 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b932c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b90862640 10 monclient: get_auth_request con 0x7f9b78004da0 auth_method 0 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.460+0000 7f9b90862640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f9b78004da0 0x7f9b78005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b91864640 1 -- 172.21.15.43:0/994143420 reap_dead start 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b91864640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b7800c0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009ab0 tx=0x7f9b8401ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60a61d80711eb261 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b840096b0 con 0x7f9b78009c40 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: dump: 2024-04-19T01:02:59.474 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.475 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b840a59a0 con 0x7f9b78009c40 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 4 set_mon_vals no callback set 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 1 -- 172.21.15.43:0/994143420 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b840ad480 con 0x7f9b78009c40 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b83fff640 10 monclient: dump: 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.476 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 10 monclient: shutdown 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 -- 172.21.15.43:0/994143420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 msgr2=0x7f9b7800c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b7800c0d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f9b84009ab0 tx=0x7f9b8401ca50 comp rx=0 tx=0).stop 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 -- 172.21.15.43:0/994143420 shutdown_connections 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 --2- 172.21.15.43:0/994143420 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b7800c0d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 -- 172.21.15.43:0/994143420 >> 172.21.15.43:0/994143420 conn(0x7f9b8c0a9900 msgr2=0x7f9b8c0a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 -- 172.21.15.43:0/994143420 shutdown_connections 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 -- 172.21.15.43:0/994143420 wait complete. 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 10 monclient: monmap: 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:59.478 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 librados: starting msgr at 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 librados: starting objecter 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 Processor -- start 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 -- start start 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 librados: setting wanted keys 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 1 librados: calling monclient init 2024-04-19T01:02:59.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.464+0000 7f9b932c7640 10 monclient: init 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient: _add_conns ranks=[2,1,0,6,5,8,3,7,4] 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b8c1ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): picked mon.noname-b con 0x7f9b78009c40 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b8c09eff0 0x7f9b8c09f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): picked mon.noname-f con 0x7f9b8c09eff0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c129d20 0x7f9b8c12a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): picked mon.noname-a con 0x7f9b8c129d20 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c0a32f0 con 0x7f9b8c129d20 2024-04-19T01:02:59.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c0a3170 con 0x7f9b8c09eff0 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f9b8c0a3470 con 0x7f9b78009c40 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): authenticate will time out at 2049.596354s 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b8c1ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c129d20 0x7f9b8c12a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b8c1ce350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44448/0 (socket says 172.21.15.43:44448) 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c129d20 0x7f9b8c12a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:46760/0 (socket says 172.21.15.43:46760) 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 -- 172.21.15.43:0/809009364 learned_addr learned my addr 172.21.15.43:0/809009364 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): get_auth_request con 0x7f9b78009c40 auth_method 0 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): get_auth_request con 0x7f9b8c129d20 auth_method 0 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b8c09eff0 0x7f9b8c09f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): get_auth_request con 0x7f9b8c09eff0 auth_method 0 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient(hunting): handle_auth_done global_id 144344 payload 824 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b8c09eff0 msgr2=0x7f9b8c09f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b8c09eff0 0x7f9b8c09f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c129d20 msgr2=0x7f9b8c12a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c129d20 0x7f9b8c12a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 monclient: found mon.noname-b 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b8c12b760 con 0x7f9b78009c40 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c129d20 0x7f9b8c12a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91063640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b8c1ce350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800e840 tx=0x7f9b8800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 client.?.objecter ms_handle_connect 0x7f9b78009c40 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:59.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b880050d0 con 0x7f9b78009c40 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: dump: 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.490 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b88018070 con 0x7f9b78009c40 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 5 monclient: authenticate success, global_id 144344 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient: _renew_subs 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b8c09ff90 con 0x7f9b78009c40 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b88005700 con 0x7f9b78009c40 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 client.144344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b82ffd640 4 set_mon_vals no callback set 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: dump: 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.492 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient: _renew_subs 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b8c09dac0 con 0x7f9b78009c40 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 librados: init done 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 0x7f9b8c0a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c06d140 con 0x7f9b8c12a6c0 2024-04-19T01:02:59.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9b88005290 con 0x7f9b78009c40 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 0x7f9b8c0a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient: get_auth_request con 0x7f9b8c12a6c0 auth_method 0 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b70038e40 0x7f9b7003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(689..689 src has 501..689) ==== 6729+0+0 (secure 0 0 0) 0x7f9b88010040 con 0x7f9b78009c40 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 client.144344.objecter ms_dispatch 0x7f9b8c000ba0 osd_map(689..689 src has 501..689) 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 3 client.144344.objecter handle_osd_map got epochs [689,689] > 0 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 3 client.144344.objecter handle_osd_map decoding full epoch 689 2024-04-19T01:02:59.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 10 monclient(hunting): handle_auth_done global_id 65610 payload 824 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b8c069630 con 0x7f9b8c12a6c0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 20 client.144344.objecter dump_active .. 0 homeless 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b70038e40 0x7f9b7003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 client.144344.objecter ms_handle_connect 0x7f9b8c12a6c0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 client.144344.objecter resend_mon_ops 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 10 monclient: get_auth_request con 0x7f9b70038e40 auth_method 0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b90862640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 0x7f9b8c0a0970 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00f770 tx=0x7f9b7c00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b7c017020 con 0x7f9b8c12a6c0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: dump: 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.495 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9b7c00d6b0 con 0x7f9b8c12a6c0 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.496 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 msgr2=0x7f9b8c0a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 0x7f9b8c0a0970 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00f770 tx=0x7f9b7c00fc30 comp rx=0 tx=0).stop 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 msgr2=0x7f9b8c0a0970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c12a6c0 0x7f9b8c0a0970 secure :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00f770 tx=0x7f9b7c00fc30 comp rx=0 tx=0).stop 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b91864640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b70038e40 0x7f9b7003b300 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800ccb0 tx=0x7f9b840046d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.468+0000 7f9b81ffb640 10 client.144344.objecter ms_handle_connect 0x7f9b70038e40 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 0x7f9b8c0a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8c09c2c0 con 0x7f9b8c0a0eb0 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 0x7f9b8c0a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient: get_auth_request con 0x7f9b8c0a0eb0 auth_method 0 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 10 monclient(hunting): handle_auth_done global_id 65619 payload 824 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 1 -- 172.21.15.43:0/809009364 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9b8c0764b0 con 0x7f9b8c0a0eb0 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 client.144344.objecter ms_handle_connect 0x7f9b8c0a0eb0 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b90862640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 0x7f9b8c0a12c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00f770 tx=0x7f9b8c073580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 client.144344.objecter resend_mon_ops 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9b7c017040 con 0x7f9b8c0a0eb0 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 monclient: dump: 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.498 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7f9b7c00de00 con 0x7f9b8c0a0eb0 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 msgr2=0x7f9b8c0a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 0x7f9b8c0a12c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00f770 tx=0x7f9b8c073580 comp rx=0 tx=0).stop 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 msgr2=0x7f9b8c0a12c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b81ffb640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9b8c0a0eb0 0x7f9b8c0a12c0 secure :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00f770 tx=0x7f9b8c073580 comp rx=0 tx=0).stop 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:02:59.500 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 705748, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.501 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.502 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.503 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:02:59.504 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.505 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.506 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:02:59.507 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.508 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.509 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.510 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.511 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.513 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:02:59.514 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:02:59.516 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.517 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:02:59.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:02:59.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.522 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:02:59.522 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 10 librados: watch_flush enter 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 10 librados: watch_flush exit 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 20 client.144344.objecter shutdown clearing up homeless session... 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 10 client.144344.objecter successfully canceled tick 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b70038e40 msgr2=0x7f9b7003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9b70038e40 0x7f9b7003b300 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f9b7800ccb0 tx=0x7f9b840046d0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 10 monclient: shutdown 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 msgr2=0x7f9b8c1ce350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 --2- 172.21.15.43:0/809009364 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f9b78009c40 0x7f9b8c1ce350 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800e840 tx=0x7f9b8800ed00 comp rx=0 tx=0).stop 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b91864640 1 -- 172.21.15.43:0/809009364 reap_dead start 2024-04-19T01:02:59.524 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b91864640 1 -- 172.21.15.43:0/809009364 reap_dead start 2024-04-19T01:02:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 shutdown_connections 2024-04-19T01:02:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 >> 172.21.15.43:0/809009364 conn(0x7f9b8c0a9900 msgr2=0x7f9b8c131490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 shutdown_connections 2024-04-19T01:02:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 -- 172.21.15.43:0/809009364 wait complete. 2024-04-19T01:02:59.525 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.476+0000 7f9b932c7640 1 librados: shutdown 2024-04-19T01:02:59.525 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:02:59.525 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T01:02:59.525 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.640+0000 7fbc38057640 10 monclient: get_monmap_and_config 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.640+0000 7fbc38057640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient: monmap: 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:59.646427+0000 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:59.646427+0000 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.643 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:59.644 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 Processor -- start 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 -- start start 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient: init 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient: _add_conns ranks=[5,3,10,8,4,0,9,1,7,2,6] 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbc30138cd0 0x7fbc301390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): picked mon.noname-h con 0x7fbc30138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): picked mon.noname-g con 0x7fbc300a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc301358b0 0x7fbc30135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): picked mon.noname-f con 0x7fbc301358b0 addr v2:172.21.15.161:3300/0 2024-04-19T01:02:59.647 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbc300a4410 con 0x7fbc301358b0 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30139b30 con 0x7fbc300a5b80 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fbc300a3dc0 con 0x7fbc30138cd0 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc38057640 10 monclient(hunting): authenticate will time out at 2049.773554s 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbc30138cd0 0x7fbc301390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc301358b0 0x7fbc30135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbc30138cd0 0x7fbc301390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:58178/0 (socket says 172.21.15.43:58178) 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 1 -- 172.21.15.43:0/3243788787 learned_addr learned my addr 172.21.15.43:0/3243788787 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc301358b0 0x7fbc30135d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:45796/0 (socket says 172.21.15.43:45796) 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): get_auth_request con 0x7fbc30138cd0 auth_method 0 2024-04-19T01:02:59.648 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:53950/0 (socket says 172.21.15.43:53950) 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): get_auth_request con 0x7fbc301358b0 auth_method 0 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 10 monclient(hunting): get_auth_request con 0x7fbc300a5b80 auth_method 0 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.649 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc35df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 1 -- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 msgr2=0x7fbc301395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 1 -- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 msgr2=0x7fbc301395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc355f2640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient(hunting): handle_auth_done global_id 144379 payload 274 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 -- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbc30138cd0 msgr2=0x7fbc301390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbc30138cd0 0x7fbc301390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 -- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 msgr2=0x7fbc301395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 monclient: found mon.noname-f 2024-04-19T01:02:59.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T01:02:59.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.644+0000 7fbc365f4640 1 -- 172.21.15.43:0/3243788787 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc300a40a0 con 0x7fbc301358b0 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc365f4640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc301358b0 0x7fbc30135d60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009d90 tx=0x7fbc2801c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e99f6a749b9224f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 1 -- 172.21.15.43:0/3243788787 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc280b2070 con 0x7fbc301358b0 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: dump: 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.658 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.659 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 1 -- 172.21.15.43:0/3243788787 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc280a5990 con 0x7fbc301358b0 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 1 -- 172.21.15.43:0/3243788787 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc280a5bc0 con 0x7fbc301358b0 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc34df1640 10 monclient: dump: 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.660 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 5 monclient: authenticate success, global_id 144379 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 4 set_mon_vals no callback set 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 10 monclient: shutdown 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 -- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc301358b0 msgr2=0x7fbc30135d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc301358b0 0x7fbc30135d60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009d90 tx=0x7fbc2801c7d0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 -- 172.21.15.43:0/3243788787 shutdown_connections 2024-04-19T01:02:59.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc301358b0 0x7fbc30135d60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc300a5b80 0x7fbc301395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 --2- 172.21.15.43:0/3243788787 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbc30138cd0 0x7fbc301390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 -- 172.21.15.43:0/3243788787 >> 172.21.15.43:0/3243788787 conn(0x7fbc300ab2f0 msgr2=0x7fbc300ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 -- 172.21.15.43:0/3243788787 shutdown_connections 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 -- 172.21.15.43:0/3243788787 wait complete. 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 10 monclient: monmap: 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:59.662 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 librados: starting msgr at 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 librados: starting objecter 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 Processor -- start 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 -- start start 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 librados: setting wanted keys 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 1 librados: calling monclient init 2024-04-19T01:02:59.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.648+0000 7fbc38057640 10 monclient: init 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient: _add_conns ranks=[8,6,3,2,7,0,1,4,5] 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbc300a5b80 0x7fbc30134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): picked mon.noname-e con 0x7fbc300a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbc301358b0 0x7fbc301351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): picked mon.noname-d con 0x7fbc301358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc30138cd0 0x7fbc301d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): picked mon.noname-g con 0x7fbc30138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbc300a2010 con 0x7fbc30138cd0 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fbc300a1e90 con 0x7fbc301358b0 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fbc30142aa0 con 0x7fbc300a5b80 2024-04-19T01:02:59.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): authenticate will time out at 2049.780293s 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbc300a5b80 0x7fbc30134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbc301358b0 0x7fbc301351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbc300a5b80 0x7fbc30134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59802/0 (socket says 172.21.15.43:59802) 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 -- 172.21.15.43:0/2195931676 learned_addr learned my addr 172.21.15.43:0/2195931676 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): get_auth_request con 0x7fbc300a5b80 auth_method 0 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): get_auth_request con 0x7fbc301358b0 auth_method 0 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc30138cd0 0x7fbc301d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 10 monclient(hunting): get_auth_request con 0x7fbc30138cd0 auth_method 0 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient(hunting): handle_auth_done global_id 144350 payload 824 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbc301358b0 msgr2=0x7fbc301351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbc301358b0 0x7fbc301351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc30138cd0 msgr2=0x7fbc301d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbc30138cd0 0x7fbc301d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 monclient: found mon.noname-e 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3013d290 con 0x7fbc300a5b80 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbc301358b0 0x7fbc301351a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc35df3640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbc300a5b80 0x7fbc30134c60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00e890 tx=0x7fbc2c00ed50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 client.?.objecter ms_handle_connect 0x7fbc300a5b80 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc2c017070 con 0x7fbc300a5b80 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: dump: 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.667 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.668 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc2c00cdb0 con 0x7fbc300a5b80 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 5 monclient: authenticate success, global_id 144350 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient: _renew_subs 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc301356e0 con 0x7fbc300a5b80 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc2c0138e0 con 0x7fbc300a5b80 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: dump: 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.669 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc277fe640 4 set_mon_vals no callback set 2024-04-19T01:02:59.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 client.144350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient: _renew_subs 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc300a3590 con 0x7fbc300a5b80 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 librados: init done 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 0x7fbc301d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3013aa90 con 0x7fbc301d11f0 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 0x7fbc301d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbc2c01a070 con 0x7fbc300a5b80 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient: get_auth_request con 0x7fbc301d11f0 auth_method 0 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbc14038e10 0x7fbc1403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 10 monclient(hunting): handle_auth_done global_id 144353 payload 824 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc300a3040 con 0x7fbc301d11f0 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(689..689 src has 501..689) ==== 6729+0+0 (secure 0 0 0) 0x7fbc2c00f040 con 0x7fbc300a5b80 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 client.144350.objecter ms_dispatch 0x7fbc30000ba0 osd_map(689..689 src has 501..689) 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 3 client.144350.objecter handle_osd_map got epochs [689,689] > 0 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 3 client.144350.objecter handle_osd_map decoding full epoch 689 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 20 client.144350.objecter dump_active .. 0 homeless 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 client.144350.objecter ms_handle_connect 0x7fbc301d11f0 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 client.144350.objecter resend_mon_ops 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc355f2640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 0x7fbc301d15d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbc3013fc90 tx=0x7fbc2000e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc20018020 con 0x7fbc301d11f0 2024-04-19T01:02:59.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbc14038e10 0x7fbc1403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: dump: 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.673 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 10 monclient: get_auth_request con 0x7fbc14038e10 auth_method 0 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbc20004030 con 0x7fbc301d11f0 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 msgr2=0x7fbc301d15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 0x7fbc301d15d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbc3013fc90 tx=0x7fbc2000e5a0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 msgr2=0x7fbc301d15d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc301d11f0 0x7fbc301d15d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc365f4640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbc14038e10 0x7fbc1403b2d0 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fbc280af000 tx=0x7fbc280a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.652+0000 7fbc267fc640 10 client.144350.objecter ms_handle_connect 0x7fbc14038e10 2024-04-19T01:02:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:02:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 0x7fbc3009de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbc3006d140 con 0x7fbc3009da80 2024-04-19T01:02:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 0x7fbc3009de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient: get_auth_request con 0x7fbc3009da80 auth_method 0 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 10 monclient(hunting): handle_auth_done global_id 144362 payload 824 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 1 -- 172.21.15.43:0/2195931676 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc301d1b10 con 0x7fbc3009da80 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc355f2640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 0x7fbc3009de90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbc20019000 tx=0x7fbc200046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 client.144350.objecter ms_handle_connect 0x7fbc3009da80 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 client.144350.objecter resend_mon_ops 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbc20018040 con 0x7fbc3009da80 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:02:59.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 monclient: dump: 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.678 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9276 (secure 0 0 0) 0x7fbc20020e50 con 0x7fbc3009da80 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 msgr2=0x7fbc3009de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 0x7fbc3009de90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbc20019000 tx=0x7fbc200046d0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 msgr2=0x7fbc3009de90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:02:59.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc267fc640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbc3009da80 0x7fbc3009de90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.680 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 144290, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:02:59.681 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.682 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.683 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.683 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.683 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.684 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:02:59.685 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.686 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:02:59.687 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.688 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.689 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.690 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.691 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.692 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.693 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.694 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:02:59.695 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.696 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.697 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:02:59.700 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.701 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:02:59.702 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 10 librados: watch_flush enter 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 10 librados: watch_flush exit 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 20 client.144350.objecter shutdown clearing up homeless session... 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 10 client.144350.objecter successfully canceled tick 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbc14038e10 msgr2=0x7fbc1403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbc14038e10 0x7fbc1403b2d0 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fbc280af000 tx=0x7fbc280a7070 comp rx=0 tx=0).stop 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 10 monclient: shutdown 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbc300a5b80 msgr2=0x7fbc30134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.704 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 --2- 172.21.15.43:0/2195931676 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbc300a5b80 0x7fbc30134c60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00e890 tx=0x7fbc2c00ed50 comp rx=0 tx=0).stop 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc365f4640 1 -- 172.21.15.43:0/2195931676 reap_dead start 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc365f4640 1 -- 172.21.15.43:0/2195931676 reap_dead start 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 shutdown_connections 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 >> 172.21.15.43:0/2195931676 conn(0x7fbc300ab2f0 msgr2=0x7fbc301368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 shutdown_connections 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 -- 172.21.15.43:0/2195931676 wait complete. 2024-04-19T01:02:59.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.660+0000 7fbc38057640 1 librados: shutdown 2024-04-19T01:02:59.705 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:02:59.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T01:02:59.706 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:02:59.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.820+0000 7f81baa1e640 10 monclient: get_monmap_and_config 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.820+0000 7f81baa1e640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.820+0000 7f81baa1e640 10 monclient: monmap: 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:02:59.826164+0000 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:02:59.826164+0000 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.823 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:02:59.824 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 Processor -- start 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 -- start start 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient: init 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient: _add_conns ranks=[2,3,8,0,10,9,4,6,5,7,1] 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b4138cd0 0x7f81b41390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): picked mon.noname-b con 0x7f81b4138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:59.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b40a5b80 0x7f81b41395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): picked mon.noname-g con 0x7f81b40a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 0x7f81b4135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): picked mon.noname-e con 0x7f81b41358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f81b40a4410 con 0x7f81b4138cd0 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f81b4139b30 con 0x7f81b40a5b80 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f81b40a3dc0 con 0x7f81b41358b0 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81baa1e640 10 monclient(hunting): authenticate will time out at 2049.953245s 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 0x7f81b4135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b4138cd0 0x7f81b41390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 0x7f81b4135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59810/0 (socket says 172.21.15.43:59810) 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 -- 172.21.15.43:0/267986250 learned_addr learned my addr 172.21.15.43:0/267986250 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): get_auth_request con 0x7f81b41358b0 auth_method 0 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): get_auth_request con 0x7f81b4138cd0 auth_method 0 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b37fe640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b40a5b80 0x7f81b41395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient(hunting): handle_auth_done global_id 144359 payload 274 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 -- 172.21.15.43:0/267986250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b40a5b80 msgr2=0x7f81b41395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b40a5b80 0x7f81b41395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 -- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b4138cd0 msgr2=0x7f81b41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b4138cd0 0x7f81b41390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 monclient: found mon.noname-e 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 -- 172.21.15.43:0/267986250 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b40a40a0 con 0x7f81b41358b0 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b37fe640 10 monclient: get_auth_request con 0x7f81b40a5b80 auth_method 0 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b3fff640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b4138cd0 0x7f81b41390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b37fe640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b40a5b80 0x7f81b41395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.824+0000 7f81b8fbb640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 0x7f81b4135d30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009e80 tx=0x7f81ac01c8b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=14fa4689a271176a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 1 -- 172.21.15.43:0/267986250 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81ac0090d0 con 0x7f81b41358b0 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: dump: 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.832 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 1 -- 172.21.15.43:0/267986250 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ac0a5a10 con 0x7f81b41358b0 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 1 -- 172.21.15.43:0/267986250 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81ac0a5c00 con 0x7f81b41358b0 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 5 monclient: authenticate success, global_id 144359 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: get_monmap_and_config success 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 4 set_mon_vals no callback set 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b2ffd640 10 monclient: dump: 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.834 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.835 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.835 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: shutdown 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 msgr2=0x7f81b4135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 0x7f81b4135d30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f81ac009e80 tx=0x7f81ac01c8b0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- 172.21.15.43:0/267986250 shutdown_connections 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f81b41358b0 0x7f81b4135d30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b40a5b80 0x7f81b41395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- 172.21.15.43:0/267986250 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b4138cd0 0x7f81b41390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- 172.21.15.43:0/267986250 >> 172.21.15.43:0/267986250 conn(0x7f81b40ab2f0 msgr2=0x7f81b40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- 172.21.15.43:0/267986250 shutdown_connections 2024-04-19T01:02:59.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- 172.21.15.43:0/267986250 wait complete. 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: build_initial_monmap 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: monmap: 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:02:59.844 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 librados: starting msgr at 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 librados: starting objecter 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 Processor -- start 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- start start 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 librados: setting wanted keys 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 librados: calling monclient init 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: init 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: _reopen_session rank -1 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: _add_conns ranks=[2,4,3,5,6,0,8,1,7] 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b40a5b80 0x7f81b41d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): picked mon.noname-b con 0x7f81b40a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b41358b0 0x7f81b40a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): picked mon.noname-c con 0x7f81b41358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b4138cd0 0x7f81b40a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): picked mon.noname-g con 0x7f81b4138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f81b40a2010 con 0x7f81b40a5b80 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f81b40a1e90 con 0x7f81b4138cd0 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f81b4142aa0 con 0x7f81b41358b0 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): _renew_subs 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient(hunting): authenticate will time out at 2049.958421s 2024-04-19T01:02:59.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b40a5b80 0x7f81b41d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b41358b0 0x7f81b40a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b41358b0 0x7f81b40a0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:57464/0 (socket says 172.21.15.43:57464) 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 -- 172.21.15.43:0/4240508881 learned_addr learned my addr 172.21.15.43:0/4240508881 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): get_auth_request con 0x7f81b41358b0 auth_method 0 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): get_auth_request con 0x7f81b40a5b80 auth_method 0 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b8fbb640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b4138cd0 0x7f81b40a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b8fbb640 10 monclient(hunting): get_auth_request con 0x7f81b4138cd0 auth_method 0 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b8fbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b8fbb640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b8fbb640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient(hunting): handle_auth_done global_id 134527 payload 824 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b4138cd0 msgr2=0x7f81b40a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f81b4138cd0 0x7f81b40a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b40a5b80 msgr2=0x7f81b41d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b40a5b80 0x7f81b41d0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:02:59.848 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 monclient: found mon.noname-c 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81b4135000 con 0x7f81b41358b0 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b3fff640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f81b40a5b80 0x7f81b41d0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b37fe640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b41358b0 0x7f81b40a0a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c820 tx=0x7f81a800cce0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 client.?.objecter ms_handle_connect 0x7f81b41358b0 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a8015070 con 0x7f81b41358b0 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: dump: 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.849 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: _finish_auth 0 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:02:59.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a8007ca0 con 0x7f81b41358b0 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 5 monclient: authenticate success, global_id 134527 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a8007e50 con 0x7f81b41358b0 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b0ff9640 10 monclient: dump: 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.851 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81b27fc640 4 set_mon_vals no callback set 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: _renew_subs 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.828+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81b40a14c0 con 0x7f81b41358b0 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 10 client.134527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 10 monclient: _renew_subs 2024-04-19T01:02:59.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f81b40a3590 con 0x7f81b41358b0 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 1 librados: init done 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 0x7f81b4133560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f81b406d140 con 0x7f81b409bd60 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 0x7f81b4133560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient: get_auth_request con 0x7f81b409bd60 auth_method 0 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f81a8018070 con 0x7f81b41358b0 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.853 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 10 monclient(hunting): handle_auth_done global_id 134536 payload 824 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f81a0038fb0 0x7f81a003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81b409cf90 con 0x7f81b409bd60 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(689..689 src has 501..689) ==== 6729+0+0 (secure 0 0 0) 0x7f81a80495a0 con 0x7f81b41358b0 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 client.134527.objecter ms_dispatch 0x7f81b4000ba0 osd_map(689..689 src has 501..689) 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 3 client.134527.objecter handle_osd_map got epochs [689,689] > 0 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 3 client.134527.objecter handle_osd_map decoding full epoch 689 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 20 client.134527.objecter dump_active .. 0 homeless 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 client.134527.objecter ms_handle_connect 0x7f81b409bd60 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 client.134527.objecter resend_mon_ops 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b3fff640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 0x7f81b4133560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f81a4011be0 tx=0x7f81a400f610 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a4019070 con 0x7f81b409bd60 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 monclient: dump: 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.855 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b8fbb640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f81a0038fb0 0x7f81a003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b8fbb640 10 monclient: get_auth_request con 0x7f81a0038fb0 auth_method 0 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f81a4004030 con 0x7f81b409bd60 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 msgr2=0x7f81b4133560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 0x7f81b4133560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f81a4011be0 tx=0x7f81a400f610 comp rx=0 tx=0).stop 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 msgr2=0x7f81b4133560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b409bd60 0x7f81b4133560 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f81a4011be0 tx=0x7f81a400f610 comp rx=0 tx=0).stop 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b0ff9640 10 client.134527.objecter ms_handle_connect 0x7f81a0038fb0 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.832+0000 7f81b8fbb640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f81a0038fb0 0x7f81a003b470 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0ae000 tx=0x7f81ac0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:02:59.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 0x7f81b4133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f81b41343f0 con 0x7f81b4133aa0 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 0x7f81b4133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient: get_auth_request con 0x7f81b4133aa0 auth_method 0 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 10 monclient(hunting): handle_auth_done global_id 134545 payload 824 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 1 -- 172.21.15.43:0/4240508881 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f81b40693b0 con 0x7f81b4133aa0 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b3fff640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 0x7f81b4133eb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f81a401b000 tx=0x7f81a40046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 client.134527.objecter ms_handle_connect 0x7f81b4133aa0 2024-04-19T01:02:59.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 client.134527.objecter resend_mon_ops 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f81a4019040 con 0x7f81b4133aa0 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 monclient: dump: 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:02:59.858 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9614 (secure 0 0 0) 0x7f81a401e450 con 0x7f81b4133aa0 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 msgr2=0x7f81b4133eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 0x7f81b4133eb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f81a401b000 tx=0x7f81a40046d0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 msgr2=0x7f81b4133eb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:02:59.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b0ff9640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b4133aa0 0x7f81b4133eb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 284544, 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:02:59.860 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.861 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.862 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:02:59.863 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:02:59.864 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:02:59.865 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.866 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:02:59.867 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.868 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.869 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.870 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.871 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.872 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.873 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:02:59.874 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.875 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.877 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.878 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:02:59.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.880 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:02:59.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:02:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:02:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:02:59.882 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:02:59.882 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:02:59.883 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 10 librados: watch_flush enter 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 10 librados: watch_flush exit 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 20 client.134527.objecter shutdown clearing up homeless session... 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 10 client.134527.objecter successfully canceled tick 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f81a0038fb0 msgr2=0x7f81a003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f81a0038fb0 0x7f81a003b470 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7f81ac0ae000 tx=0x7f81ac0a7070 comp rx=0 tx=0).stop 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 10 monclient: shutdown 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b41358b0 msgr2=0x7f81b40a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 --2- 172.21.15.43:0/4240508881 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f81b41358b0 0x7f81b40a0a40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f81a800c820 tx=0x7f81a800cce0 comp rx=0 tx=0).stop 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b8fbb640 1 -- 172.21.15.43:0/4240508881 reap_dead start 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81b8fbb640 1 -- 172.21.15.43:0/4240508881 reap_dead start 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 shutdown_connections 2024-04-19T01:02:59.884 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 >> 172.21.15.43:0/4240508881 conn(0x7f81b40ab2f0 msgr2=0x7f81b40a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:02:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 shutdown_connections 2024-04-19T01:02:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 -- 172.21.15.43:0/4240508881 wait complete. 2024-04-19T01:02:59.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:02:59.840+0000 7f81baa1e640 1 librados: shutdown 2024-04-19T01:02:59.885 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:02:59.885 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T01:02:59.885 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T01:03:00.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient: get_monmap_and_config 2024-04-19T01:03:00.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient: monmap: 2024-04-19T01:03:00.011 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.011 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.011 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:00.010727+0000 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:00.010727+0000 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.012 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 Processor -- start 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 -- start start 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient: init 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient: _add_conns ranks=[5,1,7,9,10,3,6,8,4,0,2] 2024-04-19T01:03:00.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 0x7fdca81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): picked mon.noname-h con 0x7fdca8138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:00.018 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fdca80a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:00.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca8135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): picked mon.noname-i con 0x7fdca81395f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:00.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca80a3dc0 con 0x7fdca80a5b80 2024-04-19T01:03:00.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdca80a4410 con 0x7fdca8138cd0 2024-04-19T01:03:00.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdca80a40a0 con 0x7fdca81395f0 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdcadb64640 10 monclient(hunting): authenticate will time out at 2050.138149s 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 0x7fdca81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca8135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 0x7fdca81390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:58194/0 (socket says 172.21.15.43:58194) 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 -- 172.21.15.43:0/3045426956 learned_addr learned my addr 172.21.15.43:0/3045426956 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): get_auth_request con 0x7fdca8138cd0 auth_method 0 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca6ffd640 1 --1- 172.21.15.43:0/3045426956 >> v1:172.21.15.161:6789/0 conn(0x7fdca80a5b80 0x7fdca81358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:39066/0 (socket says 172.21.15.43:39066) 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 10 monclient(hunting): get_auth_request con 0x7fdca81395f0 auth_method 0 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 1 -- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 msgr2=0x7fdca8135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 1 -- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 msgr2=0x7fdca8135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca8135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca7fff640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca8135fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca67fc640 1 -- 172.21.15.43:0/3045426956 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253453823 0 0) 0x7fdca80a3dc0 con 0x7fdca80a5b80 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca67fc640 10 monclient(hunting): _negotiate 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca67fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca67fc640 10 monclient(hunting): my global_id is 144388 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca67fc640 1 -- 172.21.15.43:0/3045426956 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc88003530 con 0x7fdca80a5b80 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient(hunting): handle_auth_done global_id 74966 payload 274 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 -- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 msgr2=0x7fdca8135fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca8135fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 -- 172.21.15.43:0/3045426956 >> v1:172.21.15.161:6789/0 conn(0x7fdca80a5b80 legacy=0x7fdca81358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 monclient: found mon.noname-h 2024-04-19T01:03:00.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:03:00.023 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.008+0000 7fdca77fe640 1 -- 172.21.15.43:0/3045426956 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca8139a80 con 0x7fdca8138cd0 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca77fe640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 0x7fdca81390b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdc900098a0 tx=0x7fdc9001c3e0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=9c7132452574849b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 1 -- 172.21.15.43:0/3045426956 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc900b1070 con 0x7fdca8138cd0 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: dump: 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.030 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 1 -- 172.21.15.43:0/3045426956 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc900ab830 con 0x7fdca8138cd0 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 1 -- 172.21.15.43:0/3045426956 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc900aa320 con 0x7fdca8138cd0 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 5 monclient: authenticate success, global_id 74966 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 4 set_mon_vals no callback set 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdca67fc640 10 monclient: dump: 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.032 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.033 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 10 monclient: shutdown 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 -- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 msgr2=0x7fdca81390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 0x7fdca81390b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdc900098a0 tx=0x7fdc9001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 -- 172.21.15.43:0/3045426956 shutdown_connections 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca8135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 --2- 172.21.15.43:0/3045426956 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdca8138cd0 0x7fdca81390b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 -- 172.21.15.43:0/3045426956 >> 172.21.15.43:0/3045426956 conn(0x7fdca80ab2f0 msgr2=0x7fdca80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 -- 172.21.15.43:0/3045426956 shutdown_connections 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 -- 172.21.15.43:0/3045426956 wait complete. 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 10 monclient: monmap: 2024-04-19T01:03:00.034 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.035 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.012+0000 7fdcadb64640 1 librados: starting msgr at 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 librados: starting objecter 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 Processor -- start 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- start start 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 librados: setting wanted keys 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 librados: calling monclient init 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: init 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: _add_conns ranks=[2,8,7,0,4,3,6,5,1] 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdca80a5b80 0x7fdca80a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): picked mon.noname-b con 0x7fdca80a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:00.036 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdca8138cd0 0x7fdca812e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): picked mon.noname-e con 0x7fdca8138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca812ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): picked mon.noname-i con 0x7fdca81395f0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdca8136500 con 0x7fdca80a5b80 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fdca80a20a0 con 0x7fdca81395f0 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdca8136680 con 0x7fdca8138cd0 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): authenticate will time out at 2050.144740s 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdca80a5b80 0x7fdca80a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdca8138cd0 0x7fdca812e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdca80a5b80 0x7fdca80a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44476/0 (socket says 172.21.15.43:44476) 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 -- 172.21.15.43:0/2191683210 learned_addr learned my addr 172.21.15.43:0/2191683210 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): get_auth_request con 0x7fdca8138cd0 auth_method 0 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): get_auth_request con 0x7fdca80a5b80 auth_method 0 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca812ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 10 monclient(hunting): get_auth_request con 0x7fdca81395f0 auth_method 0 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 msgr2=0x7fdca812ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 msgr2=0x7fdca812ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca812ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient(hunting): handle_auth_done global_id 144389 payload 824 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca812ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:00.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdca8138cd0 msgr2=0x7fdca812e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdca8138cd0 0x7fdca812e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 msgr2=0x7fdca812ee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fdca81395f0 0x7fdca812ee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 monclient: found mon.noname-b 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca80a03f0 con 0x7fdca80a5b80 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdca8138cd0 0x7fdca812e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca77fe640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdca80a5b80 0x7fdca80a0bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc900af040 tx=0x7fdc900191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 client.?.objecter ms_handle_connect 0x7fdca80a5b80 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc900044b0 con 0x7fdca80a5b80 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: dump: 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.041 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc900bc030 con 0x7fdca80a5b80 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc900b8df0 con 0x7fdca80a5b80 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:00.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca5ffb640 4 set_mon_vals no callback set 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 monclient: dump: 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.043 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.044 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.044 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.044 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.044 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.044 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.045 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.045 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.045 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.045 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.045 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 5 monclient: authenticate success, global_id 144389 2024-04-19T01:03:00.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: _renew_subs 2024-04-19T01:03:00.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:03:00.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca809e5b0 con 0x7fdca80a5b80 2024-04-19T01:03:00.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 client.144389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:00.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: _renew_subs 2024-04-19T01:03:00.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:03:00.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca80a3040 con 0x7fdca80a5b80 2024-04-19T01:03:00.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 librados: init done 2024-04-19T01:03:00.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdc900b8980 con 0x7fdca80a5b80 2024-04-19T01:03:00.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 0x7fdca809c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fdca806d140 con 0x7fdca812f350 2024-04-19T01:03:00.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc8c038e10 0x7fdc8c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(689..689 src has 501..689) ==== 6729+0+0 (secure 0 0 0) 0x7fdc900eb5f0 con 0x7fdca80a5b80 2024-04-19T01:03:00.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 10 client.144389.objecter ms_dispatch 0x7fdca8000ba0 osd_map(689..689 src has 501..689) 2024-04-19T01:03:00.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 3 client.144389.objecter handle_osd_map got epochs [689,689] > 0 2024-04-19T01:03:00.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 3 client.144389.objecter handle_osd_map decoding full epoch 689 2024-04-19T01:03:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 0x7fdca809c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient: get_auth_request con 0x7fdca812f350 auth_method 0 2024-04-19T01:03:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdc9ffff640 20 client.144389.objecter dump_active .. 0 homeless 2024-04-19T01:03:00.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc8c038e10 0x7fdc8c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca7fff640 10 monclient: get_auth_request con 0x7fdc8c038e10 auth_method 0 2024-04-19T01:03:00.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_done global_id 144384 payload 824 2024-04-19T01:03:00.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.016+0000 7fdca6ffd640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdca812f730 con 0x7fdca812f350 2024-04-19T01:03:00.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 client.144389.objecter ms_handle_connect 0x7fdca812f350 2024-04-19T01:03:00.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 client.144389.objecter resend_mon_ops 2024-04-19T01:03:00.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdca6ffd640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 0x7fdca809c4b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc98007fd0 tx=0x7fdc98004eb0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc9800fdc0 con 0x7fdca812f350 2024-04-19T01:03:00.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:00.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 monclient: dump: 2024-04-19T01:03:00.052 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.052 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.052 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.052 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.053 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 client.144389.objecter ms_handle_connect 0x7fdc8c038e10 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdca7fff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc8c038e10 0x7fdc8c03b2d0 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009610 tx=0x7fdca000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdc98018070 con 0x7fdca812f350 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 msgr2=0x7fdca809c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 0x7fdca809c4b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc98007fd0 tx=0x7fdc98004eb0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 msgr2=0x7fdca809c4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:00.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.020+0000 7fdc9ffff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca812f350 0x7fdca809c4b0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc98007fd0 tx=0x7fdc98004eb0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:00.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 0x7fdca809ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fdca8068240 con 0x7fdca809c9f0 2024-04-19T01:03:00.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 0x7fdca809ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient: get_auth_request con 0x7fdca809c9f0 auth_method 0 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 10 monclient(hunting): handle_auth_done global_id 144393 payload 824 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 1 -- 172.21.15.43:0/2191683210 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdca80693b0 con 0x7fdca809c9f0 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca6ffd640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 0x7fdca809ce00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc98014000 tx=0x7fdc98001010 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 client.144389.objecter ms_handle_connect 0x7fdca809c9f0 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 client.144389.objecter resend_mon_ops 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdc98013a70 con 0x7fdca809c9f0 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 monclient: dump: 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.058 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.059 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.059 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.059 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.059 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.059 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7fdc98013600 con 0x7fdca809c9f0 2024-04-19T01:03:00.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:00.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:00.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 msgr2=0x7fdca809ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 0x7fdca809ce00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc98014000 tx=0x7fdc98001010 comp rx=0 tx=0).stop 2024-04-19T01:03:00.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 msgr2=0x7fdca809ce00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:00.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdc9ffff640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdca809c9f0 0x7fdca809ce00 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdc98014000 tx=0x7fdc98001010 comp rx=0 tx=0).stop 2024-04-19T01:03:00.061 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:00.061 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 144662, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:00.062 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.063 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.064 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:00.065 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.066 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:00.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:00.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:00.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.077 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:00.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:00.080 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:00.083 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:00.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.084 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.084 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:00.084 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 10 librados: watch_flush enter 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 10 librados: watch_flush exit 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 20 client.144389.objecter shutdown clearing up homeless session... 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 10 client.144389.objecter successfully canceled tick 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc8c038e10 msgr2=0x7fdc8c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdc8c038e10 0x7fdc8c03b2d0 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7fdca0009610 tx=0x7fdca000b040 comp rx=0 tx=0).stop 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 10 monclient: shutdown 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdca80a5b80 msgr2=0x7fdca80a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 --2- 172.21.15.43:0/2191683210 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdca80a5b80 0x7fdca80a0bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc900af040 tx=0x7fdc900191e0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca7fff640 1 -- 172.21.15.43:0/2191683210 reap_dead start 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdca7fff640 1 -- 172.21.15.43:0/2191683210 reap_dead start 2024-04-19T01:03:00.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 shutdown_connections 2024-04-19T01:03:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 >> 172.21.15.43:0/2191683210 conn(0x7fdca80ab2f0 msgr2=0x7fdca8131a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 shutdown_connections 2024-04-19T01:03:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 -- 172.21.15.43:0/2191683210 wait complete. 2024-04-19T01:03:00.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.032+0000 7fdcadb64640 1 librados: shutdown 2024-04-19T01:03:00.087 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:00.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T01:03:00.087 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:03:00.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.200+0000 7f171cdfc640 10 monclient: get_monmap_and_config 2024-04-19T01:03:00.202 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.200+0000 7f171cdfc640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.200+0000 7f171cdfc640 10 monclient: monmap: 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:00.205819+0000 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:00.205819+0000 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.203 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:00.204 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:00.204 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.206 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 Processor -- start 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 -- start start 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient: init 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient: _add_conns ranks=[1,5,7,10,9,3,4,8,2,0,6] 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f1718133b20 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1718133f00 0x7f171813bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): picked mon.noname-h con 0x7f1718133f00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 0x7f171813e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): picked mon.noname-i con 0x7f1718134300 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17180a0f00 con 0x7f1718133b20 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f171809cf40 con 0x7f1718133f00 2024-04-19T01:03:00.207 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f171809c610 con 0x7f1718134300 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171cdfc640 10 monclient(hunting): authenticate will time out at 2050.332934s 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1718133f00 0x7f171813bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 0x7f171813e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1718133b20 0x7f171809d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:39074/0 (socket says 172.21.15.43:39074) 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f171659c640 1 -- 172.21.15.43:0/341854441 learned_addr learned my addr 172.21.15.43:0/341854441 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): get_auth_request con 0x7f1718133f00 auth_method 0 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 10 monclient(hunting): get_auth_request con 0x7f1718134300 auth_method 0 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 1 -- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 msgr2=0x7f171813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 1 -- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 msgr2=0x7f171813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 0x7f171813e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1716d9d640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 0x7f171813e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1715d9b640 1 -- 172.21.15.43:0/341854441 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1843622255 0 0) 0x7f17180a0f00 con 0x7f1718133b20 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1715d9b640 10 monclient(hunting): _negotiate 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1715d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1715d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1715d9b640 10 monclient(hunting): my global_id is 144397 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f1715d9b640 1 -- 172.21.15.43:0/341854441 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1704003530 con 0x7f1718133b20 2024-04-19T01:03:00.209 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient(hunting): handle_auth_done global_id 74975 payload 274 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 1 -- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 msgr2=0x7f171813e100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 0x7f171813e100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 1 -- 172.21.15.43:0/341854441 >> v1:172.21.15.161:6789/0 conn(0x7f1718133b20 legacy=0x7f171809d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 1 monclient: found mon.noname-h 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.204+0000 7f170ffff640 1 -- 172.21.15.43:0/341854441 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171809c8f0 con 0x7f1718133f00 2024-04-19T01:03:00.210 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1718133f00 0x7f171813bbb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f17080097f0 tx=0x7f170801c430 comp rx=0 tx=0).ready entity=mon.5 client_cookie=4a7260c70c97217c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 1 -- 172.21.15.43:0/341854441 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17080b1070 con 0x7f1718133f00 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: dump: 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.216 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 5 monclient: authenticate success, global_id 74975 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 1 -- 172.21.15.43:0/341854441 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17080ab3d0 con 0x7f1718133f00 2024-04-19T01:03:00.217 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 4 set_mon_vals no callback set 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 1 -- 172.21.15.43:0/341854441 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17080a8050 con 0x7f1718133f00 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1715d9b640 10 monclient: dump: 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.218 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: shutdown 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1718133f00 msgr2=0x7f171813bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1718133f00 0x7f171813bbb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f17080097f0 tx=0x7f170801c430 comp rx=0 tx=0).stop 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- 172.21.15.43:0/341854441 shutdown_connections 2024-04-19T01:03:00.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718134300 0x7f171813e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 --2- 172.21.15.43:0/341854441 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f1718133f00 0x7f171813bbb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- 172.21.15.43:0/341854441 >> 172.21.15.43:0/341854441 conn(0x7f17180a77d0 msgr2=0x7f17180a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- 172.21.15.43:0/341854441 shutdown_connections 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- 172.21.15.43:0/341854441 wait complete. 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: monmap: 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:00.220 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 librados: starting msgr at 2024-04-19T01:03:00.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 librados: starting objecter 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 Processor -- start 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- start start 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 librados: setting wanted keys 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 librados: calling monclient init 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: init 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient: _add_conns ranks=[7,2,6,8,1,0,3,4,5] 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 0x7f17181c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): picked mon.noname-i con 0x7f1718133b20 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1718133f00 0x7f17181326c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): picked mon.noname-b con 0x7f1718133f00 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1718134300 0x7f1718139280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): picked mon.noname-d con 0x7f1718134300 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f17181411f0 con 0x7f1718133f00 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1718141070 con 0x7f1718134300 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1718141370 con 0x7f1718133b20 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171cdfc640 10 monclient(hunting): authenticate will time out at 2050.339401s 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1718133f00 0x7f17181326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1718134300 0x7f1718139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1718133f00 0x7f17181326c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44490/0 (socket says 172.21.15.43:44490) 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 1 -- 172.21.15.43:0/2652486079 learned_addr learned my addr 172.21.15.43:0/2652486079 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): get_auth_request con 0x7f1718133f00 auth_method 0 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): get_auth_request con 0x7f1718134300 auth_method 0 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 0x7f17181c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 10 monclient(hunting): get_auth_request con 0x7f1718133b20 auth_method 0 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f1716d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 msgr2=0x7f17181c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 msgr2=0x7f17181c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 0x7f17181c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.208+0000 7f171659c640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 0x7f17181c6200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 10 monclient(hunting): handle_auth_done global_id 144402 payload 824 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 msgr2=0x7f17181c6200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1718133b20 0x7f17181c6200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1718133f00 msgr2=0x7f17181326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1718133f00 0x7f17181326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 monclient: found mon.noname-d 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171813aeb0 con 0x7f1718134300 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1716d9d640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1718134300 0x7f1718139280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f17180a39f0 tx=0x7f171000e4b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 client.?.objecter ms_handle_connect 0x7f1718134300 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f171000eda0 con 0x7f1718134300 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: dump: 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.226 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1710014890 con 0x7f1718134300 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1710013380 con 0x7f1718134300 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f1714d99640 4 set_mon_vals no callback set 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 monclient: dump: 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.228 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 5 monclient: authenticate success, global_id 144402 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 monclient: _renew_subs 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171809bb90 con 0x7f1718134300 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 client.144402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 monclient: _renew_subs 2024-04-19T01:03:00.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f171813d1a0 con 0x7f1718134300 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 1 librados: init done 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f171001b070 con 0x7f1718134300 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16f8038e10 0x7f16f803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(690..690 src has 501..690) ==== 6356+0+0 (secure 0 0 0) 0x7f1710019020 con 0x7f1718134300 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 client.144402.objecter ms_dispatch 0x7f1718000ba0 osd_map(690..690 src has 501..690) 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 3 client.144402.objecter handle_osd_map got epochs [690,690] > 0 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 3 client.144402.objecter handle_osd_map decoding full epoch 690 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 0x7f1718139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f171806d140 con 0x7f17181397c0 2024-04-19T01:03:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 20 client.144402.objecter dump_active .. 0 homeless 2024-04-19T01:03:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171659c640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16f8038e10 0x7f16f803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171659c640 10 monclient: get_auth_request con 0x7f16f8038e10 auth_method 0 2024-04-19T01:03:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f171659c640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16f8038e10 0x7f16f803b2d0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7f1700007a70 tx=0x7f1700009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.212+0000 7f170effd640 10 client.144402.objecter ms_handle_connect 0x7f16f8038e10 2024-04-19T01:03:00.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 0x7f1718139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient: get_auth_request con 0x7f17181397c0 auth_method 0 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 10 monclient(hunting): handle_auth_done global_id 144377 payload 824 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17180693b0 con 0x7f17181397c0 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170effd640 10 client.144402.objecter ms_handle_connect 0x7f17181397c0 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170effd640 10 client.144402.objecter resend_mon_ops 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.216+0000 7f170ffff640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 0x7f1718139ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f170801c260 tx=0x7f17080191d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17080ba070 con 0x7f17181397c0 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 10 monclient: dump: 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.235 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1708003620 con 0x7f17181397c0 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 msgr2=0x7f1718139ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 0x7f1718139ba0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f170801c260 tx=0x7f17080191d0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 msgr2=0x7f1718139ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.220+0000 7f170effd640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f17181397c0 0x7f1718139ba0 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f170801c260 tx=0x7f17080191d0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 0x7f17181356f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f1718068240 con 0x7f171813a0e0 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 0x7f17181356f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient: get_auth_request con 0x7f171813a0e0 auth_method 0 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 10 monclient(hunting): handle_auth_done global_id 144386 payload 824 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 1 -- 172.21.15.43:0/2652486079 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f171813cd70 con 0x7f171813a0e0 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170ffff640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 0x7f17181356f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f17180a1320 tx=0x7f17080a9000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 client.144402.objecter ms_handle_connect 0x7f171813a0e0 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 client.144402.objecter resend_mon_ops 2024-04-19T01:03:00.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f17080ba070 con 0x7f171813a0e0 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 monclient: dump: 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.238 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f17080b6450 con 0x7f171813a0e0 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 msgr2=0x7f17181356f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 0x7f17181356f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f17180a1320 tx=0x7f17080a9000 comp rx=0 tx=0).stop 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 msgr2=0x7f17181356f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f170effd640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f171813a0e0 0x7f17181356f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.239 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 144858, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:00.240 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.241 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.242 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:00.243 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.244 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.245 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:00.246 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.247 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.248 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.249 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:00.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:00.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:00.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.261 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:00.261 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:00.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 10 librados: watch_flush enter 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 10 librados: watch_flush exit 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 20 client.144402.objecter shutdown clearing up homeless session... 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 10 client.144402.objecter successfully canceled tick 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16f8038e10 msgr2=0x7f16f803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f16f8038e10 0x7f16f803b2d0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7f1700007a70 tx=0x7f1700009040 comp rx=0 tx=0).stop 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 10 monclient: shutdown 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1718134300 msgr2=0x7f1718139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 --2- 172.21.15.43:0/2652486079 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1718134300 0x7f1718139280 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f17180a39f0 tx=0x7f171000e4b0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f1716d9d640 1 -- 172.21.15.43:0/2652486079 reap_dead start 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f1716d9d640 1 -- 172.21.15.43:0/2652486079 reap_dead start 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 shutdown_connections 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 >> 172.21.15.43:0/2652486079 conn(0x7f17180a77d0 msgr2=0x7f1718071080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 shutdown_connections 2024-04-19T01:03:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 -- 172.21.15.43:0/2652486079 wait complete. 2024-04-19T01:03:00.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.228+0000 7f171cdfc640 1 librados: shutdown 2024-04-19T01:03:00.264 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:00.264 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T01:03:00.264 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.380+0000 7f1f95797640 10 monclient: get_monmap_and_config 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.380+0000 7f1f95797640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.380+0000 7f1f95797640 10 monclient: monmap: 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:00.385289+0000 2024-04-19T01:03:00.382 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:00.385289+0000 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:00.383 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:00.384 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.380+0000 7f1f95797640 1 Processor -- start 2024-04-19T01:03:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 -- start start 2024-04-19T01:03:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient: init 2024-04-19T01:03:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient: _add_conns ranks=[6,4,2,3,1,9,0,8,7,5,10] 2024-04-19T01:03:00.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f90138a50 0x7f1f90138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): picked mon.noname-d con 0x7f1f90138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 0x7f1f90139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): picked mon.noname-c con 0x7f1f900a5bb0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 0x7f1f90137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): picked mon.noname-b con 0x7f1f90137960 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f1f900a1020 con 0x7f1f90137960 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f1f901398b0 con 0x7f1f900a5bb0 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1f900a0780 con 0x7f1f90138a50 2024-04-19T01:03:00.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f95797640 10 monclient(hunting): authenticate will time out at 2050.512889s 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 0x7f1f90137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 0x7f1f90139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f90138a50 0x7f1f90138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 0x7f1f90139370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:57486/0 (socket says 172.21.15.43:57486) 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 1 -- 172.21.15.43:0/813946814 learned_addr learned my addr 172.21.15.43:0/813946814 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 0x7f1f90137dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44496/0 (socket says 172.21.15.43:44496) 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): get_auth_request con 0x7f1f900a5bb0 auth_method 0 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): get_auth_request con 0x7f1f90138a50 auth_method 0 2024-04-19T01:03:00.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): get_auth_request con 0x7f1f90137960 auth_method 0 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient(hunting): handle_auth_done global_id 144411 payload 274 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 -- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 msgr2=0x7f1f90139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 0x7f1f90139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 -- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 msgr2=0x7f1f90137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 0x7f1f90137dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 monclient: found mon.noname-d 2024-04-19T01:03:00.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:03:00.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 -- 172.21.15.43:0/813946814 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f900a0d40 con 0x7f1f90138a50 2024-04-19T01:03:00.391 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f867fc640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 0x7f1f90139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:00.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8f7fe640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 0x7f1f90137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.384+0000 7f1f8effd640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f90138a50 0x7f1f90138e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097f0 tx=0x7f1f8001c370 comp rx=0 tx=0).ready entity=mon.6 client_cookie=82ab24bd5947a735 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 1 -- 172.21.15.43:0/813946814 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f800b0070 con 0x7f1f90138a50 2024-04-19T01:03:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:03:00.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: dump: 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 1 -- 172.21.15.43:0/813946814 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f800aa810 con 0x7f1f90138a50 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 1 -- 172.21.15.43:0/813946814 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f800a9300 con 0x7f1f90138a50 2024-04-19T01:03:00.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8e7fc640 10 monclient: dump: 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 5 monclient: authenticate success, global_id 144411 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 4 set_mon_vals no callback set 2024-04-19T01:03:00.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: shutdown 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f90138a50 msgr2=0x7f1f90138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f90138a50 0x7f1f90138e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097f0 tx=0x7f1f8001c370 comp rx=0 tx=0).stop 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- 172.21.15.43:0/813946814 shutdown_connections 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f1f90137960 0x7f1f90137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f1f900a5bb0 0x7f1f90139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- 172.21.15.43:0/813946814 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f90138a50 0x7f1f90138e30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- 172.21.15.43:0/813946814 >> 172.21.15.43:0/813946814 conn(0x7f1f900a7cb0 msgr2=0x7f1f900a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- 172.21.15.43:0/813946814 shutdown_connections 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- 172.21.15.43:0/813946814 wait complete. 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: monmap: 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.407 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 librados: starting msgr at 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 librados: starting objecter 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 Processor -- start 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- start start 2024-04-19T01:03:00.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 librados: setting wanted keys 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 librados: calling monclient init 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: init 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient: _add_conns ranks=[6,8,7,5,1,4,2,0,3] 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f900a5bb0 0x7f1f901d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): picked mon.noname-d con 0x7f1f900a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:00.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1f90137960 0x7f1f90136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): picked mon.noname-e con 0x7f1f90137960 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1f90138a50 0x7f1f9013c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): picked mon.noname-i con 0x7f1f90138a50 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f1f900a2100 con 0x7f1f900a5bb0 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f1f900a1f80 con 0x7f1f90138a50 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f1f90142810 con 0x7f1f90137960 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f95797640 10 monclient(hunting): authenticate will time out at 2050.519314s 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f900a5bb0 0x7f1f901d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1f90137960 0x7f1f90136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1f90137960 0x7f1f90136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59844/0 (socket says 172.21.15.43:59844) 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f900a5bb0 0x7f1f901d07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:53536/0 (socket says 172.21.15.43:53536) 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 -- 172.21.15.43:0/4051196679 learned_addr learned my addr 172.21.15.43:0/4051196679 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): get_auth_request con 0x7f1f90137960 auth_method 0 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): get_auth_request con 0x7f1f900a5bb0 auth_method 0 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8f7fe640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1f90138a50 0x7f1f9013c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient(hunting): handle_auth_done global_id 144395 payload 824 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1f90138a50 msgr2=0x7f1f9013c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f1f90138a50 0x7f1f9013c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f900a5bb0 msgr2=0x7f1f901d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f900a5bb0 0x7f1f901d07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 monclient: found mon.noname-e 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f867fc640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f90122210 con 0x7f1f90137960 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.388+0000 7f1f8effd640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f1f900a5bb0 0x7f1f901d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f867fc640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1f90137960 0x7f1f90136850 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800e9b0 tx=0x7f1f7800ee70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 client.?.objecter ms_handle_connect 0x7f1f90137960 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f78017070 con 0x7f1f90137960 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: dump: 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.414 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f78013470 con 0x7f1f90137960 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f78013a90 con 0x7f1f90137960 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 5 monclient: authenticate success, global_id 144395 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 monclient: _renew_subs 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f9009fc90 con 0x7f1f90137960 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8dffb640 4 set_mon_vals no callback set 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: dump: 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.416 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.417 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 client.144395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 monclient: _renew_subs 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f90121f30 con 0x7f1f90137960 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 1 librados: init done 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 0x7f1f9013cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1f9006d140 con 0x7f1f90121970 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f1f7801a070 con 0x7f1f90137960 2024-04-19T01:03:00.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1f70038e10 0x7f1f7003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(690..690 src has 501..690) ==== 6356+0+0 (secure 0 0 0) 0x7f1f7800f040 con 0x7f1f90137960 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 client.144395.objecter ms_dispatch 0x7f1f90000ba0 osd_map(690..690 src has 501..690) 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 3 client.144395.objecter handle_osd_map got epochs [690,690] > 0 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 3 client.144395.objecter handle_osd_map decoding full epoch 690 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 0x7f1f9013cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient: get_auth_request con 0x7f1f90121970 auth_method 0 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 20 client.144395.objecter dump_active .. 0 homeless 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8f7fe640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1f70038e10 0x7f1f7003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8f7fe640 10 monclient: get_auth_request con 0x7f1f70038e10 auth_method 0 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 10 monclient(hunting): handle_auth_done global_id 144406 payload 824 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f901d15c0 con 0x7f1f90121970 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 client.144395.objecter ms_handle_connect 0x7f1f90121970 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8effd640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 0x7f1f9013cba0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097c0 tx=0x7f1f80019110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 client.144395.objecter resend_mon_ops 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 client.144395.objecter ms_handle_connect 0x7f1f70038e10 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f8f7fe640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1f70038e10 0x7f1f7003b2d0 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f1f900a7500 tx=0x7f1f8800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f800b0040 con 0x7f1f90121970 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: dump: 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.421 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1f800b1040 con 0x7f1f90121970 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 msgr2=0x7f1f9013cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 0x7f1f9013cba0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097c0 tx=0x7f1f80019110 comp rx=0 tx=0).stop 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 msgr2=0x7f1f9013cba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.392+0000 7f1f87fff640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f90121970 0x7f1f9013cba0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1f800097c0 tx=0x7f1f80019110 comp rx=0 tx=0).stop 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f95797640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f95797640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 0x7f1f9013d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f95797640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1f90076190 con 0x7f1f9013d0e0 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 0x7f1f9013d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient: get_auth_request con 0x7f1f9013d0e0 auth_method 0 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 10 monclient(hunting): handle_auth_done global_id 144415 payload 824 2024-04-19T01:03:00.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 1 -- 172.21.15.43:0/4051196679 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1f90068240 con 0x7f1f9013d0e0 2024-04-19T01:03:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f8effd640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 0x7f1f9013d4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009710 tx=0x7f1f800a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f87fff640 10 client.144395.objecter ms_handle_connect 0x7f1f9013d0e0 2024-04-19T01:03:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f87fff640 10 client.144395.objecter resend_mon_ops 2024-04-19T01:03:00.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f1f80006e20 con 0x7f1f9013d0e0 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f87fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f87fff640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.400+0000 7f1f87fff640 10 monclient: dump: 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.426 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f1f80003de0 con 0x7f1f9013d0e0 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:00.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 msgr2=0x7f1f9013d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 0x7f1f9013d4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1f80009710 tx=0x7f1f800a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 msgr2=0x7f1f9013d4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f87fff640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1f9013d0e0 0x7f1f9013d4f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:00.428 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 145032, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:00.429 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.430 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.431 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.432 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:00.433 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:00.434 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.435 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.436 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.437 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.438 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.439 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:00.440 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:00.441 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.442 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.443 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.444 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.445 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.446 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.447 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.448 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.449 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.450 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.451 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:00.452 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:00.453 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:00.454 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 10 librados: watch_flush enter 2024-04-19T01:03:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 10 librados: watch_flush exit 2024-04-19T01:03:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 20 client.144395.objecter shutdown clearing up homeless session... 2024-04-19T01:03:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 10 client.144395.objecter successfully canceled tick 2024-04-19T01:03:00.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1f70038e10 msgr2=0x7f1f7003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f1f70038e10 0x7f1f7003b2d0 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f1f900a7500 tx=0x7f1f8800b040 comp rx=0 tx=0).stop 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 10 monclient: shutdown 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1f90137960 msgr2=0x7f1f90136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 --2- 172.21.15.43:0/4051196679 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f1f90137960 0x7f1f90136850 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1f7800e9b0 tx=0x7f1f7800ee70 comp rx=0 tx=0).stop 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f8f7fe640 1 -- 172.21.15.43:0/4051196679 reap_dead start 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f8f7fe640 1 -- 172.21.15.43:0/4051196679 reap_dead start 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 shutdown_connections 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 >> 172.21.15.43:0/4051196679 conn(0x7f1f900a7cb0 msgr2=0x7f1f9006d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 shutdown_connections 2024-04-19T01:03:00.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 -- 172.21.15.43:0/4051196679 wait complete. 2024-04-19T01:03:00.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.404+0000 7f1f95797640 1 librados: shutdown 2024-04-19T01:03:00.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:00.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T01:03:00.458 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T01:03:00.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient: get_monmap_and_config 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient: monmap: 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:00.599897+0000 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:00.599897+0000 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.597 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:00.598 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 Processor -- start 2024-04-19T01:03:00.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 -- start start 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient: init 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient: _add_conns ranks=[8,4,3,9,5,7,1,0,10,6,2] 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 0x7f615c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): picked mon.noname-e con 0x7f615c138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f615c0a5b80 0x7f615c1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): picked mon.noname-c con 0x7f615c0a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c1358b0 0x7f615c135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): picked mon.noname-g con 0x7f615c1358b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f615c0a4410 con 0x7f615c1358b0 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f615c139b30 con 0x7f615c0a5b80 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f615c0a3dc0 con 0x7f615c138cd0 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f61624f9640 10 monclient(hunting): authenticate will time out at 2050.726992s 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 0x7f615c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f615c0a5b80 0x7f615c1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 0x7f615c1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59846/0 (socket says 172.21.15.43:59846) 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 -- 172.21.15.43:0/1922727237 learned_addr learned my addr 172.21.15.43:0/1922727237 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): get_auth_request con 0x7f615c0a5b80 auth_method 0 2024-04-19T01:03:00.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): get_auth_request con 0x7f615c138cd0 auth_method 0 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient(hunting): handle_auth_done global_id 144404 payload 274 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 -- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f615c0a5b80 msgr2=0x7f615c1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f615c0a5b80 0x7f615c1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 -- 172.21.15.43:0/1922727237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c1358b0 msgr2=0x7f615c135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f6160a96640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c1358b0 0x7f615c135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c1358b0 0x7f615c135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 monclient: found mon.noname-e 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615bfff640 1 -- 172.21.15.43:0/1922727237 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c0a40a0 con 0x7f615c138cd0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f615b7fe640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f615c0a5b80 0x7f615c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f6160a96640 10 monclient: get_auth_request con 0x7f615c1358b0 auth_method 0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.596+0000 7f6160a96640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c1358b0 0x7f615c135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615bfff640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 0x7f615c1390b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f614c0098a0 tx=0x7f614c01c3e0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a2c60b01e4e3a716 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 1 -- 172.21.15.43:0/1922727237 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f614c0b0070 con 0x7f615c138cd0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:00.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: dump: 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.614 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 1 -- 172.21.15.43:0/1922727237 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f614c0aa810 con 0x7f615c138cd0 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 5 monclient: authenticate success, global_id 144404 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 1 -- 172.21.15.43:0/1922727237 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f614c0a9300 con 0x7f615c138cd0 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 4 set_mon_vals no callback set 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f615affd640 10 monclient: dump: 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.616 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: shutdown 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 -- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 msgr2=0x7f615c1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 0x7f615c1390b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f614c0098a0 tx=0x7f614c01c3e0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 -- 172.21.15.43:0/1922727237 shutdown_connections 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c1358b0 0x7f615c135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f615c0a5b80 0x7f615c1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 --2- 172.21.15.43:0/1922727237 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c138cd0 0x7f615c1390b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 -- 172.21.15.43:0/1922727237 >> 172.21.15.43:0/1922727237 conn(0x7f615c0ab2f0 msgr2=0x7f615c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 -- 172.21.15.43:0/1922727237 shutdown_connections 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 -- 172.21.15.43:0/1922727237 wait complete. 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: build_initial_monmap 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: monmap: 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:00.618 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:00.619 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 librados: starting msgr at 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 librados: starting objecter 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 Processor -- start 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 -- start start 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 librados: setting wanted keys 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 librados: calling monclient init 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: init 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 10 monclient: _add_conns ranks=[5,8,0,2,1,7,4,6,3] 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.600+0000 7f61624f9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f615c0a5b80 0x7f615c12ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): picked mon.noname-h con 0x7f615c0a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:00.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 0x7f615c12bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): picked mon.noname-e con 0x7f615c1358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f615c138cd0 0x7f615c1334c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): picked mon.noname-a con 0x7f615c138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f615c0a2010 con 0x7f615c138cd0 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f615c0a1e90 con 0x7f615c0a5b80 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f615c142aa0 con 0x7f615c1358b0 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): authenticate will time out at 2050.732071s 2024-04-19T01:03:00.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 0x7f615c12bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 0x7f615c12bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:59860/0 (socket says 172.21.15.43:59860) 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f615c138cd0 0x7f615c1334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 -- 172.21.15.43:0/4070212452 learned_addr learned my addr 172.21.15.43:0/4070212452 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): get_auth_request con 0x7f615c1358b0 auth_method 0 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): get_auth_request con 0x7f615c138cd0 auth_method 0 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f615c0a5b80 0x7f615c12ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient(hunting): handle_auth_done global_id 144413 payload 824 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f615c0a5b80 msgr2=0x7f615c12ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f615c0a5b80 0x7f615c12ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f615c138cd0 msgr2=0x7f615c1334c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f615c138cd0 0x7f615c1334c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:00.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 monclient: found mon.noname-e 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f615c13b6c0 con 0x7f615c1358b0 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c0a5b80 auth_method 0 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f615c0a5b80 0x7f615c12ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615b7fe640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 0x7f615c12bfb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f615000ebb0 tx=0x7f615000c520 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 client.?.objecter ms_handle_connect 0x7f615c1358b0 2024-04-19T01:03:00.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6150017070 con 0x7f615c1358b0 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: dump: 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.625 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: _finish_auth 0 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f615000cc30 con 0x7f615c1358b0 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 5 monclient: authenticate success, global_id 144413 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient: _renew_subs 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f615c0a3590 con 0x7f615c1358b0 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f615000cde0 con 0x7f615c1358b0 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 monclient: dump: 2024-04-19T01:03:00.627 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:00.628 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615a7fc640 4 set_mon_vals no callback set 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 client.144413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient: _renew_subs 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f615c13a0c0 con 0x7f615c1358b0 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 librados: init done 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:00.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f615c06d140 con 0x7f615c133a00 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f615001a070 con 0x7f615c1358b0 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6148038e10 0x7f614803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c133a00 auth_method 0 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(691..691 src has 501..691) ==== 6729+0+0 (secure 0 0 0) 0x7f615000f040 con 0x7f615c1358b0 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:00.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 client.144413.objecter ms_dispatch 0x7f615c000ba0 osd_map(691..691 src has 501..691) 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 3 client.144413.objecter handle_osd_map got epochs [691,691] > 0 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 3 client.144413.objecter handle_osd_map decoding full epoch 691 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 20 client.144413.objecter dump_active .. 0 homeless 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6148038e10 0x7f614803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 10 monclient: get_auth_request con 0x7f6148038e10 auth_method 0 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6158ff9640 10 client.144413.objecter ms_handle_connect 0x7f6148038e10 2024-04-19T01:03:00.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.604+0000 7f6160a96640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6148038e10 0x7f614803b2d0 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7f615c134b00 tx=0x7f6154007780 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c133a00 auth_method 0 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:00.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:00.808+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c133a00 auth_method 0 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:01.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:01.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:01.212+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c133a00 auth_method 0 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:02.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:02.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:02.016+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:03.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.604+0000 7f61437fe640 10 monclient: tick 2024-04-19T01:03:03.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.604+0000 7f61437fe640 10 monclient: _check_auth_tickets 2024-04-19T01:03:03.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.604+0000 7f61437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:03.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.604+0000 7f61437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c133a00 auth_method 0 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:03.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:03.620+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 10 monclient: tick 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 10 monclient: _check_auth_tickets 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 0x7f61340059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 msgr2=0x7f615c133de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c133a00 0x7f615c133de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f61437fe640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6134006090 con 0x7f6134003550 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 0x7f61340059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f6134003550 auth_method 0 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.606 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 10 monclient(hunting): handle_auth_done global_id 164100 payload 824 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6134006240 con 0x7f6134003550 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 0x7f61340059e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f614c009870 tx=0x7f614c0191e0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f6158ff9640 10 client.144413.objecter ms_handle_connect 0x7f6134003550 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f6158ff9640 10 client.144413.objecter resend_mon_ops 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f614c0b0040 con 0x7f6134003550 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f6158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f6158ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.604+0000 7f6158ff9640 10 monclient: dump: 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.607 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.608 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f614c0036a0 con 0x7f6134003550 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 msgr2=0x7f61340059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 0x7f61340059e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f614c009870 tx=0x7f614c0191e0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 msgr2=0x7f61340059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:04.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.608+0000 7f6158ff9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6134003550 0x7f61340059e0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f614c009870 tx=0x7f614c0191e0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f61624f9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f61624f9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 0x7f615c137a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f61624f9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f615c068240 con 0x7f615c134320 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 0x7f615c137a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient: get_auth_request con 0x7f615c134320 auth_method 0 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 10 monclient(hunting): handle_auth_done global_id 164109 payload 824 2024-04-19T01:03:04.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 1 -- 172.21.15.43:0/4070212452 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f615c135660 con 0x7f615c134320 2024-04-19T01:03:04.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f6158ff9640 10 client.144413.objecter ms_handle_connect 0x7f615c134320 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f6158ff9640 10 client.144413.objecter resend_mon_ops 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.616+0000 7f615bfff640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 0x7f615c137a90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f614c0098a0 tx=0x7f614c009240 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f614c0b0040 con 0x7f615c134320 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 10 monclient: dump: 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.621 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.622 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f614c0c1a90 con 0x7f615c134320 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 msgr2=0x7f615c137a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 0x7f615c137a90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f614c0098a0 tx=0x7f614c009240 comp rx=0 tx=0).stop 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 msgr2=0x7f615c137a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:04.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6158ff9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f615c134320 0x7f615c137a90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f6160a96640 1 -- 172.21.15.43:0/4070212452 reap_dead start 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15987, 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:04.624 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:04.625 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:04.626 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:04.627 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:04.628 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:04.629 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.630 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.631 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.632 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.632 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.632 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:04.632 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:04.633 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:04.634 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:04.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.637 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:04.639 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:04.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.642 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.644 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.646 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.648 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:04.648 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 10 librados: watch_flush enter 2024-04-19T01:03:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 10 librados: watch_flush exit 2024-04-19T01:03:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 20 client.144413.objecter shutdown clearing up homeless session... 2024-04-19T01:03:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 10 client.144413.objecter successfully canceled tick 2024-04-19T01:03:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6148038e10 msgr2=0x7f614803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.650 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6148038e10 0x7f614803b2d0 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7f615c134b00 tx=0x7f6154007780 comp rx=0 tx=0).stop 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 10 monclient: shutdown 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 msgr2=0x7f615c12bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 0x7f615c12bfb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f615000ebb0 tx=0x7f615000c520 comp rx=0 tx=0).stop 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 shutdown_connections 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6148038e10 0x7f614803b2d0 unknown :-1 s=CLOSED pgs=3634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 --2- 172.21.15.43:0/4070212452 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f615c1358b0 0x7f615c12bfb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 >> 172.21.15.43:0/4070212452 conn(0x7f615c0ab2f0 msgr2=0x7f615c0a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 shutdown_connections 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 -- 172.21.15.43:0/4070212452 wait complete. 2024-04-19T01:03:04.651 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.620+0000 7f61624f9640 1 librados: shutdown 2024-04-19T01:03:04.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:04.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T01:03:04.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.776+0000 7f2525ea8640 10 monclient: get_monmap_and_config 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.776+0000 7f2525ea8640 10 monclient: build_initial_monmap 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.776+0000 7f2525ea8640 10 monclient: monmap: 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:04.781962+0000 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:04.781962+0000 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:04.779 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:04.780 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 Processor -- start 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 -- start start 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient: init 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient: _add_conns ranks=[2,3,10,6,7,8,1,9,5,4,0] 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 0x7f25201390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f2520138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f25200a5b80 0x7f25201395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): picked mon.noname-g con 0x7f25200a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f25201358b0 0x7f2520135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): picked mon.noname-f con 0x7f25201358b0 addr v2:172.21.15.161:3300/0 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f25200a4410 con 0x7f25201358b0 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2520139b30 con 0x7f2520138cd0 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f25200a3dc0 con 0x7f25200a5b80 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f2525ea8640 10 monclient(hunting): authenticate will time out at 2054.909078s 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 0x7f25201390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 0x7f25201390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60782/0 (socket says 172.21.15.43:60782) 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 -- 172.21.15.43:0/3851133755 learned_addr learned my addr 172.21.15.43:0/3851133755 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): get_auth_request con 0x7f2520138cd0 auth_method 0 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 1 --2- 172.21.15.43:0/3851133755 >> v2:172.21.15.161:3300/0 conn(0x7f25201358b0 0x7f2520135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 1 --2- 172.21.15.43:0/3851133755 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f25200a5b80 0x7f25201395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): get_auth_request con 0x7f25200a5b80 auth_method 0 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): get_auth_request con 0x7f25201358b0 auth_method 0 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient(hunting): handle_auth_done global_id 144416 payload 274 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 -- 172.21.15.43:0/3851133755 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f25200a5b80 msgr2=0x7f25201395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3851133755 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f25200a5b80 0x7f25201395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 -- 172.21.15.43:0/3851133755 >> v2:172.21.15.161:3300/0 conn(0x7f25201358b0 msgr2=0x7f2520135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3851133755 >> v2:172.21.15.161:3300/0 conn(0x7f25201358b0 0x7f2520135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 monclient: found mon.noname-b 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:04.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.780+0000 7f251f7fe640 1 -- 172.21.15.43:0/3851133755 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25200a40a0 con 0x7f2520138cd0 2024-04-19T01:03:04.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3851133755 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 0x7f25201390b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2514009870 tx=0x7f251401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c88052719f5754f server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 1 -- 172.21.15.43:0/3851133755 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25140b0070 con 0x7f2520138cd0 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: dump: 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.795 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 1 -- 172.21.15.43:0/3851133755 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25140aa810 con 0x7f2520138cd0 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 1 -- 172.21.15.43:0/3851133755 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25140a9300 con 0x7f2520138cd0 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 5 monclient: authenticate success, global_id 144416 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 4 set_mon_vals no callback set 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f251e7fc640 10 monclient: dump: 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.797 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.798 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 10 monclient: shutdown 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 -- 172.21.15.43:0/3851133755 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 msgr2=0x7f25201390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3851133755 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 0x7f25201390b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2514009870 tx=0x7f251401c3c0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 -- 172.21.15.43:0/3851133755 shutdown_connections 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3851133755 >> v2:172.21.15.161:3300/0 conn(0x7f25201358b0 0x7f2520135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3851133755 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f25200a5b80 0x7f25201395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3851133755 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f2520138cd0 0x7f25201390b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 -- 172.21.15.43:0/3851133755 >> 172.21.15.43:0/3851133755 conn(0x7f25200ab2f0 msgr2=0x7f25200ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:04.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 -- 172.21.15.43:0/3851133755 shutdown_connections 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 -- 172.21.15.43:0/3851133755 wait complete. 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 10 monclient: build_initial_monmap 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 10 monclient: monmap: 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:04.800 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 librados: starting msgr at 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 librados: starting objecter 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 Processor -- start 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.784+0000 7f2525ea8640 1 -- start start 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 librados: setting wanted keys 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 librados: calling monclient init 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: init 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: _add_conns ranks=[4,8,0,7,3,2,5,6,1] 2024-04-19T01:03:04.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f25200a5b80 0x7f25201d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f25200a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f25201358b0 0x7f25201d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): picked mon.noname-e con 0x7f25201358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2520138cd0 0x7f25201d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f2520138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f25200a2010 con 0x7f2520138cd0 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f25200a1e90 con 0x7f25200a5b80 2024-04-19T01:03:04.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f2520142aa0 con 0x7f25201358b0 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): authenticate will time out at 2054.916646s 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f25200a5b80 0x7f25201d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f25201358b0 0x7f25201d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2520138cd0 0x7f25201d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f25200a5b80 0x7f25201d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47538/0 (socket says 172.21.15.43:47538) 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 1 -- 172.21.15.43:0/3689592908 learned_addr learned my addr 172.21.15.43:0/3689592908 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): get_auth_request con 0x7f25200a5b80 auth_method 0 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): get_auth_request con 0x7f2520138cd0 auth_method 0 2024-04-19T01:03:04.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): get_auth_request con 0x7f25201358b0 auth_method 0 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient(hunting): handle_auth_done global_id 65637 payload 824 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f25201358b0 msgr2=0x7f25201d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f25201358b0 0x7f25201d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f25200a5b80 msgr2=0x7f25201d0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f25200a5b80 0x7f25201d0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 monclient: found mon.noname-a 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25200a3040 con 0x7f2520138cd0 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f25201358b0 0x7f25201d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251ffff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2520138cd0 0x7f25201d14b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f251800efc0 tx=0x7f251800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 client.?.objecter ms_handle_connect 0x7f2520138cd0 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:04.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2518018020 con 0x7f2520138cd0 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: dump: 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: _finish_auth 0 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251800cd30 con 0x7f2520138cd0 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f2518014d60 con 0x7f2520138cd0 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 5 monclient: authenticate success, global_id 65637 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: _renew_subs 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25200a3590 con 0x7f2520138cd0 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 monclient: dump: 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.808 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251d7fa640 4 set_mon_vals no callback set 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 client.65637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: _renew_subs 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25200a0730 con 0x7f2520138cd0 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 librados: init done 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 0x7f2520134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f252006d140 con 0x7f25201348a0 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f251801b020 con 0x7f2520138cd0 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 0x7f2520134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f250c038e10 0x7f250c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient: get_auth_request con 0x7f25201348a0 auth_method 0 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(695..695 src has 501..695) ==== 6356+0+0 (secure 0 0 0) 0x7f251804a120 con 0x7f2520138cd0 2024-04-19T01:03:04.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 client.65637.objecter ms_dispatch 0x7f2520000ba0 osd_map(695..695 src has 501..695) 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 3 client.65637.objecter handle_osd_map got epochs [695,695] > 0 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 3 client.65637.objecter handle_osd_map decoding full epoch 695 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 20 client.65637.objecter dump_active .. 0 homeless 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f250c038e10 0x7f250c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251effd640 10 monclient: get_auth_request con 0x7f250c038e10 auth_method 0 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 10 monclient(hunting): handle_auth_done global_id 74984 payload 824 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25200693b0 con 0x7f25201348a0 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 client.65637.objecter ms_handle_connect 0x7f25201348a0 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f2507fff640 10 client.65637.objecter resend_mon_ops 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.788+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 0x7f2520134c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f251401c1f0 tx=0x7f25140191c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 10 client.65637.objecter ms_handle_connect 0x7f250c038e10 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f251effd640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f250c038e10 0x7f250c03b2d0 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7f25200a8ce0 tx=0x7f2510009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25140ba070 con 0x7f25201348a0 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 10 monclient: dump: 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.812 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f25140093d0 con 0x7f25201348a0 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 msgr2=0x7f2520134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 0x7f2520134c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f251401c1f0 tx=0x7f25140191c0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 msgr2=0x7f2520134c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:04.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.792+0000 7f2507fff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201348a0 0x7f2520134c80 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f251401c1f0 tx=0x7f25140191c0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f2525ea8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 0x7f252013bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f2525ea8640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f2520068240 con 0x7f25201351c0 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 0x7f252013bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient: get_auth_request con 0x7f25201351c0 auth_method 0 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.796+0000 7f251f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f251f7fe640 10 monclient(hunting): handle_auth_done global_id 74993 payload 824 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f251f7fe640 1 -- 172.21.15.43:0/3689592908 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f252009f150 con 0x7f25201351c0 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f251f7fe640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 0x7f252013bcf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f251401c400 tx=0x7f25140a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 client.65637.objecter ms_handle_connect 0x7f25201351c0 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 client.65637.objecter resend_mon_ops 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f25140cd020 con 0x7f25201351c0 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 monclient: dump: 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.816 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f25140b6440 con 0x7f25201351c0 2024-04-19T01:03:04.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 msgr2=0x7f252013bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 0x7f252013bcf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f251401c400 tx=0x7f25140a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 msgr2=0x7f252013bcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2507fff640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f25201351c0 0x7f252013bcf0 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f251401c400 tx=0x7f25140a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:04.818 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 564417, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 5, 2024-04-19T01:03:04.819 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:04.820 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:04.821 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:04.822 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:04.823 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:04.824 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:04.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:04.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.830 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:04.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.835 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:04.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:04.839 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:04.840 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 10 librados: watch_flush enter 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 10 librados: watch_flush exit 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 20 client.65637.objecter shutdown clearing up homeless session... 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 10 client.65637.objecter successfully canceled tick 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f250c038e10 msgr2=0x7f250c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f250c038e10 0x7f250c03b2d0 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7f25200a8ce0 tx=0x7f2510009190 comp rx=0 tx=0).stop 2024-04-19T01:03:04.842 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 10 monclient: shutdown 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2520138cd0 msgr2=0x7f25201d14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 --2- 172.21.15.43:0/3689592908 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f2520138cd0 0x7f25201d14b0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f251800efc0 tx=0x7f251800c490 comp rx=0 tx=0).stop 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f251ffff640 1 -- 172.21.15.43:0/3689592908 reap_dead start 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f251ffff640 1 -- 172.21.15.43:0/3689592908 reap_dead start 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 shutdown_connections 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 >> 172.21.15.43:0/3689592908 conn(0x7f25200ab2f0 msgr2=0x7f2520137660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 shutdown_connections 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 -- 172.21.15.43:0/3689592908 wait complete. 2024-04-19T01:03:04.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.800+0000 7f2525ea8640 1 librados: shutdown 2024-04-19T01:03:04.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:04.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T01:03:04.844 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.964+0000 7f7ed7d76640 10 monclient: get_monmap_and_config 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.964+0000 7f7ed7d76640 10 monclient: build_initial_monmap 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.964+0000 7f7ed7d76640 10 monclient: monmap: 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:04.970187+0000 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:04.970187+0000 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:04.967 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:04.968 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 Processor -- start 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 -- start start 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient: init 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient: _add_conns ranks=[1,9,5,7,4,6,3,8,2,10,0] 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f7ed0138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7ed00a5b80 0x7f7ed013e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): picked mon.noname-a con 0x7f7ed00a5b80 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7ed01397c0 0x7f7ed01407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): picked mon.noname-h con 0x7f7ed01397c0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ed00a3dc0 con 0x7f7ed0138cd0 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f7ed00a4410 con 0x7f7ed00a5b80 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7ed00a40a0 con 0x7f7ed01397c0 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed7d76640 10 monclient(hunting): authenticate will time out at 2055.097356s 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7ed00a5b80 0x7f7ed013e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f7ed00a5b80 0x7f7ed013e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44928/0 (socket says 172.21.15.43:44928) 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 -- 172.21.15.43:0/726870863 learned_addr learned my addr 172.21.15.43:0/726870863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): get_auth_request con 0x7f7ed00a5b80 auth_method 0 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed6313640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7ed01397c0 0x7f7ed01407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed6313640 10 monclient(hunting): get_auth_request con 0x7f7ed01397c0 auth_method 0 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed6313640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed6313640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed6313640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient(hunting): handle_auth_done global_id 65646 payload 274 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 -- 172.21.15.43:0/726870863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7ed01397c0 msgr2=0x7f7ed01407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7ed01397c0 0x7f7ed01407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 -- 172.21.15.43:0/726870863 >> v1:172.21.15.161:6789/0 conn(0x7f7ed0138cd0 legacy=0x7f7ed01390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient: _finish_hunting 0 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 monclient: found mon.noname-a 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed5311640 1 -- 172.21.15.43:0/726870863 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed00a2ae0 con 0x7f7ed00a5b80 2024-04-19T01:03:04.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.968+0000 7f7ed6313640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7ed01397c0 0x7f7ed01407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:03:04.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed5311640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed013e270 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009870 tx=0x7f7ecc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a67538c81c6e6c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 1 -- 172.21.15.43:0/726870863 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ecc0b0070 con 0x7f7ed00a5b80 2024-04-19T01:03:04.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: dump: 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.981 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: _finish_auth 0 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: _check_auth_tickets 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 5 monclient: authenticate success, global_id 65646 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 1 -- 172.21.15.43:0/726870863 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ecc0aa810 con 0x7f7ed00a5b80 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:04.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 4 set_mon_vals no callback set 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 1 -- 172.21.15.43:0/726870863 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ecc0a9300 con 0x7f7ed00a5b80 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed4b10640 10 monclient: dump: 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.983 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.984 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.984 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: shutdown 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- 172.21.15.43:0/726870863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 msgr2=0x7f7ed013e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed013e270 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009870 tx=0x7f7ecc01c420 comp rx=0 tx=0).stop 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- 172.21.15.43:0/726870863 shutdown_connections 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7ed01397c0 0x7f7ed01407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/726870863 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed013e270 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- 172.21.15.43:0/726870863 >> 172.21.15.43:0/726870863 conn(0x7f7ed00ab2f0 msgr2=0x7f7ed00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- 172.21.15.43:0/726870863 shutdown_connections 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- 172.21.15.43:0/726870863 wait complete. 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: build_initial_monmap 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: monmap: 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:04.986 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 librados: starting msgr at 2024-04-19T01:03:04.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 librados: starting objecter 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 Processor -- start 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- start start 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 librados: setting wanted keys 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 librados: calling monclient init 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: init 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient: _add_conns ranks=[0,1,2,6,8,5,4,3,7] 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed0136f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): picked mon.noname-a con 0x7f7ed00a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ed0138cd0 0x7f7ed009df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): picked mon.noname-f con 0x7f7ed0138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:04.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ed01397c0 0x7f7ed009e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): picked mon.noname-b con 0x7f7ed01397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed0143730 con 0x7f7ed00a5b80 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed00a20a0 con 0x7f7ed0138cd0 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f7ed01438b0 con 0x7f7ed01397c0 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed7d76640 10 monclient(hunting): authenticate will time out at 2055.103691s 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ed01397c0 0x7f7ed009e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed5b12640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed0136f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ed01397c0 0x7f7ed009e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60788/0 (socket says 172.21.15.43:60788) 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 1 -- 172.21.15.43:0/209733838 learned_addr learned my addr 172.21.15.43:0/209733838 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed5b12640 10 monclient(hunting): get_auth_request con 0x7f7ed00a5b80 auth_method 0 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed5b12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed5b12640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed5b12640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 10 monclient(hunting): get_auth_request con 0x7f7ed01397c0 auth_method 0 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.972+0000 7f7ed6313640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ed0138cd0 0x7f7ed009df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): get_auth_request con 0x7f7ed0138cd0 auth_method 0 2024-04-19T01:03:04.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 10 monclient(hunting): handle_auth_done global_id 65655 payload 824 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ed01397c0 msgr2=0x7f7ed009e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ed01397c0 0x7f7ed009e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ed0138cd0 msgr2=0x7f7ed009df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7ed0138cd0 0x7f7ed009df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 10 monclient: _finish_hunting 0 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 monclient: found mon.noname-a 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed01218f0 con 0x7f7ed00a5b80 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f7ed01397c0 0x7f7ed009e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5b12640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed0136f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00a72e0 tx=0x7f7ec000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 client.?.objecter ms_handle_connect 0x7f7ed00a5b80 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ec0003600 con 0x7f7ed00a5b80 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: dump: 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.993 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.994 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec0003800 con 0x7f7ed00a5b80 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ec0003e20 con 0x7f7ed00a5b80 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec7fff640 4 set_mon_vals no callback set 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: dump: 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:04.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:04.996 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 5 monclient: authenticate success, global_id 65655 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 monclient: _renew_subs 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed00a3590 con 0x7f7ed00a5b80 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 client.65655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 monclient: _renew_subs 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed00a1890 con 0x7f7ed00a5b80 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 1 librados: init done 2024-04-19T01:03:04.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 0x7f7ed002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7ed00a0730 con 0x7f7ed01c8740 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7ec0019020 con 0x7f7ed00a5b80 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e40 0x7f7eb403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 0x7f7ed002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient: get_auth_request con 0x7f7ed01c8740 auth_method 0 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(695..695 src has 501..695) ==== 6356+0+0 (secure 0 0 0) 0x7f7ec00486e0 con 0x7f7ed00a5b80 2024-04-19T01:03:04.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 client.65655.objecter ms_dispatch 0x7f7ed0000ba0 osd_map(695..695 src has 501..695) 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 3 client.65655.objecter handle_osd_map got epochs [695,695] > 0 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 3 client.65655.objecter handle_osd_map decoding full epoch 695 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 20 client.65655.objecter dump_active .. 0 homeless 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e40 0x7f7eb403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 10 monclient: get_auth_request con 0x7f7eb4038e40 auth_method 0 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 10 monclient(hunting): handle_auth_done global_id 164104 payload 824 2024-04-19T01:03:04.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ed00a3040 con 0x7f7ed01c8740 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed5311640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 0x7f7ed002c450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00a7450 tx=0x7f7ecc0191c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 client.65655.objecter ms_handle_connect 0x7f7ed01c8740 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 client.65655.objecter resend_mon_ops 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 client.65655.objecter ms_handle_connect 0x7f7eb4038e40 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ed6313640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e40 0x7f7eb403b300 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8003eb0 tx=0x7f7ec8009870 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ecc0b9030 con 0x7f7ed01c8740 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.976+0000 7f7ec67fc640 10 monclient: dump: 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.000 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:05.001 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ecc0b1040 con 0x7f7ed01c8740 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 msgr2=0x7f7ed002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.002 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 0x7f7ed002c450 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00a7450 tx=0x7f7ecc0191c0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 msgr2=0x7f7ed002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:05.003 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.980+0000 7f7ec67fc640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed01c8740 0x7f7ed002c450 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00a7450 tx=0x7f7ecc0191c0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed7d76640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 0x7f7ed002cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed7d76640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7ed006d140 con 0x7f7ed002c990 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 0x7f7ed002cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient: get_auth_request con 0x7f7ed002c990 auth_method 0 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:05.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 10 monclient(hunting): handle_auth_done global_id 164113 payload 824 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 1 -- 172.21.15.43:0/209733838 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ed00693b0 con 0x7f7ed002c990 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ed5311640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 0x7f7ed002cda0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc01aad0 tx=0x7f7ecc0a8000 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ec67fc640 10 client.65655.objecter ms_handle_connect 0x7f7ed002c990 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.984+0000 7f7ec67fc640 10 client.65655.objecter resend_mon_ops 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7ecc0b9040 con 0x7f7ed002c990 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 10 monclient: dump: 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:05.006 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:05.007 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f7ecc003c00 con 0x7f7ed002c990 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 msgr2=0x7f7ed002cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 0x7f7ed002cda0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc01aad0 tx=0x7f7ecc0a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 msgr2=0x7f7ed002cda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:05.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ec67fc640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7ed002c990 0x7f7ed002cda0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc01aad0 tx=0x7f7ecc0a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16355, 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:05.009 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 6, 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:05.010 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:05.011 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:05.012 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:05.013 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:05.014 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:05.015 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:05.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.017 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:05.019 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.020 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.021 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.022 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.022 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:05.022 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:05.022 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:05.027 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:05.028 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.029 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.032 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:05.032 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:05.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:05.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:05.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:05.034 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:05.035 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:05.036 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:05.037 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 10 librados: watch_flush enter 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 10 librados: watch_flush exit 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 20 client.65655.objecter shutdown clearing up homeless session... 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 10 client.65655.objecter successfully canceled tick 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e40 msgr2=0x7f7eb403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7eb4038e40 0x7f7eb403b300 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8003eb0 tx=0x7f7ec8009870 comp rx=0 tx=0).stop 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 10 monclient: shutdown 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 msgr2=0x7f7ed0136f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 --2- 172.21.15.43:0/209733838 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7ed00a5b80 0x7f7ed0136f60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00a72e0 tx=0x7f7ec000c820 comp rx=0 tx=0).stop 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed6313640 1 -- 172.21.15.43:0/209733838 reap_dead start 2024-04-19T01:03:05.038 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed6313640 1 -- 172.21.15.43:0/209733838 reap_dead start 2024-04-19T01:03:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 shutdown_connections 2024-04-19T01:03:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 >> 172.21.15.43:0/209733838 conn(0x7f7ed00ab2f0 msgr2=0x7f7ed00a99d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 shutdown_connections 2024-04-19T01:03:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 -- 172.21.15.43:0/209733838 wait complete. 2024-04-19T01:03:05.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:04.988+0000 7f7ed7d76640 1 librados: shutdown 2024-04-19T01:03:05.039 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:05.039 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T01:03:05.039 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T01:03:05.157 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.156+0000 7f6db0ddb640 10 monclient: get_monmap_and_config 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.156+0000 7f6db0ddb640 10 monclient: build_initial_monmap 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.156+0000 7f6db0ddb640 10 monclient: monmap: 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:05.160871+0000 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:05.160871+0000 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:05.158 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:05.159 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.156+0000 7f6db0ddb640 1 Processor -- start 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.156+0000 7f6db0ddb640 1 -- start start 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.156+0000 7f6db0ddb640 10 monclient: init 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient: _add_conns ranks=[1,8,0,3,6,10,9,7,4,2,5] 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f6dac138a50 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6dac139540 0x7f6dac13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:05.161 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient(hunting): picked mon.noname-e con 0x7f6dac139540 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6dac0a5bb0 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dac0a0d40 con 0x7f6dac0a5bb0 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dac0a0a60 con 0x7f6dac138a50 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6dac0a1020 con 0x7f6dac139540 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient(hunting): authenticate will time out at 2055.288759s 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6dac139540 0x7f6dac13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6daad9d640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f6dac0a5bb0 0x7f6dac140530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47048/0 (socket says 172.21.15.43:47048) 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6daad9d640 1 -- 172.21.15.43:0/2913920207 learned_addr learned my addr 172.21.15.43:0/2913920207 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): get_auth_request con 0x7f6dac139540 auth_method 0 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:05.162 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 1 -- 172.21.15.43:0/2913920207 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1916677620 0 0) 0x7f6dac0a0d40 con 0x7f6dac0a5bb0 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient(hunting): _negotiate 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient(hunting): my global_id is 65664 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 1 -- 172.21.15.43:0/2913920207 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d94003530 con 0x7f6dac0a5bb0 2024-04-19T01:03:05.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_done global_id 144431 payload 274 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 1 -- 172.21.15.43:0/2913920207 >> v1:172.21.15.161:6789/0 conn(0x7f6dac138a50 legacy=0x7f6dac138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 1 -- 172.21.15.43:0/2913920207 >> v1:172.21.15.43:6789/0 conn(0x7f6dac0a5bb0 legacy=0x7f6dac140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 1 monclient: found mon.noname-e 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:05.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 1 -- 172.21.15.43:0/2913920207 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac0a17b0 con 0x7f6dac139540 2024-04-19T01:03:05.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da9d9b640 1 --2- 172.21.15.43:0/2913920207 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6dac139540 0x7f6dac13dfe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6da00097f0 tx=0x7f6da001c400 comp rx=0 tx=0).ready entity=mon.8 client_cookie=55df17477b67afd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:05.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 1 -- 172.21.15.43:0/2913920207 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6da00b0070 con 0x7f6dac139540 2024-04-19T01:03:05.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:05.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:05.171 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: dump: 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:05.172 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: _finish_auth 0 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: _check_auth_tickets 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 1 -- 172.21.15.43:0/2913920207 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da00a4e50 con 0x7f6dac139540 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 5 monclient: authenticate success, global_id 144431 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:05.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 1 -- 172.21.15.43:0/2913920207 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6da00aac80 con 0x7f6dac139540 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6da959a640 10 monclient: dump: 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.174 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.160+0000 7f6db0ddb640 4 set_mon_vals no callback set 2024-04-19T01:03:05.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient: shutdown 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- 172.21.15.43:0/2913920207 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6dac139540 msgr2=0x7f6dac13dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/2913920207 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6dac139540 0x7f6dac13dfe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6da00097f0 tx=0x7f6da001c400 comp rx=0 tx=0).stop 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- 172.21.15.43:0/2913920207 shutdown_connections 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/2913920207 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6dac139540 0x7f6dac13dfe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- 172.21.15.43:0/2913920207 >> 172.21.15.43:0/2913920207 conn(0x7f6dac0a7cb0 msgr2=0x7f6dac0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- 172.21.15.43:0/2913920207 shutdown_connections 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- 172.21.15.43:0/2913920207 wait complete. 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient: build_initial_monmap 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient: monmap: 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:05.176 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 librados: starting msgr at 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 librados: starting objecter 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 Processor -- start 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- start start 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 librados: setting wanted keys 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 librados: calling monclient init 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient: init 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient: _add_conns ranks=[7,5,1,8,6,3,4,0,2] 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6dac0a5bb0 0x7f6dac09ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): picked mon.noname-i con 0x7f6dac0a5bb0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 0x7f6dac09bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): picked mon.noname-h con 0x7f6dac138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6dac139540 0x7f6dac09c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:05.178 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): picked mon.noname-f con 0x7f6dac139540 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6dac144ca0 con 0x7f6dac139540 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f6dac0a0780 con 0x7f6dac138a50 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f6dac144e20 con 0x7f6dac0a5bb0 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6db0ddb640 10 monclient(hunting): authenticate will time out at 2055.294750s 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6dac139540 0x7f6dac09c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 0x7f6dac09bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6dac0a5bb0 0x7f6dac09ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:05.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 0x7f6dac09bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:49982/0 (socket says 172.21.15.43:49982) 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 -- 172.21.15.43:0/1227535292 learned_addr learned my addr 172.21.15.43:0/1227535292 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6dac0a5bb0 0x7f6dac09ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3303/0 says I am v2:172.21.15.43:49522/0 (socket says 172.21.15.43:49522) 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6dac139540 0x7f6dac09c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:51736/0 (socket says 172.21.15.43:51736) 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): get_auth_request con 0x7f6dac138a50 auth_method 0 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): get_auth_request con 0x7f6dac139540 auth_method 0 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): get_auth_request con 0x7f6dac0a5bb0 auth_method 0 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:05.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6daa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient(hunting): handle_auth_done global_id 75011 payload 824 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 -- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6dac0a5bb0 msgr2=0x7f6dac09ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6dac0a5bb0 0x7f6dac09ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 -- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6dac139540 msgr2=0x7f6dac09c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6dac139540 0x7f6dac09c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 monclient: found mon.noname-h 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.164+0000 7f6da9d9b640 1 -- 172.21.15.43:0/1227535292 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dac09fc90 con 0x7f6dac138a50 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6da9d9b640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 0x7f6dac09bfb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6da001cca0 tx=0x7f6da00191a0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 client.?.objecter ms_handle_connect 0x7f6dac138a50 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6da00b0040 con 0x7f6dac138a50 2024-04-19T01:03:05.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: dump: 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:05.183 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: _finish_auth 0 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 5 monclient: authenticate success, global_id 75011 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 monclient: _renew_subs 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dac09ee60 con 0x7f6dac138a50 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da00036e0 con 0x7f6dac138a50 2024-04-19T01:03:05.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6da8d99640 4 set_mon_vals no callback set 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 client.75011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 monclient: _renew_subs 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dac134760 con 0x7f6dac138a50 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 1 librados: init done 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6da0003d00 con 0x7f6dac138a50 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 monclient: dump: 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.185 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:05.186 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dac0008e0 con 0x7f6dac138a50 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6da0003890 con 0x7f6dac138a50 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(695..695 src has 501..695) ==== 6356+0+0 (secure 0 0 0) 0x7f6da00e2e50 con 0x7f6dac138a50 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 client.75011.objecter ms_dispatch 0x7f6dac000ba0 osd_map(695..695 src has 501..695) 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 3 client.75011.objecter handle_osd_map got epochs [695,695] > 0 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 3 client.75011.objecter handle_osd_map decoding full epoch 695 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 20 client.75011.objecter dump_active .. 0 homeless 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:05.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 10 monclient: get_auth_request con 0x7f6d8c038e10 auth_method 0 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 10 client.75011.objecter ms_handle_connect 0x7f6d8c038e10 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c007fd0 tx=0x7f6d9c007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 1 -- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 msgr2=0x7f6d8c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 1 -- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 msgr2=0x7f6d8c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c007fd0 tx=0x7f6d9c007e40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6daa59c640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c007fd0 tx=0x7f6d9c007e40 comp rx=0 tx=0).stop 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 msgr2=0x7f6d8c03b2d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:05.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.168+0000 7f6d9affd640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 unknown :-1 s=CLOSED pgs=3637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.184+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6da00160b0 con 0x7f6dac138a50 2024-04-19T01:03:05.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.184+0000 7f6d9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:05.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.184+0000 7f6d9affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:05.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.412+0000 7f6db0ddb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T01:03:05.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.412+0000 7f6db0ddb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T01:03:05.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.412+0000 7f6db0ddb640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:03:05.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.412+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6dac09e730 con 0x7f6dac138a50 2024-04-19T01:03:05.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=913052086,config=2586702516,health=928171084,kv=3951634984,logm=925328689}) 2024-04-19T01:03:05.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.416+0000 7f6d9affd640 1 -- 172.21.15.43:0/1227535292 <== mon.5 v2:172.21.15.161:3302/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6da00b4dd0 con 0x7f6dac138a50 2024-04-19T01:03:05.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.416+0000 7f6d9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T01:03:05.418 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.419 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=10,kv=12,logm=49} crc {auth=891375599,config=2586702516,health=4268280663,kv=4196696685,logm=2190121588}) 2024-04-19T01:03:05.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.416+0000 7f6d9affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:05.420 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=1473289495,config=2586702516,health=928171084,kv=4196696685,logm=1246259811}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=25,config=2,health=11,kv=12,logm=50} crc {auth=3675768892,config=2586702516,health=1126726361,kv=4196696685,logm=3215644580}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=4008899608,config=2586702516,health=928171084,kv=4196696685,logm=2469203438}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=27,config=2,health=10,kv=12,logm=49} crc {auth=891375599,config=2586702516,health=4268280663,kv=4196696685,logm=2190121588}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.421 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=1716525237,config=2586702516,health=597586315,kv=4196696685,logm=911425846}) 2024-04-19T01:03:05.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=4008899608,config=2586702516,health=210149722,kv=139348691,logm=976116647}) 2024-04-19T01:03:05.422 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.416+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=4008899608,config=2586702516,health=928171084,kv=4196696685,logm=2693310559}) 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 10 librados: watch_flush enter 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 10 librados: watch_flush exit 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 20 client.75011.objecter shutdown clearing up homeless session... 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 10 client.75011.objecter successfully canceled tick 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 10 monclient: shutdown 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 msgr2=0x7f6dac09bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 0x7f6dac09bfb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6da001cca0 tx=0x7f6da00191a0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 shutdown_connections 2024-04-19T01:03:05.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6d8c038e10 0x7f6d8c03b2d0 unknown :-1 s=CLOSED pgs=3637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6dac139540 0x7f6dac09c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f6dac138a50 0x7f6dac09bfb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 --2- 172.21.15.43:0/1227535292 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f6dac0a5bb0 0x7f6dac09ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:05.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 >> 172.21.15.43:0/1227535292 conn(0x7f6dac0a7cb0 msgr2=0x7f6dac0a8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:05.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 shutdown_connections 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963375880}) 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=440526319}) 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.427 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=65663018}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=115291765}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3323829857}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=440526319}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4194946003}) 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.428 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1016993876}) 2024-04-19T01:03:05.429 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.420+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2304335492}) 2024-04-19T01:03:05.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.420+0000 7f6db0ddb640 1 -- 172.21.15.43:0/1227535292 wait complete. 2024-04-19T01:03:05.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:05.424+0000 7f6db0ddb640 1 librados: shutdown 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2405098553}) 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089115195}) 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3330163386}) 2024-04-19T01:03:05.431 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3151761069}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1610156051}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2089115195}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3190288434}) 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.432 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=630942753}) 2024-04-19T01:03:05.433 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.428+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2162595471}) 2024-04-19T01:03:05.434 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.434 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1521337011}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2648010957}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=328479497}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3000535370}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1878000700}) 2024-04-19T01:03:05.435 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2648010957}) 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2498688055}) 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2780512453}) 2024-04-19T01:03:05.436 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.432+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2865229054}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2356098182}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1591694672}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2415608371}) 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.439 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1825478258}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=118998899}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1591694672}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1249257362}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404747726}) 2024-04-19T01:03:05.440 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.436+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3160372510}) 2024-04-19T01:03:05.442 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.442 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.442 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3743369810}) 2024-04-19T01:03:05.442 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.442 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.442 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280358415}) 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2924045968}) 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2773227485}) 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.443 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2864175927}) 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=280358415}) 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=460530189}) 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967607168}) 2024-04-19T01:03:05.444 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.440+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2292649799}) 2024-04-19T01:03:05.446 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.446 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1706788822}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616630316}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3128971780}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2833447706}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=228998868}) 2024-04-19T01:03:05.447 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1616630316}) 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2732716729}) 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019521919}) 2024-04-19T01:03:05.448 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.444+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=371027934}) 2024-04-19T01:03:05.449 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T01:03:05.450 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.450 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.450 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3782968548}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1997316359}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093482392}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3479600439}) 2024-04-19T01:03:05.451 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4244025601}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1997316359}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=462171856}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210878933}) 2024-04-19T01:03:05.452 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.448+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3890727224}) 2024-04-19T01:03:05.454 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269344491}) 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078902323}) 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3608324119}) 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.455 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2254898660}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=802478478}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4078902323}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3876852970}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954012723}) 2024-04-19T01:03:05.456 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.452+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3660839541}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852017008}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610077711}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1576659410}) 2024-04-19T01:03:05.461 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3333673878}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1952921819}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=610077711}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2608127056}) 2024-04-19T01:03:05.462 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862731505}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.456+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3568162389}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2136243025}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512487246}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=715520048}) 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.463 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1080382187}) 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1171676945}) 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3512487246}) 2024-04-19T01:03:05.464 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.465 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.465 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1251741723}) 2024-04-19T01:03:05.465 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.465 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247738083}) 2024-04-19T01:03:05.465 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=737403399}) 2024-04-19T01:03:05.471 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.471 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.471 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126882539}) 2024-04-19T01:03:05.471 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.460+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279887997}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1372442596}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=27557332}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3689533853}) 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.472 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2279887997}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4041444944}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783132454}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=224895825}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394889547}) 2024-04-19T01:03:05.473 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53143066}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177823346}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=576633068}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=313589662}) 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.474 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2584453709}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2109890625}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=940256140}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.464+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4233138727}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884322895}) 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.475 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854995182}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=689793479}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2459927432}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3897410418}) 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.476 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4290659647}) 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3312185555}) 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93531344}) 2024-04-19T01:03:05.477 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.468+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2881233792}) 2024-04-19T01:03:05.484 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858276229}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604888142}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1689590840}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.485 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=428659843}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1902281329}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=795195551}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1484241057}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222658558}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.472+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2626488394}) 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.486 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261254884}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483512168}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3548404594}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=652982278}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.487 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2561728898}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1856368850}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3063163295}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477194429}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.476+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3074444844}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.488 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101971280}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=334891390}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2302409258}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3864658882}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2201911302}) 2024-04-19T01:03:05.489 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3917675175}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1967317169}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348401392}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=864964653}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.490 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2960152671}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720411018}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3018630605}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2692670095}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3797414169}) 2024-04-19T01:03:05.491 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1856165157}) 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4176012484}) 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261341918}) 2024-04-19T01:03:05.492 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.480+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2821619358}) 2024-04-19T01:03:05.497 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.497 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.497 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4025576996}) 2024-04-19T01:03:05.497 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.497 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=537256508}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3546433124}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3281239679}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3835106603}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3339249225}) 2024-04-19T01:03:05.498 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=714658910}) 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820067460}) 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.484+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1026886046}) 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1343140408}) 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.499 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2416928721}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1539776409}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3787396528}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.500 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2413344715}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=610914553}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4100695815}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402446959}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.488+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2481733925}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084305526}) 2024-04-19T01:03:05.501 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=143639044}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=832097964}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2396666422}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2855402708}) 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.502 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3546593135}) 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3182637572}) 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188975625}) 2024-04-19T01:03:05.503 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.496+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2201792828}) 2024-04-19T01:03:05.524 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020818441}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=579780385}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823714659}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.520+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3861934244}) 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.525 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=294273885}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2792027650}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=339084870}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31465204}) 2024-04-19T01:03:05.526 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.524+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2496279403}) 2024-04-19T01:03:05.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122067729}) 2024-04-19T01:03:05.544 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1679722317}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=642909439}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4292060116}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3930917760}) 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.545 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3385923000}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3467324073}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791774496}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2051121458}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313190396}) 2024-04-19T01:03:05.546 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2066964486}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2421531227}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3431883033}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=441633642}) 2024-04-19T01:03:05.547 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1396608972}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=821694296}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362977707}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.528+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1533896770}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177991919}) 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.548 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264574400}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1742799795}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1062546827}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=653859817}) 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.549 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2080944358}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1392089905}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475098006}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2090214246}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003317767}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=776191306}) 2024-04-19T01:03:05.550 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1476827619}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2227187237}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=84750}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.551 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3381411736}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2858438753}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315057130}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1432571635}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432144019}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616173209}) 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.552 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1551842526}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3548221853}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2640634932}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4088988066}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.553 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3865454874}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876666195}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.532+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3334066569}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3843101728}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1928045294}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3175967445}) 2024-04-19T01:03:05.554 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2713854590}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2568323423}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3231096313}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=152191558}) 2024-04-19T01:03:05.555 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=524163737}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1830731127}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3050034572}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935660230}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=486861583}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.556 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=294770086}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3776630453}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2106115404}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1286954239}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712874589}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.536+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2770223245}) 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.557 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2520510951}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158217276}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4003491673}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2728135942}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2885576042}) 2024-04-19T01:03:05.558 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2532649434}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1017057199}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465384511}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1206014902}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890181890}) 2024-04-19T01:03:05.559 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=132120256}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=830628600}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=802796157}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2967599608}) 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.560 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3162320010}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1966036002}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521391965}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.540+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3489547514}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901572297}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3198025567}) 2024-04-19T01:03:05.561 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3595212444}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1151291601}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=480326437}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.562 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=793545256}) 2024-04-19T01:03:05.563 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.563 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.563 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=125617419}) 2024-04-19T01:03:05.563 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.563 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2414017596}) 2024-04-19T01:03:05.563 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2571180800}) 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128094825}) 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=517698545}) 2024-04-19T01:03:05.791 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3506355058}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=972344804}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2588186875}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1406629013}) 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.792 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1130601844}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=286516352}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3612478971}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747545172}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204890894}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.793 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1473887401}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1290652626}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3851975518}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2019771974}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4129929110}) 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.794 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770461338}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.544+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2473286845}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223995070}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507667554}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1174796072}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4003258028}) 2024-04-19T01:03:05.795 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2216533109}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3619224416}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1377641525}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063938121}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4082021227}) 2024-04-19T01:03:05.796 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2466702663}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922690968}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3299084387}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=369966232}) 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.797 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3836293146}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1564402850}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=75832841}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371377800}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.548+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2400132751}) 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.798 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132081245}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2139621428}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=709642448}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2149543098}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.799 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2190445667}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=387507925}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2399024714}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642454932}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1938760530}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13825178}) 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.800 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1203341879}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3426373430}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1470601574}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2623406497}) 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.801 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3391716674}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1982703518}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832663479}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.552+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3764701182}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377086393}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.802 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=48115870}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=14846856}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=673713389}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2259116728}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1353910993}) 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.803 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3856201763}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722787545}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=189004349}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1373083861}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223215343}) 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.804 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1172538940}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2157927122}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1112193267}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=785223543}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.805 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2367603530}) 2024-04-19T01:03:05.806 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.806 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=784053885}) 2024-04-19T01:03:05.806 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2387244668}) 2024-04-19T01:03:05.806 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.809 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3993127060}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1844003105}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3566791389}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=529440346}) 2024-04-19T01:03:05.810 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3996516415}) 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1409539847}) 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=206487563}) 2024-04-19T01:03:05.811 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666771520}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.556+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1827102926}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1853768295}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787441920}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2184658370}) 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.812 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3537623396}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3928596820}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3197899581}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2294503798}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121582784}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2331633585}) 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.813 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641648834}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1840660186}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=824606689}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1545340032}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1468997923}) 2024-04-19T01:03:05.814 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2911400252}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1171500231}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669102876}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.560+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=141900092}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794826952}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.815 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153699507}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4112358485}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1392749175}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3358037031}) 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.816 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1576106404}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3644987231}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969712728}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=43627056}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520242382}) 2024-04-19T01:03:05.817 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1164717156}) 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3831351578}) 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.818 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4089912422}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=249743151}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1015221815}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3209481640}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361818284}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.564+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3505119166}) 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.819 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762204551}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340337480}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4047528888}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3824537287}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.820 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2893488348}) 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1129990171}) 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=163098843}) 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838186572}) 2024-04-19T01:03:05.821 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1632466005}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726588634}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846430497}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=708745686}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.822 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1333319614}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2263452680}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4251101161}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3704638542}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610552884}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1473086897}) 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.823 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081579082}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986492121}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=971947498}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1380824468}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4068633328}) 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.824 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=671810988}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=62229568}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147694543}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.568+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2455937008}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341875512}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1332827834}) 2024-04-19T01:03:05.825 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2670514253}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2388927377}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2211359137}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4054338802}) 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.826 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3684590695}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=653464266}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2211083954}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511871036}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2704356421}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=394742732}) 2024-04-19T01:03:05.827 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3490367679}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1423480643}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1375136263}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3202376239}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295300038}) 2024-04-19T01:03:05.828 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2281580361}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.572+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2736743180}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407029978}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3505284199}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2084010721}) 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.829 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2931785120}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=228375359}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4278865870}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680913981}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1627043456}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781079841}) 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.830 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459070291}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4007819291}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2692570322}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1823093418}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.831 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1457833655}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1278714326}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219149904}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1103101875}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183322775}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3023925820}) 2024-04-19T01:03:05.832 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2052527380}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1698494673}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=364811140}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2910967056}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=750089369}) 2024-04-19T01:03:05.833 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086484030}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.576+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=723043201}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792068118}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3458109689}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=424172947}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.834 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3122104528}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1972105048}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1845362274}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=984667070}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972008758}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=302984762}) 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.835 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564476389}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=714009100}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4108896527}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4120189098}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.836 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3213083813}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1221582448}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1290329739}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651846851}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.580+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2563427464}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4176241859}) 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.837 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3652874934}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=156279934}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2296384000}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3400795208}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1292383124}) 2024-04-19T01:03:05.838 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2400840371}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=108818213}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=316357207}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927741811}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3537415878}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1852133695}) 2024-04-19T01:03:05.839 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3399567061}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=919057631}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=973450752}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=781762635}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948825555}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2543320912}) 2024-04-19T01:03:05.840 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2771352385}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457175567}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3422876506}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3436641703}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.841 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=562147932}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2780835173}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4087737414}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1887531771}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.584+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=836020096}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.842 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424211538}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573178845}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2137326520}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=770646016}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2243367670}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.843 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=637880857}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=840038308}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3083225262}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2338121955}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553093264}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787684526}) 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.844 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3941488082}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=303386849}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3132992574}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=67035669}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1841646599}) 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.845 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176307047}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.588+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3379100041}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1201780726}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=687798416}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=172224110}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2891565836}) 2024-04-19T01:03:05.846 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1409190627}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2295618443}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3606968505}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817183120}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3350199598}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.847 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3726014188}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901115369}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4205584588}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3937519815}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=869657833}) 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.848 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2101834915}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3210150506}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280740215}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=517039523}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=929092574}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=756334987}) 2024-04-19T01:03:05.849 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=623322261}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=524927092}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3623005824}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.592+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=229596533}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.850 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=438918030}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626510453}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3630552920}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899180294}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2105843911}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1932608529}) 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.851 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2111544824}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=969125793}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1360426781}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2761832275}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163409455}) 2024-04-19T01:03:05.852 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2063924477}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479324788}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2883495372}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1209144242}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1946642710}) 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.853 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2376223654}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3590391791}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=923492584}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.596+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719697143}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2550528346}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=629755196}) 2024-04-19T01:03:05.854 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2981221223}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3981013364}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=729200987}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1321129491}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.855 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3925616141}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=328508892}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919071257}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3119088632}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489659501}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.856 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1396572991}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1478449085}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1653879602}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3095106596}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=98397051}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.857 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=175388012}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2382142586}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.600+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2122826573}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443102063}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969445951}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3344913297}) 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.858 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2699821840}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4248295584}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=436409217}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2851451080}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546696496}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3912490675}) 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.859 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1371669246}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430599386}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1951461277}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=640946535}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2167599424}) 2024-04-19T01:03:05.860 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=784244802}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2581853515}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534455041}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.604+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3995815325}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2207508050}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.861 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2171857495}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2690231373}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1279505426}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3469101833}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1623189001}) 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.862 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=435599265}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322313493}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=292381170}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265236754}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996796222}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=690315702}) 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.863 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1641146047}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3523762589}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=586072476}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1665782294}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309884493}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3513375564}) 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.864 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203309534}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.608+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=448816331}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4154229589}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3696697570}) 2024-04-19T01:03:05.865 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2246868374}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2211872369}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3681558654}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248432664}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2101215855}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.866 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271122005}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352062109}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3135105922}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2541158994}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4047220759}) 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.867 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4185466632}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1692455930}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497102903}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2171210003}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060143364}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2711043580}) 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.868 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.612+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758739237}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3291084442}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2749101647}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=319529310}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1673282214}) 2024-04-19T01:03:05.869 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1575424691}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2819482980}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3913468193}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514108635}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373950387}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.870 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3199020369}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2794140855}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1545711074}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4029255653}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233603899}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2349644934}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.871 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110985869}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192093642}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2359885145}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3170208849}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3323290227}) 2024-04-19T01:03:05.872 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4050664418}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3689103300}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=114354372}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.616+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3766126133}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840637706}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.873 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1564599029}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4022145077}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3567834328}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=115709712}) 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.874 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2203971435}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2986349178}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227200761}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3206897515}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156542585}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.875 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=631289970}) 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2075089731}) 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=299729332}) 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.876 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2862269044}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2209037161}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1368571632}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072746447}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.620+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=884437151}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.877 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839137102}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3482307016}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3592591460}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3406260775}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1092274554}) 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.878 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2291884257}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2828173139}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12013152}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1638854157}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677280780}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2232163205}) 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.879 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3144784374}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2586345508}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4049837377}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.880 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2723623969}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3924999533}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516854747}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.624+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1831635009}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270844194}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728490325}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.881 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=533035388}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=567289781}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1059879749}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=574329820}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1665946229}) 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.882 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207960499}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3755411462}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704404691}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2547251927}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682879137}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2060629047}) 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.883 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3536059279}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2103036341}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1396824226}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2174461156}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.628+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2335064906}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589974811}) 2024-04-19T01:03:05.884 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644945138}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3540049368}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1673921546}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=749689492}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.885 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=18151195}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2834766293}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=651061047}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3329455060}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356912333}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=774852219}) 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.886 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3387912774}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1864215128}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=476187642}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3800905163}) 2024-04-19T01:03:05.888 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1932293463}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477899814}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.632+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2837388833}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652288213}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196210672}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3047746278}) 2024-04-19T01:03:05.889 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1363220026}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1557478151}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2973755869}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3355228347}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457662295}) 2024-04-19T01:03:05.890 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1881947940}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663329495}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2446010736}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2150009373}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2088872852}) 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.891 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3101641748}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2014291475}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3020179622}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021954661}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.636+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=968255721}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212380969}) 2024-04-19T01:03:05.892 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983799263}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=972357505}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3242037358}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2129589605}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=238081983}) 2024-04-19T01:03:05.893 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2537297939}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51822334}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=722666527}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2282618013}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925492835}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.894 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3911373262}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1178399641}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2348816337}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1226017401}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=442788480}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346526683}) 2024-04-19T01:03:05.895 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=833848145}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=713262405}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.640+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903706552}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4187315997}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2726981259}) 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.896 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2034665617}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4150106965}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=581881307}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280137958}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1142012055}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=191162329}) 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.897 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3730443270}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1546108881}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3212139146}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2099008707}) 2024-04-19T01:03:05.898 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4257924267}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3069123420}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182340497}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1223792184}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.644+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220634369}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399821224}) 2024-04-19T01:03:05.899 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=680693364}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2838244383}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4147459678}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3356141215}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.900 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1312287062}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014686180}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2679395931}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580894023}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=490153587}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2551614660}) 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.901 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3892338756}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1878860966}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4121628150}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=501656551}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199844862}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.648+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=92528879}) 2024-04-19T01:03:05.902 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3398705977}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750071173}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3800519796}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3940457799}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.903 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2916716538}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1248969349}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1248105587}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098624663}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2092208265}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1317895819}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.904 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514507472}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=649595095}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4060398533}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4128153643}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2010304054}) 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.905 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3316034426}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282069799}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.652+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2780791210}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723359463}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657557741}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.906 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3601651621}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1809431847}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2646177827}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1116172033}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4246921399}) 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.907 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=297511266}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1415139490}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2726616791}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1857025219}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2191554856}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2877822278}) 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.908 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2284833612}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3867816542}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3669463408}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394221195}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.656+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3996243490}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360152574}) 2024-04-19T01:03:05.909 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922129227}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3399402937}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2035618196}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2540861291}) 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.910 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1641258845}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2473105146}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228522883}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2737335723}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476565018}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627942283}) 2024-04-19T01:03:05.911 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1898430609}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3222093480}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2430925874}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=16483943}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=187022945}) 2024-04-19T01:03:05.912 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919395752}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.660+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3674523392}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087380606}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2214191045}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1691608450}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2700690851}) 2024-04-19T01:03:05.913 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=539241085}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2884175190}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1344441406}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244605362}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3085484357}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862238349}) 2024-04-19T01:03:05.914 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1590926542}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3677647619}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=226604160}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=408884733}) 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.915 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2642993561}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2609510457}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756715731}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.664+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4131816735}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=31433303}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665042387}) 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.916 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1241955282}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2790573681}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1647989500}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3430334674}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3074489256}) 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.917 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218152204}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2797423389}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666022060}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2029493795}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1768990789}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.918 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4287281707}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2448634996}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=793975951}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2837576457}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2563603723}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.668+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2492268969}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250952987}) 2024-04-19T01:03:05.919 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4256546787}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3670209128}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3589290690}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=474578384}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.920 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4023538668}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4244248096}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766120034}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3177245102}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453360860}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3115707543}) 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.921 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1410196242}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4094395463}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1205588082}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2328185669}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.922 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1676747991}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458670996}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2301619823}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928927716}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=601112602}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2366367024}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.923 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2783432793}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1688814733}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=731370104}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=81922338}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=545381762}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.672+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3576864315}) 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.924 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1919919900}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2831491164}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=952251864}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2606205529}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=491069115}) 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.925 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3340311288}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=835668715}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039319594}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4018792590}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1245036639}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1740461434}) 2024-04-19T01:03:05.926 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2812480630}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3155478575}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3004769816}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4283139986}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1247330142}) 2024-04-19T01:03:05.927 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303143061}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.676+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1208685997}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=12709962}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=418275498}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2646580269}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3980099791}) 2024-04-19T01:03:05.928 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=649044725}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1708828251}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3810773261}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316723770}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2846688595}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.929 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688304700}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=812837979}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1405331096}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=288784784}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=338786167}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.930 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=943165503}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3591465716}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401531058}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.680+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4095484722}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603486925}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1445767101}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.931 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4278486091}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1310182820}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=961345717}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4077864414}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2680711166}) 2024-04-19T01:03:05.932 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422138097}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=851363673}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379258245}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=643643815}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2248658009}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.933 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2194968029}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1692837004}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=48770738}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1147817050}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625481307}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.684+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2072093991}) 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.934 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584391152}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1121184349}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1475945894}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1746659628}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2120391338}) 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.935 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2917095779}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3712744446}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038833797}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=85495593}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568696422}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787147275}) 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.936 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2269401794}) 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3395162204}) 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=281869531}) 2024-04-19T01:03:05.937 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=569194085}) 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.688+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3928894897}) 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3995809198}) 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2235288776}) 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.938 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473136517}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1343894234}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1993129045}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=785922786}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3553624038}) 2024-04-19T01:03:05.939 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2726526875}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1470085073}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691924958}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1868023649}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129954496}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029313364}) 2024-04-19T01:03:05.940 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2051849731}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3218133000}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3076487671}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2231165137}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.941 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3614027280}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.692+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=170174153}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1334409047}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853468664}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3859864158}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1450686482}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.942 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2502417597}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4086325150}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=140699853}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1649488663}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2968887763}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1376583806}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.943 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1533839052}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=220841683}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1302903083}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1607390079}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=726565061}) 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.944 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2086885099}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1025782829}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266736324}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.696+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2288669360}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=114254864}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1562393853}) 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.945 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=986858366}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2233926679}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3779334649}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=72661514}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3084653635}) 2024-04-19T01:03:05.946 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976870100}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3038361369}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1346336723}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917375152}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1247566804}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=221387651}) 2024-04-19T01:03:05.947 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3122200592}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3763845937}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=154599048}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777460157}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.700+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=790210654}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650896933}) 2024-04-19T01:03:05.948 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=814852145}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=360885706}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1181787187}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2225800958}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.949 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=116105719}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2731835254}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596210528}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=62572687}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=549445495}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2595433828}) 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.950 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.704+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=708636464}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=796414787}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2670772017}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3426813430}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1913582118}) 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.951 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007949310}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3262779429}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3220827456}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376520300}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1690179989}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2240794609}) 2024-04-19T01:03:05.952 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=125880996}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3917410758}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1102317521}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375282463}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3399739866}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.708+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1702163635}) 2024-04-19T01:03:05.953 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047405275}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=810153407}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1052168591}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2775380481}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.954 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2912184628}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2264026108}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603362002}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3980923667}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121029734}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1259399983}) 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.955 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2214276424}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=942479745}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1715958035}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3695512523}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3898241395}) 2024-04-19T01:03:05.956 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404222302}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=287423361}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106846563}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671502776}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1835431228}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.712+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1708063129}) 2024-04-19T01:03:05.957 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4228125972}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3303381310}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1668021675}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3625743910}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2393432459}) 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.958 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678528570}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318642756}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3687367658}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1185838449}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3144451851}) 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.959 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3901990796}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=944337715}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090563286}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3926709670}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2199845546}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719339838}) 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.960 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3503052277}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=965022679}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2030124044}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3070259960}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4261912671}) 2024-04-19T01:03:05.961 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2172154812}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.716+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3320774911}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659373977}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657083005}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1615880395}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.962 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=575333407}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3737034854}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=656986695}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2399451904}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865553624}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.720+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4096874871}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.963 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4078131078}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=558816342}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=652548594}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3233176285}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=397188806}) 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.964 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1392830128}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2854263310}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2225982781}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3938276687}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370695214}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979581309}) 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.965 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=609999169}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1153037318}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2959016111}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3545040766}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2580573346}) 2024-04-19T01:03:05.966 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=145595129}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.724+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1695604815}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170540140}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=179088382}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1698414245}) 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.967 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=957433054}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4181369289}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=160059152}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=650074846}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767976198}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1976226219}) 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.968 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251482874}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2097974499}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=805371472}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2851704665}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1192683349}) 2024-04-19T01:03:05.969 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1877293543}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1125301156}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669223324}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=22107908}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3396223528}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3019984584}) 2024-04-19T01:03:05.970 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2552517420}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.728+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3845990053}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1941370632}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=719544086}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.971 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2372727203}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215978345}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1635453027}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674232142}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2502637418}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2687263834}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.972 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1836238442}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2619759848}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3676950216}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1527348267}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584665895}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2217317876}) 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.973 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110785525}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106340122}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3627788011}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.974 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1455490873}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1339340359}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2453526059}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1853796480}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027241357}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.732+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3878686774}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.975 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892758038}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471140295}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=593675836}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2425638405}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=556064877}) 2024-04-19T01:03:05.976 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4189333318}) 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2779594542}) 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.977 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060060427}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1963673993}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1288919130}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524753765}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3215387922}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.978 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2736121423}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4214647075}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1419004225}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1182645156}) 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.979 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685488898}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=116255597}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712771708}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220354116}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2006291083}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=189206545}) 2024-04-19T01:03:05.980 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=209284133}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1982064232}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=351361048}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.736+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2428937172}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=227374747}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.981 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=2389497733,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2609489162}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=4095310349,mdsmap=3764978221,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2609489162}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=85,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=506161923,mdsmap=2840952586,mgr=4027397349,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2609489162}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=1436305139,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=429449293}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=3468380907,mdsmap=2840952586,mgr=2803061145,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=4271225260}) 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.982 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=85,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=4} crc {logm=3901688450,mdsmap=2840952586,mgr=2803061145,mgr_metadata=2750671369,mgrstat=321805460}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=85,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=4178822466,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=4271225260}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=2} crc {logm=537781481,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3810081778}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=87,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=2} crc {logm=2263631440,mdsmap=1401145748,mgr=4237663773,mgr_metadata=2750671369,mgrstat=4271225260}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=17,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=35,osdmap=32} crc {mgrstat=415382491,mon_config_key=4212284759,monmap=3140514009,osd_metadata=438442333,osd_pg_creating=2320266033,osd_snap=69939280,osdmap=2009881171}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=32} crc {mgrstat=415382491,mon_config_key=2224109317,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=2649587201,osdmap=1300097507}) 2024-04-19T01:03:05.983 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mgrstat=7,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=29} crc {mgrstat=2380245761,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=2580924278,osdmap=3348479478}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=37,osdmap=30} crc {mgrstat=415382491,mon_config_key=3051443354,monmap=4080741109,osd_metadata=9197251,osd_pg_creating=2320266033,osd_snap=2580924278,osdmap=2351446889}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.740+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=31} crc {mgrstat=415382491,mon_config_key=2352145932,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3798196657,osdmap=3951238129}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mgrstat=8,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=29} crc {mgrstat=2698135813,mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=10552284,osdmap=4103636205}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.984 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=31} crc {mgrstat=415382491,mon_config_key=1815755404,monmap=157429745,osd_metadata=1091806040,osd_pg_creating=2320266033,osd_snap=523764658,osdmap=378627418}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=30} crc {mgrstat=415382491,mon_config_key=825420,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=3061618503,osdmap=4276989595}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mgrstat=7,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=31} crc {mgrstat=146996376,mon_config_key=3114496400,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=2320266033,osd_snap=2580924278,osdmap=2534349412}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2947386902}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4227164366}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=2960887068}) 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.985 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=2424875149}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=3665721512}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=3491378274}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=690303636}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2036766298}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.744+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=2762876455}) 2024-04-19T01:03:05.986 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=557373194}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2584858517}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=100} crc {osdmap=940754227}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=100} crc {osdmap=3410025448}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=100} crc {osdmap=890569555}) 2024-04-19T01:03:05.987 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=100} crc {osdmap=2070390865}) 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=100} crc {osdmap=4089229823}) 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.988 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=315346054}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.748+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=100} crc {osdmap=4092448298}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=71} crc {osdmap=2499381092}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=39342274}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=70} crc {osdmap=3714987496}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=67} crc {osdmap=267762000}) 2024-04-19T01:03:05.989 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=71} crc {osdmap=2897479825}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=70} crc {osdmap=1913340422}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=69} crc {osdmap=1469084669}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=66} crc {osdmap=2835832886}) 2024-04-19T01:03:05.990 INFO:tasks.ceph.mon.b.smithi043.stderr:2024-04-19T01:03:05.760+0000 7fefddb70640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=70} crc {osdmap=556431949}) 2024-04-19T01:03:06.450 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T01:03:06.450 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:03:06.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.596+0000 7fe2a257e640 10 monclient: get_monmap_and_config 2024-04-19T01:03:06.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.596+0000 7fe2a257e640 10 monclient: build_initial_monmap 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.596+0000 7fe2a257e640 10 monclient: monmap: 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:06.601776+0000 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:06.601776+0000 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:06.600 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 Processor -- start 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 -- start start 2024-04-19T01:03:06.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient: init 2024-04-19T01:03:06.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:06.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient: _add_conns ranks=[9,1,3,4,10,2,0,5,7,6,8] 2024-04-19T01:03:06.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe29c0a2930 0x7fe29c09d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:06.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): picked mon.noname-a con 0x7fe29c0a2930 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:06.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fe29c0a2d10 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c13be80 0x7fe29c13e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): picked mon.noname-g con 0x7fe29c13be80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe29c0a0a60 con 0x7fe29c0a2d10 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c09c5c0 con 0x7fe29c0a2930 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c09c8a0 con 0x7fe29c13be80 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe2a257e640 10 monclient(hunting): authenticate will time out at 2056.729255s 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe29c0a2930 0x7fe29c09d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe29c0a2930 0x7fe29c09d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44962/0 (socket says 172.21.15.43:44962) 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 -- 172.21.15.43:0/3402918525 learned_addr learned my addr 172.21.15.43:0/3402918525 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): get_auth_request con 0x7fe29c0a2930 auth_method 0 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:06.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient(hunting): handle_auth_done global_id 65673 payload 274 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 -- 172.21.15.43:0/3402918525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c13be80 msgr2=0x7fe29c13e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 --2- 172.21.15.43:0/3402918525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c13be80 0x7fe29c13e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 -- 172.21.15.43:0/3402918525 >> v1:172.21.15.161:6789/0 conn(0x7fe29c0a2d10 legacy=0x7fe29c0a3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 monclient: found mon.noname-a 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:06.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.600+0000 7fe29bfff640 1 -- 172.21.15.43:0/3402918525 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c09bae0 con 0x7fe29c0a2930 2024-04-19T01:03:06.605 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.604+0000 7fe29bfff640 1 --2- 172.21.15.43:0/3402918525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2930 0x7fe29c09d840 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe284009870 tx=0x7fe28401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32869099f6e551a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 1 -- 172.21.15.43:0/3402918525 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe2840b0070 con 0x7fe29c0a2930 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 10 monclient: dump: 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:06.664 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 10 monclient: _finish_auth 0 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:06.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 1 -- 172.21.15.43:0/3402918525 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2840aa810 con 0x7fe29c0a2930 2024-04-19T01:03:06.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:06.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 5 monclient: authenticate success, global_id 65673 2024-04-19T01:03:06.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:06.666 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 4 set_mon_vals no callback set 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: shutdown 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- 172.21.15.43:0/3402918525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2930 msgr2=0x7fe29c09d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 --2- 172.21.15.43:0/3402918525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2930 0x7fe29c09d840 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe284009870 tx=0x7fe28401c420 comp rx=0 tx=0).stop 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- 172.21.15.43:0/3402918525 shutdown_connections 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 --2- 172.21.15.43:0/3402918525 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c13be80 0x7fe29c13e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 --2- 172.21.15.43:0/3402918525 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2930 0x7fe29c09d840 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- 172.21.15.43:0/3402918525 >> 172.21.15.43:0/3402918525 conn(0x7fe29c0a7520 msgr2=0x7fe29c0a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- 172.21.15.43:0/3402918525 shutdown_connections 2024-04-19T01:03:06.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- 172.21.15.43:0/3402918525 wait complete. 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: build_initial_monmap 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: monmap: 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:06.672 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 librados: starting msgr at 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 librados: starting objecter 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 Processor -- start 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- start start 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 librados: setting wanted keys 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 librados: calling monclient init 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: init 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient: _add_conns ranks=[3,0,7,1,4,8,6,5,2] 2024-04-19T01:03:06.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c0a2930 0x7fe29c1d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): picked mon.noname-g con 0x7fe29c0a2930 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 0x7fe29c134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): picked mon.noname-a con 0x7fe29c0a2d10 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe29c13be80 0x7fe29c09e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): picked mon.noname-i con 0x7fe29c13be80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c141360 con 0x7fe29c0a2d10 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c1411e0 con 0x7fe29c0a2930 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe29c1414e0 con 0x7fe29c13be80 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:06.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a257e640 10 monclient(hunting): authenticate will time out at 2056.795285s 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 0x7fe29c134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 0x7fe29c134ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:44970/0 (socket says 172.21.15.43:44970) 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 -- 172.21.15.43:0/2570183550 learned_addr learned my addr 172.21.15.43:0/2570183550 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): get_auth_request con 0x7fe29c0a2d10 auth_method 0 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient(hunting): handle_auth_done global_id 65682 payload 824 2024-04-19T01:03:06.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29bfff640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c0a2930 0x7fe29c1d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe2a0b1b640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe29c13be80 0x7fe29c09e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 -- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe29c13be80 msgr2=0x7fe29c09e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe29c13be80 0x7fe29c09e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 -- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c0a2930 msgr2=0x7fe29c1d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c0a2930 0x7fe29c1d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 monclient: found mon.noname-a 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:06.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 -- 172.21.15.43:0/2570183550 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c09bec0 con 0x7fe29c0a2d10 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe29b7fe640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 0x7fe29c134ab0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c7c0 tx=0x7fe28c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe298ff9640 10 client.?.objecter ms_handle_connect 0x7fe29c0a2d10 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.664+0000 7fe298ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe28c016020 con 0x7fe29c0a2d10 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: dump: 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:06.677 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: _finish_auth 0 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:03:06.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe28c003e10 con 0x7fe29c0a2d10 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe28c00ac60 con 0x7fe29c0a2d10 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 monclient: dump: 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:06.679 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe29a7fc640 4 set_mon_vals no callback set 2024-04-19T01:03:06.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 5 monclient: authenticate success, global_id 65682 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 monclient: _renew_subs 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c0a0050 con 0x7fe29c0a2d10 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 client.65682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 monclient: _renew_subs 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c13a360 con 0x7fe29c0a2d10 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 1 librados: init done 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe28c019070 con 0x7fe29c0a2d10 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe27c038e70 0x7fe27c03b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:06.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(696..696 src has 501..696) ==== 6729+0+0 (secure 0 0 0) 0x7fe28c048d40 con 0x7fe29c0a2d10 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 client.65682.objecter ms_dispatch 0x7fe29c000ba0 osd_map(696..696 src has 501..696) 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe29c0693b0 con 0x7fe29c0a2d10 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 3 client.65682.objecter handle_osd_map got epochs [696,696] > 0 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 3 client.65682.objecter handle_osd_map decoding full epoch 696 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 20 client.65682.objecter dump_active .. 0 homeless 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe29bfff640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe27c038e70 0x7fe27c03b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe29bfff640 10 monclient: get_auth_request con 0x7fe27c038e70 auth_method 0 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe298ff9640 10 client.65682.objecter ms_handle_connect 0x7fe27c038e70 2024-04-19T01:03:06.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.668+0000 7fe29bfff640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe27c038e70 0x7fe27c03b330 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7fe2840ad000 tx=0x7fe2840a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:06.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.672+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe28c00a7f0 con 0x7fe29c0a2d10 2024-04-19T01:03:06.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.672+0000 7fe298ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:06.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.672+0000 7fe298ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:06.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe2a257e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:03:06.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe2a257e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:03:06.916 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe2a257e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:06.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe29c076520 con 0x7fe29c0a2d10 2024-04-19T01:03:06.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe298ff9640 1 -- 172.21.15.43:0/2570183550 <== mon.0 v2:172.21.15.43:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7fe28c010dc0 con 0x7fe29c0a2d10 2024-04-19T01:03:06.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe298ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:03:06.917 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.916+0000 7fe298ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:06.917 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":118,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 10 librados: watch_flush enter 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 10 librados: watch_flush exit 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 20 client.65682.objecter shutdown clearing up homeless session... 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 10 client.65682.objecter successfully canceled tick 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe27c038e70 msgr2=0x7fe27c03b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe27c038e70 0x7fe27c03b330 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7fe2840ad000 tx=0x7fe2840a6000 comp rx=0 tx=0).stop 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 10 monclient: shutdown 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 msgr2=0x7fe29c134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 0x7fe29c134ab0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c7c0 tx=0x7fe28c00cc80 comp rx=0 tx=0).stop 2024-04-19T01:03:06.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 shutdown_connections 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe27c038e70 0x7fe27c03b330 unknown :-1 s=CLOSED pgs=3638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe29c13be80 0x7fe29c09e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe29c0a2d10 0x7fe29c134ab0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 --2- 172.21.15.43:0/2570183550 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fe29c0a2930 0x7fe29c1d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 >> 172.21.15.43:0/2570183550 conn(0x7fe29c0a7520 msgr2=0x7fe29c13cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 shutdown_connections 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 -- 172.21.15.43:0/2570183550 wait complete. 2024-04-19T01:03:06.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:06.920+0000 7fe2a257e640 1 librados: shutdown 2024-04-19T01:03:06.936 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T01:03:06.937 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-19T01:03:06.937 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T01:03:06.937 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:03:07.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient: get_monmap_and_config 2024-04-19T01:03:07.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient: monmap: 2024-04-19T01:03:07.063 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:07.066703+0000 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:07.066703+0000 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.064 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.065 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:07.065 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:07.065 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 Processor -- start 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 -- start start 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient: init 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient: _add_conns ranks=[8,3,2,6,4,9,5,0,1,7,10] 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 0x7fbdac1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): picked mon.noname-e con 0x7fbdac138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbdac0a5b80 0x7fbdac1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): picked mon.noname-g con 0x7fbdac0a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 0x7fbdac135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): picked mon.noname-b con 0x7fbdac1358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fbdac0a4410 con 0x7fbdac1358b0 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbdac139b30 con 0x7fbdac0a5b80 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fbdac0a3dc0 con 0x7fbdac138cd0 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb46cd640 10 monclient(hunting): authenticate will time out at 2057.193844s 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 0x7fbdac1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 0x7fbdac135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 0x7fbdac135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60814/0 (socket says 172.21.15.43:60814) 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 0x7fbdac1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52750/0 (socket says 172.21.15.43:52750) 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 -- 172.21.15.43:0/3693474236 learned_addr learned my addr 172.21.15.43:0/3693474236 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): get_auth_request con 0x7fbdac1358b0 auth_method 0 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): get_auth_request con 0x7fbdac138cd0 auth_method 0 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb1c68640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbdac0a5b80 0x7fbdac1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient(hunting): handle_auth_done global_id 144434 payload 274 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 -- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 msgr2=0x7fbdac1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 0x7fbdac1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 -- 172.21.15.43:0/3693474236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbdac0a5b80 msgr2=0x7fbdac1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbdac0a5b80 0x7fbdac1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 monclient: found mon.noname-b 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2c6a640 1 -- 172.21.15.43:0/3693474236 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac0a40a0 con 0x7fbdac1358b0 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb1c68640 10 monclient: get_auth_request con 0x7fbdac0a5b80 auth_method 0 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb2469640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 0x7fbdac1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:07.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.064+0000 7fbdb1c68640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbdac0a5b80 0x7fbdac1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb2c6a640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 0x7fbdac135d30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbda80098a0 tx=0x7fbda801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db5b84773d40e28a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 1 -- 172.21.15.43:0/3693474236 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbda80b0070 con 0x7fbdac1358b0 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: dump: 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.072 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.073 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb46cd640 5 monclient: authenticate success, global_id 144434 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb46cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 1 -- 172.21.15.43:0/3693474236 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbda80aa810 con 0x7fbdac1358b0 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb46cd640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb46cd640 4 set_mon_vals no callback set 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 1 -- 172.21.15.43:0/3693474236 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbda80a9300 con 0x7fbdac1358b0 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.068+0000 7fbdb1467640 10 monclient: dump: 2024-04-19T01:03:07.074 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.075 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: shutdown 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 msgr2=0x7fbdac135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 0x7fbdac135d30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbda80098a0 tx=0x7fbda801c3e0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3693474236 shutdown_connections 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fbdac1358b0 0x7fbdac135d30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbdac0a5b80 0x7fbdac1395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3693474236 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fbdac138cd0 0x7fbdac1390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3693474236 >> 172.21.15.43:0/3693474236 conn(0x7fbdac0ab2f0 msgr2=0x7fbdac0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3693474236 shutdown_connections 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3693474236 wait complete. 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: monmap: 2024-04-19T01:03:07.076 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.077 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 librados: starting msgr at 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 librados: starting objecter 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 Processor -- start 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- start start 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 librados: setting wanted keys 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 librados: calling monclient init 2024-04-19T01:03:07.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: init 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: _add_conns ranks=[1,4,5,6,2,7,0,3,8] 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdac0a5b80 0x7fbdac1d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): picked mon.noname-f con 0x7fbdac0a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbdac1358b0 0x7fbdac1d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): picked mon.noname-c con 0x7fbdac1358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbdac138cd0 0x7fbdac1d1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): picked mon.noname-h con 0x7fbdac138cd0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac0a2010 con 0x7fbdac0a5b80 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fbdac0a1e90 con 0x7fbdac1358b0 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fbdac142aa0 con 0x7fbdac138cd0 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): authenticate will time out at 2057.202172s 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbdac1358b0 0x7fbdac1d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbdac1358b0 0x7fbdac1d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47546/0 (socket says 172.21.15.43:47546) 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 -- 172.21.15.43:0/3843090587 learned_addr learned my addr 172.21.15.43:0/3843090587 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): get_auth_request con 0x7fbdac1358b0 auth_method 0 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2469640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdac0a5b80 0x7fbdac1d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2469640 10 monclient(hunting): get_auth_request con 0x7fbdac0a5b80 auth_method 0 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2469640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2469640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2469640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2c6a640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbdac138cd0 0x7fbdac1d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2c6a640 10 monclient(hunting): get_auth_request con 0x7fbdac138cd0 auth_method 0 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2c6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2c6a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb2c6a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient(hunting): handle_auth_done global_id 134581 payload 824 2024-04-19T01:03:07.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbdac138cd0 msgr2=0x7fbdac1d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbdac138cd0 0x7fbdac1d1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdac0a5b80 msgr2=0x7fbdac1d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbdac0a5b80 0x7fbdac1d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 monclient: found mon.noname-c 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdac09e400 con 0x7fbdac1358b0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb1c68640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbdac1358b0 0x7fbdac1d0fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c7f0 tx=0x7fbd9c00ccb0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 client.?.objecter ms_handle_connect 0x7fbdac1358b0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbd9c015070 con 0x7fbdac1358b0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.088 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: dump: 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.089 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 5 monclient: authenticate success, global_id 134581 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: _renew_subs 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd9c007ca0 con 0x7fbdac1358b0 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdac0a3590 con 0x7fbdac1358b0 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbd9c007e50 con 0x7fbdac1358b0 2024-04-19T01:03:07.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb0c66640 4 set_mon_vals no callback set 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 client.134581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbda2ffd640 10 monclient: dump: 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.091 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: _renew_subs 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdac0a06f0 con 0x7fbdac1358b0 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 librados: init done 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 0x7fbdac134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.092 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.072+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac09fe50 con 0x7fbdac1348a0 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 0x7fbdac134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbd9c018070 con 0x7fbdac1358b0 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient: get_auth_request con 0x7fbdac1348a0 auth_method 0 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbd90038fb0 0x7fbd9003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 10 monclient(hunting): handle_auth_done global_id 65691 payload 824 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdac0693b0 con 0x7fbdac1348a0 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(697..697 src has 501..697) ==== 6742+0+0 (secure 0 0 0) 0x7fbd9c0491f0 con 0x7fbdac1358b0 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 client.134581.objecter ms_dispatch 0x7fbdac000ba0 osd_map(697..697 src has 501..697) 2024-04-19T01:03:07.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 3 client.134581.objecter handle_osd_map got epochs [697,697] > 0 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 3 client.134581.objecter handle_osd_map decoding full epoch 697 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2469640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 0x7fbdac134c80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009f40 tx=0x7fbda40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 20 client.134581.objecter dump_active .. 0 homeless 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 client.134581.objecter ms_handle_connect 0x7fbdac1348a0 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 client.134581.objecter resend_mon_ops 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2c6a640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbd90038fb0 0x7fbd9003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbda401a020 con 0x7fbdac1348a0 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 monclient: dump: 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.094 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2c6a640 10 monclient: get_auth_request con 0x7fbd90038fb0 auth_method 0 2024-04-19T01:03:07.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbda4016470 con 0x7fbdac1348a0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 msgr2=0x7fbdac134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 0x7fbdac134c80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009f40 tx=0x7fbda40115a0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 msgr2=0x7fbdac134c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1348a0 0x7fbdac134c80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbda2ffd640 10 client.134581.objecter ms_handle_connect 0x7fbd90038fb0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.076+0000 7fbdb2c6a640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbd90038fb0 0x7fbd9003b470 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7fbda80ad000 tx=0x7fbda80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb46cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 0x7fbdac13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb46cd640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbdac06d140 con 0x7fbdac1351c0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 0x7fbdac13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient: get_auth_request con 0x7fbdac1351c0 auth_method 0 2024-04-19T01:03:07.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 10 monclient(hunting): handle_auth_done global_id 65700 payload 824 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.080+0000 7fbdb2469640 1 -- 172.21.15.43:0/3843090587 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbdac09f180 con 0x7fbdac1351c0 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb2469640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 0x7fbdac13bcc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009fd0 tx=0x7fbda4004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 client.134581.objecter ms_handle_connect 0x7fbdac1351c0 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 client.134581.objecter resend_mon_ops 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbda401a050 con 0x7fbdac1351c0 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:07.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 monclient: dump: 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.098 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.099 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9444 (secure 0 0 0) 0x7fbda401f420 con 0x7fbdac1351c0 2024-04-19T01:03:07.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:07.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:07.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 msgr2=0x7fbdac13bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 0x7fbdac13bcc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009fd0 tx=0x7fbda4004a50 comp rx=0 tx=0).stop 2024-04-19T01:03:07.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 msgr2=0x7fbdac13bcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbda2ffd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbdac1351c0 0x7fbdac13bcc0 secure :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbda4009fd0 tx=0x7fbda4004a50 comp rx=0 tx=0).stop 2024-04-19T01:03:07.100 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:07.100 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:07.100 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 713355, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:07.101 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 8, 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.102 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.103 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:07.104 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.105 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.106 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.107 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.108 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.109 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:07.110 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.111 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.112 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.113 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.114 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.115 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.116 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.117 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.118 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:07.119 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.120 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.121 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.122 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.123 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.123 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.123 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:07.123 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 10 librados: watch_flush enter 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 10 librados: watch_flush exit 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 20 client.134581.objecter shutdown clearing up homeless session... 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 10 client.134581.objecter successfully canceled tick 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbd90038fb0 msgr2=0x7fbd9003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbd90038fb0 0x7fbd9003b470 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7fbda80ad000 tx=0x7fbda80a6000 comp rx=0 tx=0).stop 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 10 monclient: shutdown 2024-04-19T01:03:07.125 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbdac1358b0 msgr2=0x7fbdac1d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 --2- 172.21.15.43:0/3843090587 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbdac1358b0 0x7fbdac1d0fe0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c7f0 tx=0x7fbd9c00ccb0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb2c6a640 1 -- 172.21.15.43:0/3843090587 reap_dead start 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb2c6a640 1 -- 172.21.15.43:0/3843090587 reap_dead start 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 shutdown_connections 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 >> 172.21.15.43:0/3843090587 conn(0x7fbdac0ab2f0 msgr2=0x7fbdac1368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 shutdown_connections 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 -- 172.21.15.43:0/3843090587 wait complete. 2024-04-19T01:03:07.126 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.084+0000 7fbdb46cd640 1 librados: shutdown 2024-04-19T01:03:07.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:07.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T01:03:07.127 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.236+0000 7f64f9e6f640 10 monclient: get_monmap_and_config 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient: monmap: 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:07.242498+0000 2024-04-19T01:03:07.239 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:07.242498+0000 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:07.240 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 Processor -- start 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 -- start start 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient: init 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient: _add_conns ranks=[1,6,8,3,7,9,5,4,2,10,0] 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f64f4138cd0 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:07.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 0x7f64f413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): picked mon.noname-d con 0x7f64f40a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 0x7f64f41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): picked mon.noname-e con 0x7f64f41397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f40a3dc0 con 0x7f64f4138cd0 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f64f40a4410 con 0x7f64f40a5b80 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f64f40a40a0 con 0x7f64f41397c0 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f9e6f640 10 monclient(hunting): authenticate will time out at 2057.369587s 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 0x7f64f413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 0x7f64f41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 0x7f64f41407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52766/0 (socket says 172.21.15.43:52766) 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 0x7f64f413e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:47494/0 (socket says 172.21.15.43:47494) 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 1 -- 172.21.15.43:0/3835717197 learned_addr learned my addr 172.21.15.43:0/3835717197 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): get_auth_request con 0x7f64f41397c0 auth_method 0 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): get_auth_request con 0x7f64f40a5b80 auth_method 0 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_done global_id 144429 payload 274 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 -- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 msgr2=0x7f64f41407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 --2- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 0x7f64f41407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 -- 172.21.15.43:0/3835717197 >> v1:172.21.15.161:6789/0 conn(0x7f64f4138cd0 legacy=0x7f64f41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 monclient: found mon.noname-d 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f2ffd640 1 -- 172.21.15.43:0/3835717197 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f40a2ae0 con 0x7f64f40a5b80 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.240+0000 7f64f3fff640 1 --2- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 0x7f64f41407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 1 --2- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 0x7f64f413e270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64e80097f0 tx=0x7f64e801c3a0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=fb822a634f54ff62 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 1 -- 172.21.15.43:0/3835717197 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64e80b0070 con 0x7f64f40a5b80 2024-04-19T01:03:07.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: dump: 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.248 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 1 -- 172.21.15.43:0/3835717197 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e80aa810 con 0x7f64f40a5b80 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 1 -- 172.21.15.43:0/3835717197 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64e80a9300 con 0x7f64f40a5b80 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f27fc640 10 monclient: dump: 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.249 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 5 monclient: authenticate success, global_id 144429 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 4 set_mon_vals no callback set 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: shutdown 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 msgr2=0x7f64f413e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 0x7f64f413e270 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f64e80097f0 tx=0x7f64e801c3a0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- 172.21.15.43:0/3835717197 shutdown_connections 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f64f41397c0 0x7f64f41407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/3835717197 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f40a5b80 0x7f64f413e270 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- 172.21.15.43:0/3835717197 >> 172.21.15.43:0/3835717197 conn(0x7f64f40ab2f0 msgr2=0x7f64f40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- 172.21.15.43:0/3835717197 shutdown_connections 2024-04-19T01:03:07.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- 172.21.15.43:0/3835717197 wait complete. 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: monmap: 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:07.258 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 librados: starting msgr at 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 librados: starting objecter 2024-04-19T01:03:07.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 Processor -- start 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- start start 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 librados: setting wanted keys 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 librados: calling monclient init 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: init 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient: _add_conns ranks=[2,6,3,4,8,7,0,1,5] 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f40a5b80 0x7f64f41d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): picked mon.noname-b con 0x7f64f40a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f4138cd0 0x7f64f4136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): picked mon.noname-d con 0x7f64f4138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f64f41397c0 0x7f64f409d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): picked mon.noname-g con 0x7f64f41397c0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f64f4143730 con 0x7f64f40a5b80 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f64f40a20a0 con 0x7f64f41397c0 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f64f41438b0 con 0x7f64f4138cd0 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f9e6f640 10 monclient(hunting): authenticate will time out at 2057.375155s 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f4138cd0 0x7f64f4136c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f40a5b80 0x7f64f41d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f4138cd0 0x7f64f4136c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:47502/0 (socket says 172.21.15.43:47502) 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 1 -- 172.21.15.43:0/2284520056 learned_addr learned my addr 172.21.15.43:0/2284520056 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f40a5b80 0x7f64f41d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60828/0 (socket says 172.21.15.43:60828) 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): get_auth_request con 0x7f64f4138cd0 auth_method 0 2024-04-19T01:03:07.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): get_auth_request con 0x7f64f40a5b80 auth_method 0 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f3fff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f64f41397c0 0x7f64f409d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient(hunting): handle_auth_done global_id 144443 payload 824 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f4138cd0 msgr2=0x7f64f4136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f64f4138cd0 0x7f64f4136c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f64f41397c0 msgr2=0x7f64f409d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f64f41397c0 0x7f64f409d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 monclient: found mon.noname-b 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f409f5d0 con 0x7f64f40a5b80 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f3fff640 10 monclient: get_auth_request con 0x7f64f41397c0 auth_method 0 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f3fff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f64f41397c0 0x7f64f409d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:07.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64f37fe640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f40a5b80 0x7f64f41d0830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000f70 tx=0x7f64e400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 client.?.objecter ms_handle_connect 0x7f64f40a5b80 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64e4017070 con 0x7f64f40a5b80 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 monclient: dump: 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.264 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e4003600 con 0x7f64f40a5b80 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.244+0000 7f64dbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64e4003c60 con 0x7f64f40a5b80 2024-04-19T01:03:07.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: dump: 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.266 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f1ffb640 4 set_mon_vals no callback set 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 5 monclient: authenticate success, global_id 144443 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 monclient: _renew_subs 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f40a3590 con 0x7f64f40a5b80 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 client.144443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 monclient: _renew_subs 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f409fe80 con 0x7f64f40a5b80 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 1 librados: init done 2024-04-19T01:03:07.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 0x7f64f402c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f64f406d140 con 0x7f64f409e600 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 0x7f64f402c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f64e40037f0 con 0x7f64f40a5b80 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient: get_auth_request con 0x7f64f409e600 auth_method 0 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_done global_id 144452 payload 824 2024-04-19T01:03:07.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f64e0038e40 0x7f64e003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64f40a3040 con 0x7f64f409e600 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(697..697 src has 501..697) ==== 6742+0+0 (secure 0 0 0) 0x7f64e40487c0 con 0x7f64f40a5b80 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 client.144443.objecter ms_dispatch 0x7f64f4000ba0 osd_map(697..697 src has 501..697) 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 3 client.144443.objecter handle_osd_map got epochs [697,697] > 0 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 3 client.144443.objecter handle_osd_map decoding full epoch 697 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 20 client.144443.objecter dump_active .. 0 homeless 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f2ffd640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 0x7f64f402c450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64e80ae040 tx=0x7f64e8019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 client.144443.objecter ms_handle_connect 0x7f64f409e600 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 client.144443.objecter resend_mon_ops 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64e80b0040 con 0x7f64f409e600 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f3fff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f64e0038e40 0x7f64e003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: dump: 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.270 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f3fff640 10 monclient: get_auth_request con 0x7f64e0038e40 auth_method 0 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f64e8003620 con 0x7f64f409e600 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 msgr2=0x7f64f402c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 0x7f64f402c450 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64e80ae040 tx=0x7f64e8019140 comp rx=0 tx=0).stop 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 msgr2=0x7f64f402c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e600 0x7f64f402c450 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f64e80ae040 tx=0x7f64e8019140 comp rx=0 tx=0).stop 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64dbfff640 10 client.144443.objecter ms_handle_connect 0x7f64e0038e40 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.248+0000 7f64f3fff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f64e0038e40 0x7f64e003b300 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009440 tx=0x7f64ec00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f9e6f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:07.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 0x7f64f402c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f9e6f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f64f4068240 con 0x7f64f409e9e0 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 0x7f64f402c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient: get_auth_request con 0x7f64f409e9e0 auth_method 0 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.252+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f2ffd640 10 monclient(hunting): handle_auth_done global_id 144461 payload 824 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f2ffd640 1 -- 172.21.15.43:0/2284520056 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64f40a11b0 con 0x7f64f409e9e0 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f2ffd640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 0x7f64f402c990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64e801c3e0 tx=0x7f64e80a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 client.144443.objecter ms_handle_connect 0x7f64f409e9e0 2024-04-19T01:03:07.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 client.144443.objecter resend_mon_ops 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f64e80b5af0 con 0x7f64f409e9e0 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 monclient: dump: 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.273 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9276 (secure 0 0 0) 0x7f64e80b5680 con 0x7f64f409e9e0 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 msgr2=0x7f64f402c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 0x7f64f402c990 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64e801c3e0 tx=0x7f64e80a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 msgr2=0x7f64f402c990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64dbfff640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f409e9e0 0x7f64f402c990 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f64e801c3e0 tx=0x7f64e80a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 151886, 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:07.275 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 8, 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.276 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.277 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:07.278 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.279 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:07.280 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.282 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.283 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.284 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.285 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.286 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.287 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.288 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.289 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:07.290 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.291 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:07.292 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.293 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.294 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.295 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.296 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:03:07.297 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.297 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.297 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.297 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:07.297 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 10 librados: watch_flush enter 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 10 librados: watch_flush exit 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 20 client.144443.objecter shutdown clearing up homeless session... 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 10 client.144443.objecter successfully canceled tick 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f64e0038e40 msgr2=0x7f64e003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f64e0038e40 0x7f64e003b300 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009440 tx=0x7f64ec00b040 comp rx=0 tx=0).stop 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 10 monclient: shutdown 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f40a5b80 msgr2=0x7f64f41d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.299 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 --2- 172.21.15.43:0/2284520056 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f64f40a5b80 0x7f64f41d0830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64e4000f70 tx=0x7f64e400c8b0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f3fff640 1 -- 172.21.15.43:0/2284520056 reap_dead start 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f3fff640 1 -- 172.21.15.43:0/2284520056 reap_dead start 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 shutdown_connections 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 >> 172.21.15.43:0/2284520056 conn(0x7f64f40ab2f0 msgr2=0x7f64f40a9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 shutdown_connections 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 -- 172.21.15.43:0/2284520056 wait complete. 2024-04-19T01:03:07.300 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.256+0000 7f64f9e6f640 1 librados: shutdown 2024-04-19T01:03:07.300 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:07.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T01:03:07.301 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient: get_monmap_and_config 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient: monmap: 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:07.410542+0000 2024-04-19T01:03:07.407 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:07.410542+0000 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:07.408 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 1 Processor -- start 2024-04-19T01:03:07.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 1 -- start start 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient: init 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient: _add_conns ranks=[0,3,1,2,9,5,4,7,10,8,6] 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbad8138a50 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbad8139540 0x7fbad813dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient(hunting): picked mon.noname-g con 0x7fbad8139540 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbad80a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad80a0d40 con 0x7fbad8138a50 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbad80a0a60 con 0x7fbad80a5bb0 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbad80a1020 con 0x7fbad8139540 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 10 monclient(hunting): authenticate will time out at 2057.537621s 2024-04-19T01:03:07.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fbad8138a50 0x7fbad8138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47054/0 (socket says 172.21.15.43:47054) 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 1 -- 172.21.15.43:0/3930534719 learned_addr learned my addr 172.21.15.43:0/3930534719 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbad8139540 0x7fbad813dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): get_auth_request con 0x7fbad8139540 auth_method 0 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4287968928 0 0) 0x7fbad80a0d40 con 0x7fbad8138a50 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): _negotiate 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): my global_id is 65709 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbac4003530 con 0x7fbad8138a50 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3573636969 0 0) 0x7fbac4003530 con 0x7fbad8138a50 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbad8139540 msgr2=0x7fbad813dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbad8139540 0x7fbad813dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 >> v1:172.21.15.161:6789/0 conn(0x7fbad80a5bb0 legacy=0x7fbad8140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad80a17b0 con 0x7fbad8138a50 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fbac8001970 con 0x7fbad8138a50 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 5 monclient: authenticate success, global_id 65709 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: dump: 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.423 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T01:03:07.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 >> v1:172.21.15.43:6789/0 conn(0x7fbad8138a50 legacy=0x7fbad8138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient: _add_conns ranks=[1,6,4,5,7,2,8,3,0] 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbac4004240 0x7fbac4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): picked mon.f con 0x7fbac4004240 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbac4004da0 0x7fbac4005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): picked mon.d con 0x7fbac4004da0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbac4009c60 0x7fbac400c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): picked mon.c con 0x7fbac4009c60 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbad80a1020 con 0x7fbac4004240 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fbac400c9d0 con 0x7fbac4009c60 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.425 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fbac400cb80 con 0x7fbac4004da0 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbacffff640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadf2d5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbac4004da0 0x7fbac4005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbac4009c60 0x7fbac400c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): get_auth_request con 0x7fbac4004da0 auth_method 0 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): get_auth_request con 0x7fbac4009c60 auth_method 0 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbac4004240 0x7fbac4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 10 monclient(hunting): get_auth_request con 0x7fbac4004240 auth_method 0 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd071640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient(hunting): handle_auth_done global_id 65709 payload 293 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 -- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbac4009c60 msgr2=0x7fbac400c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.427 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbac4009c60 0x7fbac400c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 -- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbac4004240 msgr2=0x7fbac4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbac4004240 0x7fbac4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 monclient: found mon.d 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 1 -- 172.21.15.43:0/3930534719 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbac4003530 con 0x7fbac4004da0 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadc870640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.408+0000 7fbadd872640 1 -- 172.21.15.43:0/3930534719 reap_dead start 2024-04-19T01:03:07.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbac4004da0 0x7fbac4005220 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009ab0 tx=0x7fbad401ca50 comp rx=0 tx=0).ready entity=mon.6 client_cookie=53845a495e771203 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbad40096b0 con 0x7fbac4004da0 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: dump: 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.429 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad40a68a0 con 0x7fbac4004da0 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 1 -- 172.21.15.43:0/3930534719 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbad40a7410 con 0x7fbac4004da0 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 4 set_mon_vals no callback set 2024-04-19T01:03:07.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbacffff640 10 monclient: dump: 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.431 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: shutdown 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbac4004da0 msgr2=0x7fbac4005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbac4004da0 0x7fbac4005220 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbad4009ab0 tx=0x7fbad401ca50 comp rx=0 tx=0).stop 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- 172.21.15.43:0/3930534719 shutdown_connections 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 --2- 172.21.15.43:0/3930534719 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbac4004da0 0x7fbac4005220 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- 172.21.15.43:0/3930534719 >> 172.21.15.43:0/3930534719 conn(0x7fbad80a7cb0 msgr2=0x7fbad80a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- 172.21.15.43:0/3930534719 shutdown_connections 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- 172.21.15.43:0/3930534719 wait complete. 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: monmap: 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.433 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 librados: starting msgr at 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 librados: starting objecter 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 Processor -- start 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- start start 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 librados: setting wanted keys 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 librados: calling monclient init 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: init 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient: _add_conns ranks=[3,0,5,4,7,2,6,8,1] 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbac4004da0 0x7fbad8121cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): picked mon.noname-g con 0x7fbac4004da0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbad8136de0 0x7fbad8137220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): picked mon.noname-a con 0x7fbad8136de0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbad81d0a40 0x7fbad81d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): picked mon.noname-h con 0x7fbad81d0a40 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8138500 con 0x7fbad8136de0 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbad8138380 con 0x7fbac4004da0 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fbad8138680 con 0x7fbad81d0a40 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadf2d5640 10 monclient(hunting): authenticate will time out at 2057.543537s 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbad8136de0 0x7fbad8137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbad8136de0 0x7fbad8137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45000/0 (socket says 172.21.15.43:45000) 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 -- 172.21.15.43:0/1463500495 learned_addr learned my addr 172.21.15.43:0/1463500495 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): get_auth_request con 0x7fbad8136de0 auth_method 0 2024-04-19T01:03:07.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadd071640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbac4004da0 0x7fbad8121cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadd071640 10 monclient(hunting): get_auth_request con 0x7fbac4004da0 auth_method 0 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadd071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadd071640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadd071640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient(hunting): handle_auth_done global_id 65718 payload 824 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadd872640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbad81d0a40 0x7fbad81d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbad81d0a40 msgr2=0x7fbad81d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fbad81d0a40 0x7fbad81d0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbac4004da0 msgr2=0x7fbad8121cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbac4004da0 0x7fbad8121cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.412+0000 7fbadc870640 1 monclient: found mon.noname-a 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadc870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadc870640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadc870640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad809e610 con 0x7fbad8136de0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadc870640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbad8136de0 0x7fbad8137220 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbad401c880 tx=0x7fbad40197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 client.?.objecter ms_handle_connect 0x7fbad8136de0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbad40a6c10 con 0x7fbad8136de0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:07.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: dump: 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.438 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 5 monclient: authenticate success, global_id 65718 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad40a6db0 con 0x7fbad8136de0 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbad40b9d20 con 0x7fbad8136de0 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacf7fe640 4 set_mon_vals no callback set 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: dump: 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.439 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 monclient: _renew_subs 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad8137760 con 0x7fbad8136de0 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 client.65718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 monclient: _renew_subs 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad809ee60 con 0x7fbad8136de0 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 1 librados: init done 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 0x7fbad80a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fbad8138500 con 0x7fbad81d13e0 2024-04-19T01:03:07.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 0x7fbad80a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient: get_auth_request con 0x7fbad81d13e0 auth_method 0 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbad40b9440 con 0x7fbad8136de0 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 10 monclient(hunting): handle_auth_done global_id 134590 payload 824 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbad80008e0 con 0x7fbad81d13e0 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbabc038e10 0x7fbabc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(697..697 src has 501..697) ==== 6742+0+0 (secure 0 0 0) 0x7fbad400f040 con 0x7fbad8136de0 2024-04-19T01:03:07.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 client.65718.objecter ms_dispatch 0x7fbad8000ba0 osd_map(697..697 src has 501..697) 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 3 client.65718.objecter handle_osd_map got epochs [697,697] > 0 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 3 client.65718.objecter handle_osd_map decoding full epoch 697 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd071640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 0x7fbad80a25b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbac8010780 tx=0x7fbac8010c40 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 20 client.65718.objecter dump_active .. 0 homeless 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 client.65718.objecter ms_handle_connect 0x7fbad81d13e0 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 client.65718.objecter resend_mon_ops 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbac8018020 con 0x7fbad81d13e0 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: dump: 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.443 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd872640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbabc038e10 0x7fbabc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd872640 10 monclient: get_auth_request con 0x7fbabc038e10 auth_method 0 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbac8001820 con 0x7fbad81d13e0 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 msgr2=0x7fbad80a25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 0x7fbad80a25b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbac8010780 tx=0x7fbac8010c40 comp rx=0 tx=0).stop 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 msgr2=0x7fbad80a25b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad81d13e0 0x7fbad80a25b0 secure :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbac8010780 tx=0x7fbac8010c40 comp rx=0 tx=0).stop 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbacdffb640 10 client.65718.objecter ms_handle_connect 0x7fbabc038e10 2024-04-19T01:03:07.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.416+0000 7fbadd872640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbabc038e10 0x7fbabc03b2d0 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fbad8139ac0 tx=0x7fbad0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:07.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 0x7fbad80a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fbad8122c10 con 0x7fbad80a2af0 2024-04-19T01:03:07.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 0x7fbad80a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient: get_auth_request con 0x7fbad80a2af0 auth_method 0 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 10 monclient(hunting): handle_auth_done global_id 134599 payload 824 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 1 -- 172.21.15.43:0/1463500495 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbad8076190 con 0x7fbad80a2af0 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd071640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 0x7fbad80a2f20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbac8019000 tx=0x7fbac8001c30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 client.65718.objecter ms_handle_connect 0x7fbad80a2af0 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 client.65718.objecter resend_mon_ops 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbac8018040 con 0x7fbad80a2af0 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 monclient: dump: 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.449 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9614 (secure 0 0 0) 0x7fbac801f0d0 con 0x7fbad80a2af0 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 msgr2=0x7fbad80a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 0x7fbad80a2f20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbac8019000 tx=0x7fbac8001c30 comp rx=0 tx=0).stop 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 msgr2=0x7fbad80a2f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbacdffb640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fbad80a2af0 0x7fbad80a2f20 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbac8019000 tx=0x7fbac8001c30 comp rx=0 tx=0).stop 2024-04-19T01:03:07.450 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 292128, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:07.451 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 8, 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.453 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:07.454 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.455 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:07.456 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.457 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:07.458 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.462 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:07.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.470 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.471 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 4 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:07.473 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 10 librados: watch_flush enter 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 10 librados: watch_flush exit 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 20 client.65718.objecter shutdown clearing up homeless session... 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 10 client.65718.objecter successfully canceled tick 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbabc038e10 msgr2=0x7fbabc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbabc038e10 0x7fbabc03b2d0 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fbad8139ac0 tx=0x7fbad0009040 comp rx=0 tx=0).stop 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 10 monclient: shutdown 2024-04-19T01:03:07.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbad8136de0 msgr2=0x7fbad8137220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 --2- 172.21.15.43:0/1463500495 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbad8136de0 0x7fbad8137220 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fbad401c880 tx=0x7fbad40197f0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd872640 1 -- 172.21.15.43:0/1463500495 reap_dead start 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadd872640 1 -- 172.21.15.43:0/1463500495 reap_dead start 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 shutdown_connections 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 >> 172.21.15.43:0/1463500495 conn(0x7fbad80a7cb0 msgr2=0x7fbad80a4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 shutdown_connections 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 -- 172.21.15.43:0/1463500495 wait complete. 2024-04-19T01:03:07.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.424+0000 7fbadf2d5640 1 librados: shutdown 2024-04-19T01:03:07.476 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:07.476 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T01:03:07.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T01:03:07.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient: get_monmap_and_config 2024-04-19T01:03:07.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.591 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient: monmap: 2024-04-19T01:03:07.591 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.591 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:07.594699+0000 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:07.594699+0000 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.592 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:07.593 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:07.593 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 Processor -- start 2024-04-19T01:03:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 -- start start 2024-04-19T01:03:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient: init 2024-04-19T01:03:07.595 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient: _add_conns ranks=[6,1,9,2,4,7,8,10,0,3,5] 2024-04-19T01:03:07.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): picked mon.noname-d con 0x7fbea4138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:07.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fbea40a5bb0 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:07.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbea4139370 0x7fbea4138070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): picked mon.noname-a con 0x7fbea4139370 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbea40a0d40 con 0x7fbea40a5bb0 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fbea40a1020 con 0x7fbea4139370 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fbea40a0780 con 0x7fbea4138a50 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea9971640 10 monclient(hunting): authenticate will time out at 2057.721834s 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbea4139370 0x7fbea4138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fbea4139370 0x7fbea4138070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45002/0 (socket says 172.21.15.43:45002) 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:47520/0 (socket says 172.21.15.43:47520) 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 -- 172.21.15.43:0/3794727925 learned_addr learned my addr 172.21.15.43:0/3794727925 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): get_auth_request con 0x7fbea4139370 auth_method 0 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): get_auth_request con 0x7fbea4138a50 auth_method 0 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient(hunting): handle_auth_done global_id 65727 payload 274 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 -- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 msgr2=0x7fbea4138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 --2- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4138e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 -- 172.21.15.43:0/3794727925 >> v1:172.21.15.161:6789/0 conn(0x7fbea40a5bb0 legacy=0x7fbea4137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 monclient: found mon.noname-a 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea37fe640 1 -- 172.21.15.43:0/3794727925 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea40a0a60 con 0x7fbea4139370 2024-04-19T01:03:07.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.592+0000 7fbea2ffd640 1 --2- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea37fe640 1 --2- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbea4139370 0x7fbea4138070 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009e80 tx=0x7fbe9c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced2b9869ddcd904 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 1 -- 172.21.15.43:0/3794727925 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbe9c0090d0 con 0x7fbea4139370 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: dump: 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.607 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.608 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 5 monclient: authenticate success, global_id 65727 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 1 -- 172.21.15.43:0/3794727925 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe9c0a5990 con 0x7fbea4139370 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 4 set_mon_vals no callback set 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 1 -- 172.21.15.43:0/3794727925 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbe9c0ad3f0 con 0x7fbea4139370 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:07.609 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea1ffb640 10 monclient: dump: 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.610 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: shutdown 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbea4139370 msgr2=0x7fbea4138070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 --2- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbea4139370 0x7fbea4138070 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009e80 tx=0x7fbe9c01c8d0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- 172.21.15.43:0/3794727925 shutdown_connections 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 --2- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fbea4139370 0x7fbea4138070 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 --2- 172.21.15.43:0/3794727925 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4138e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- 172.21.15.43:0/3794727925 >> 172.21.15.43:0/3794727925 conn(0x7fbea40a7cb0 msgr2=0x7fbea40a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- 172.21.15.43:0/3794727925 shutdown_connections 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- 172.21.15.43:0/3794727925 wait complete. 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: monmap: 2024-04-19T01:03:07.611 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.612 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 librados: starting msgr at 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 librados: starting objecter 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 Processor -- start 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- start start 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 librados: setting wanted keys 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 librados: calling monclient init 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: init 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient: _add_conns ranks=[3,6,7,8,5,2,0,4,1] 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbea40a5bb0 0x7fbea4072e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): picked mon.noname-g con 0x7fbea40a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): picked mon.noname-d con 0x7fbea4138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:07.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fbea4139370 0x7fbea4135660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): picked mon.noname-i con 0x7fbea4139370 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fbea4138730 con 0x7fbea40a5bb0 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fbea41385b0 con 0x7fbea4138a50 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fbea4143820 con 0x7fbea4139370 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea9971640 10 monclient(hunting): authenticate will time out at 2057.727839s 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.596+0000 7fbea27fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4135120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:47528/0 (socket says 172.21.15.43:47528) 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 -- 172.21.15.43:0/2984344277 learned_addr learned my addr 172.21.15.43:0/2984344277 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): get_auth_request con 0x7fbea4138a50 auth_method 0 2024-04-19T01:03:07.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbea40a5bb0 0x7fbea4072e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): get_auth_request con 0x7fbea40a5bb0 auth_method 0 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fbea4139370 0x7fbea4135660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 10 monclient(hunting): get_auth_request con 0x7fbea4139370 auth_method 0 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient(hunting): handle_auth_done global_id 144456 payload 824 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fbea4139370 msgr2=0x7fbea4135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fbea4139370 0x7fbea4135660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbea40a5bb0 msgr2=0x7fbea4072e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fbea40a5bb0 0x7fbea4072e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 monclient: found mon.noname-d 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbea406ff40 con 0x7fbea4138a50 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea27fc640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4135120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800c410 tx=0x7fbe9800c8d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 client.?.objecter ms_handle_connect 0x7fbea4138a50 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:07.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbe98011a00 con 0x7fbea4138a50 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: dump: 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.617 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe98017070 con 0x7fbea4138a50 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 5 monclient: authenticate success, global_id 144456 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 monclient: _renew_subs 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbea4137220 con 0x7fbea4138a50 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbe98011c60 con 0x7fbea4138a50 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 client.144456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea17fa640 4 set_mon_vals no callback set 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: dump: 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.619 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 monclient: _renew_subs 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7fbea409ee60 con 0x7fbea4138a50 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 1 librados: init done 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 0x7fbea40a31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fbea406d140 con 0x7fbea4135ba0 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fbe98011590 con 0x7fbea4138a50 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 0x7fbea40a31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient: get_auth_request con 0x7fbea4135ba0 auth_method 0 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbe90038e40 0x7fbe9003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(697..697 src has 501..697) ==== 6742+0+0 (secure 0 0 0) 0x7fbe98049290 con 0x7fbea4138a50 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 client.144456.objecter ms_dispatch 0x7fbea4000ba0 osd_map(697..697 src has 501..697) 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_done global_id 144465 payload 824 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 3 client.144456.objecter handle_osd_map got epochs [697,697] > 0 2024-04-19T01:03:07.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 3 client.144456.objecter handle_osd_map decoding full epoch 697 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbea4070a00 con 0x7fbea4135ba0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 20 client.144456.objecter dump_active .. 0 homeless 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 client.144456.objecter ms_handle_connect 0x7fbea4135ba0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 client.144456.objecter resend_mon_ops 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea2ffd640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 0x7fbea40a31b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011600 tx=0x7fbe94011ac0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbe90038e40 0x7fbe9003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 10 monclient: get_auth_request con 0x7fbe90038e40 auth_method 0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbe9400fbb0 con 0x7fbea4135ba0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: dump: 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.622 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.623 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbe9400f740 con 0x7fbea4135ba0 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 msgr2=0x7fbea40a31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 0x7fbea40a31b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011600 tx=0x7fbe94011ac0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 msgr2=0x7fbea40a31b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4135ba0 0x7fbea40a31b0 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011600 tx=0x7fbe94011ac0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbe8bfff640 10 client.144456.objecter ms_handle_connect 0x7fbe90038e40 2024-04-19T01:03:07.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.600+0000 7fbea37fe640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbe90038e40 0x7fbe9003b300 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0ae000 tx=0x7fbe9c0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 0x7fbea40a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fbea4135f80 con 0x7fbea40a36f0 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 0x7fbea40a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient: get_auth_request con 0x7fbea40a36f0 auth_method 0 2024-04-19T01:03:07.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 10 monclient(hunting): handle_auth_done global_id 144474 payload 824 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 1 -- 172.21.15.43:0/2984344277 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbea4137460 con 0x7fbea40a36f0 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 client.144456.objecter ms_handle_connect 0x7fbea40a36f0 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 client.144456.objecter resend_mon_ops 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea2ffd640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 0x7fbea40a3b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011600 tx=0x7fbe940046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fbe94004490 con 0x7fbea40a36f0 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:07.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 monclient: dump: 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.628 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7fbe94005f70 con 0x7fbea40a36f0 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 msgr2=0x7fbea40a3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 0x7fbea40a3b00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011600 tx=0x7fbe940046d0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 msgr2=0x7fbea40a3b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbe8bfff640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea40a36f0 0x7fbea40a3b00 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe94011600 tx=0x7fbe940046d0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:07.629 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 152238, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:07.630 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 8, 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.631 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.632 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:07.633 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.634 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.635 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.636 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.637 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:07.638 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.639 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:07.640 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.641 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:07.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:07.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.648 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.649 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.650 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:07.651 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 10 librados: watch_flush enter 2024-04-19T01:03:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 10 librados: watch_flush exit 2024-04-19T01:03:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 20 client.144456.objecter shutdown clearing up homeless session... 2024-04-19T01:03:07.653 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 10 client.144456.objecter successfully canceled tick 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbe90038e40 msgr2=0x7fbe9003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fbe90038e40 0x7fbe9003b300 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c0ae000 tx=0x7fbe9c0a7070 comp rx=0 tx=0).stop 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 10 monclient: shutdown 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 msgr2=0x7fbea4135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 --2- 172.21.15.43:0/2984344277 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fbea4138a50 0x7fbea4135120 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbe9800c410 tx=0x7fbe9800c8d0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea37fe640 1 -- 172.21.15.43:0/2984344277 reap_dead start 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea37fe640 1 -- 172.21.15.43:0/2984344277 reap_dead start 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 shutdown_connections 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 >> 172.21.15.43:0/2984344277 conn(0x7fbea40a7cb0 msgr2=0x7fbea40a26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 shutdown_connections 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 -- 172.21.15.43:0/2984344277 wait complete. 2024-04-19T01:03:07.654 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.608+0000 7fbea9971640 1 librados: shutdown 2024-04-19T01:03:07.655 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:07.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T01:03:07.655 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient: get_monmap_and_config 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient: monmap: 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:07.782657+0000 2024-04-19T01:03:07.791 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:07.782657+0000 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:07.792 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 Processor -- start 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 -- start start 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient: init 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient: _add_conns ranks=[7,9,2,6,0,4,5,3,1,8,10] 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b409e4c0 0x7f41b4136e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): picked mon.noname-i con 0x7f41b409e4c0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f41b409e8a0 0x7f41b40a3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): picked mon.noname-a con 0x7f41b409e8a0 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 0x7f41b40a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): picked mon.noname-b con 0x7f41b409eca0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f41b40a1c50 con 0x7f41b409e8a0 2024-04-19T01:03:07.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f41b40a1e20 con 0x7f41b409eca0 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f41b406c760 con 0x7f41b409e4c0 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b9a25640 10 monclient(hunting): authenticate will time out at 2057.909896s 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 0x7f41b40a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f41b409e8a0 0x7f41b40a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 0x7f41b40a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60860/0 (socket says 172.21.15.43:60860) 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 1 -- 172.21.15.43:0/4191377772 learned_addr learned my addr 172.21.15.43:0/4191377772 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): get_auth_request con 0x7f41b409eca0 auth_method 0 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): get_auth_request con 0x7f41b409e8a0 auth_method 0 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b2ffd640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b409e4c0 0x7f41b4136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b2ffd640 10 monclient(hunting): get_auth_request con 0x7f41b409e4c0 auth_method 0 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient(hunting): handle_auth_done global_id 65736 payload 274 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 -- 172.21.15.43:0/4191377772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b409e4c0 msgr2=0x7f41b4136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b409e4c0 0x7f41b4136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 -- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 msgr2=0x7f41b40a3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 0x7f41b40a3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 monclient: found mon.noname-a 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b27fc640 1 -- 172.21.15.43:0/4191377772 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b40a5510 con 0x7f41b409e8a0 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b37fe640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 0x7f41b40a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:07.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.780+0000 7f41b2ffd640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b409e4c0 0x7f41b4136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b27fc640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e8a0 0x7f41b40a3940 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097c0 tx=0x7f41a801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ab1c898b11e634 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 1 -- 172.21.15.43:0/4191377772 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a80b0070 con 0x7f41b409e8a0 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: dump: 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.797 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 1 -- 172.21.15.43:0/4191377772 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a80aa810 con 0x7f41b409e8a0 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 5 monclient: authenticate success, global_id 65736 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 4 set_mon_vals no callback set 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 1 -- 172.21.15.43:0/4191377772 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a80a9300 con 0x7f41b409e8a0 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b1ffb640 10 monclient: dump: 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.799 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: shutdown 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e8a0 msgr2=0x7f41b40a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e8a0 0x7f41b40a3940 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097c0 tx=0x7f41a801c3f0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- 172.21.15.43:0/4191377772 shutdown_connections 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f41b409eca0 0x7f41b40a3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e8a0 0x7f41b40a3940 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- 172.21.15.43:0/4191377772 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f41b409e4c0 0x7f41b4136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- 172.21.15.43:0/4191377772 >> 172.21.15.43:0/4191377772 conn(0x7f41b40a85c0 msgr2=0x7f41b40a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- 172.21.15.43:0/4191377772 shutdown_connections 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- 172.21.15.43:0/4191377772 wait complete. 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: monmap: 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.801 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 librados: starting msgr at 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 librados: starting objecter 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 Processor -- start 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- start start 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 librados: setting wanted keys 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 librados: calling monclient init 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: init 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient: _add_conns ranks=[0,5,3,4,2,7,8,6,1] 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 0x7f41b41320a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): picked mon.noname-a con 0x7f41b409e4c0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41b409e8a0 0x7f41b41325e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): picked mon.noname-h con 0x7f41b409e8a0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41b409eca0 0x7f41b4132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): picked mon.noname-g con 0x7f41b409eca0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f41b41406e0 con 0x7f41b409e4c0 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f41b40a47d0 con 0x7f41b409eca0 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f41b4140860 con 0x7f41b409e8a0 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b9a25640 10 monclient(hunting): authenticate will time out at 2057.915262s 2024-04-19T01:03:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 0x7f41b41320a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 0x7f41b41320a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45020/0 (socket says 172.21.15.43:45020) 2024-04-19T01:03:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 -- 172.21.15.43:0/3262200939 learned_addr learned my addr 172.21.15.43:0/3262200939 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): get_auth_request con 0x7f41b409e4c0 auth_method 0 2024-04-19T01:03:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b27fc640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41b409e8a0 0x7f41b41325e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient(hunting): handle_auth_done global_id 65745 payload 824 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b37fe640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41b409eca0 0x7f41b4132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41b409e8a0 msgr2=0x7f41b41325e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41b409e8a0 0x7f41b41325e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41b409eca0 msgr2=0x7f41b4132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f41b409eca0 0x7f41b4132b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 monclient: found mon.noname-a 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41b406b9e0 con 0x7f41b409e4c0 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b27fc640 10 monclient: get_auth_request con 0x7f41b409e8a0 auth_method 0 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b27fc640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f41b409e8a0 0x7f41b41325e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f41b2ffd640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 0x7f41b41320a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f41a400e840 tx=0x7f41a400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f419bfff640 10 client.?.objecter ms_handle_connect 0x7f41b409e4c0 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.784+0000 7f419bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a4017070 con 0x7f41b409e4c0 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: dump: 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.807 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 5 monclient: authenticate success, global_id 65745 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a400cb40 con 0x7f41b409e4c0 2024-04-19T01:03:07.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 monclient: _renew_subs 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41b4070270 con 0x7f41b409e4c0 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a400f040 con 0x7f41b409e4c0 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: dump: 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.809 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 client.65745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 monclient: _renew_subs 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41b4133bf0 con 0x7f41b409e4c0 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 1 librados: init done 2024-04-19T01:03:07.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b17fa640 4 set_mon_vals no callback set 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 0x7f41b412aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f41b406d140 con 0x7f41b412a640 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 0x7f41b412aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient: get_auth_request con 0x7f41b412a640 auth_method 0 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f41a401a070 con 0x7f41b409e4c0 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 10 monclient(hunting): handle_auth_done global_id 144458 payload 824 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41b4070a70 con 0x7f41b412a640 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41a0038e40 0x7f41a003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(698..698 src has 501..698) ==== 6356+0+0 (secure 0 0 0) 0x7f41a4049020 con 0x7f41b409e4c0 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 client.65745.objecter ms_dispatch 0x7f41b4000ba0 osd_map(698..698 src has 501..698) 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 3 client.65745.objecter handle_osd_map got epochs [698,698] > 0 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 3 client.65745.objecter handle_osd_map decoding full epoch 698 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b27fc640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 0x7f41b412aa20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f41b40a5480 tx=0x7f41a8019190 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 20 client.65745.objecter dump_active .. 0 homeless 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 client.65745.objecter ms_handle_connect 0x7f41b412a640 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 client.65745.objecter resend_mon_ops 2024-04-19T01:03:07.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a80b9030 con 0x7f41b412a640 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: dump: 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.813 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b37fe640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41a0038e40 0x7f41a003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b37fe640 10 monclient: get_auth_request con 0x7f41a0038e40 auth_method 0 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41a8003830 con 0x7f41b412a640 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 msgr2=0x7f41b412aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 0x7f41b412aa20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f41b40a5480 tx=0x7f41a8019190 comp rx=0 tx=0).stop 2024-04-19T01:03:07.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 msgr2=0x7f41b412aa20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412a640 0x7f41b412aa20 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f41b40a5480 tx=0x7f41a8019190 comp rx=0 tx=0).stop 2024-04-19T01:03:07.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f419bfff640 10 client.65745.objecter ms_handle_connect 0x7f41a0038e40 2024-04-19T01:03:07.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.788+0000 7f41b37fe640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41a0038e40 0x7f41a003b300 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7f41b409df80 tx=0x7f41ac00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:07.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 0x7f41b40a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f41b4070890 con 0x7f41b412af60 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 0x7f41b40a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient: get_auth_request con 0x7f41b412af60 auth_method 0 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 10 monclient(hunting): handle_auth_done global_id 144467 payload 824 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 1 -- 172.21.15.43:0/3262200939 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41b4072e70 con 0x7f41b412af60 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 client.65745.objecter ms_handle_connect 0x7f41b412af60 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 client.65745.objecter resend_mon_ops 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b27fc640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 0x7f41b40a5ec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097c0 tx=0x7f41a80a8000 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f41a80b9040 con 0x7f41b412af60 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 monclient: dump: 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.818 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f41a80c0b70 con 0x7f41b412af60 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:07.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 msgr2=0x7f41b40a5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 0x7f41b40a5ec0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097c0 tx=0x7f41a80a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 msgr2=0x7f41b40a5ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f419bfff640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f41b412af60 0x7f41b40a5ec0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097c0 tx=0x7f41a80a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:07.820 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 152426, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 8, 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.821 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.822 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.823 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:07.824 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.825 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:07.827 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:07.830 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.832 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:07.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:07.835 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:07.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:07.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:07.840 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:07.841 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:07.842 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 10 librados: watch_flush enter 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 10 librados: watch_flush exit 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 20 client.65745.objecter shutdown clearing up homeless session... 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 10 client.65745.objecter successfully canceled tick 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41a0038e40 msgr2=0x7f41a003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f41a0038e40 0x7f41a003b300 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7f41b409df80 tx=0x7f41ac00b040 comp rx=0 tx=0).stop 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 10 monclient: shutdown 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 msgr2=0x7f41b41320a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 0x7f41b41320a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f41a400e840 tx=0x7f41a400ed00 comp rx=0 tx=0).stop 2024-04-19T01:03:07.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b37fe640 1 -- 172.21.15.43:0/3262200939 reap_dead start 2024-04-19T01:03:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 shutdown_connections 2024-04-19T01:03:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 --2- 172.21.15.43:0/3262200939 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f41b409e4c0 0x7f41b41320a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 >> 172.21.15.43:0/3262200939 conn(0x7f41b40a85c0 msgr2=0x7f41b4130d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 shutdown_connections 2024-04-19T01:03:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 -- 172.21.15.43:0/3262200939 wait complete. 2024-04-19T01:03:07.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.796+0000 7f41b9a25640 1 librados: shutdown 2024-04-19T01:03:07.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:07.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T01:03:07.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient: get_monmap_and_config 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient: monmap: 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:07.974540+0000 2024-04-19T01:03:07.971 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:07.974540+0000 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:07.972 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 Processor -- start 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 -- start start 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient: init 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient: _add_conns ranks=[10,4,2,0,9,1,7,8,5,6,3] 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f51d40a1f20 0x7f51d40a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): picked mon.noname-f con 0x7f51d40a1f20 addr v2:172.21.15.161:3300/0 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f51d40a1270 0x7f51d40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): picked mon.noname-c con 0x7f51d40a1270 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): picked mon.noname-b con 0x7f51d40a4430 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f51d40a0920 con 0x7f51d40a1f20 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f51d406bf40 con 0x7f51d40a4430 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f51d409fff0 con 0x7f51d40a1270 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51db365640 10 monclient(hunting): authenticate will time out at 2058.101661s 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f51d40a1270 0x7f51d40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d413fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60876/0 (socket says 172.21.15.43:60876) 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 -- 172.21.15.43:0/2993477766 learned_addr learned my addr 172.21.15.43:0/2993477766 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): get_auth_request con 0x7f51d40a4430 auth_method 0 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f51d40a1270 0x7f51d40a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47576/0 (socket says 172.21.15.43:47576) 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): get_auth_request con 0x7f51d40a1270 auth_method 0 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:07.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d8900640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9101640 1 --2- 172.21.15.43:0/2993477766 >> v2:172.21.15.161:3300/0 conn(0x7f51d40a1f20 0x7f51d40a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9101640 10 monclient(hunting): get_auth_request con 0x7f51d40a1f20 auth_method 0 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9101640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9101640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient(hunting): handle_auth_done global_id 144479 payload 274 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 -- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f51d40a1270 msgr2=0x7f51d40a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 --2- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f51d40a1270 0x7f51d40a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 -- 172.21.15.43:0/2993477766 >> v2:172.21.15.161:3300/0 conn(0x7f51d40a1f20 msgr2=0x7f51d40a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:07.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 --2- 172.21.15.43:0/2993477766 >> v2:172.21.15.161:3300/0 conn(0x7f51d40a1f20 0x7f51d40a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient: _finish_hunting 0 2024-04-19T01:03:07.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 monclient: found mon.noname-b 2024-04-19T01:03:07.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:07.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:07.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.972+0000 7f51d9902640 1 -- 172.21.15.43:0/2993477766 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d40a02d0 con 0x7f51d40a4430 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 --2- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d413fba0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009e80 tx=0x7f51cc01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab96bba5bb9aab89 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 1 -- 172.21.15.43:0/2993477766 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f51cc0090d0 con 0x7f51d40a4430 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: dump: 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.991 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: _finish_auth 0 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 1 -- 172.21.15.43:0/2993477766 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51cc0a5aa0 con 0x7f51d40a4430 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:07.992 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 1 -- 172.21.15.43:0/2993477766 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f51cc0a5c90 con 0x7f51d40a4430 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 5 monclient: authenticate success, global_id 144479 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51cbfff640 10 monclient: dump: 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:07.993 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 4 set_mon_vals no callback set 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: shutdown 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 msgr2=0x7f51d413fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:07.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d413fba0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009e80 tx=0x7f51cc01c8f0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- 172.21.15.43:0/2993477766 shutdown_connections 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d413fba0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- 172.21.15.43:0/2993477766 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f51d40a1270 0x7f51d40a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- 172.21.15.43:0/2993477766 >> v2:172.21.15.161:3300/0 conn(0x7f51d40a1f20 0x7f51d40a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- 172.21.15.43:0/2993477766 >> 172.21.15.43:0/2993477766 conn(0x7f51d40a85a0 msgr2=0x7f51d40a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- 172.21.15.43:0/2993477766 shutdown_connections 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- 172.21.15.43:0/2993477766 wait complete. 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: build_initial_monmap 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: monmap: 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:07.995 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:07.996 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 librados: starting msgr at 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 librados: starting objecter 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 Processor -- start 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- start start 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 librados: setting wanted keys 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 librados: calling monclient init 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: init 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:07.997 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient: _add_conns ranks=[7,0,2,4,8,3,5,6,1] 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f51d40a1270 0x7f51d41c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): picked mon.noname-i con 0x7f51d40a1270 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51d40a1f20 0x7f51d41d1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): picked mon.noname-a con 0x7f51d40a1f20 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d4122a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): picked mon.noname-b con 0x7f51d40a4430 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f51d40a2480 con 0x7f51d40a1f20 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f51d40a2300 con 0x7f51d40a4430 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f51d4142900 con 0x7f51d40a1270 2024-04-19T01:03:07.998 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51db365640 10 monclient(hunting): authenticate will time out at 2058.106989s 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d4122a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51d40a1f20 0x7f51d41d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d4122a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:60886/0 (socket says 172.21.15.43:60886) 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 -- 172.21.15.43:0/1182060337 learned_addr learned my addr 172.21.15.43:0/1182060337 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): get_auth_request con 0x7f51d40a4430 auth_method 0 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): get_auth_request con 0x7f51d40a1f20 auth_method 0 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:07.999 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9101640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f51d40a1270 0x7f51d41c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9101640 10 monclient(hunting): get_auth_request con 0x7f51d40a1270 auth_method 0 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9101640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9101640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient(hunting): handle_auth_done global_id 144488 payload 824 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f51d40a1270 msgr2=0x7f51d41c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f51d40a1270 0x7f51d41c7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51d40a1f20 msgr2=0x7f51d41d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.000 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51d40a1f20 0x7f51d41d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 monclient: found mon.noname-b 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d4073100 con 0x7f51d40a4430 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d8900640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f51d40a1f20 0x7f51d41d1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51d9902640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d4122a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009040 tx=0x7f51cc019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51c9ffb640 10 client.?.objecter ms_handle_connect 0x7f51d40a4430 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51c9ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f51cc0bc020 con 0x7f51d40a4430 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.976+0000 7f51c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: dump: 2024-04-19T01:03:08.001 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.002 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 5 monclient: authenticate success, global_id 144488 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51cc003e10 con 0x7f51d40a4430 2024-04-19T01:03:08.004 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f51cc0a5840 con 0x7f51d40a4430 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51cb7fe640 4 set_mon_vals no callback set 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: dump: 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.005 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 monclient: _renew_subs 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d409f6c0 con 0x7f51d40a4430 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 client.144488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 monclient: _renew_subs 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:03:08.006 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d4125ad0 con 0x7f51d40a4430 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 1 librados: init done 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 0x7f51d4123320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51d406d140 con 0x7f51d4122f40 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 0x7f51d4123320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f51cc0bf020 con 0x7f51d40a4430 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient: get_auth_request con 0x7f51d4122f40 auth_method 0 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.007 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f51b8038e10 0x7f51b803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(698..698 src has 501..698) ==== 6356+0+0 (secure 0 0 0) 0x7f51cc0b3040 con 0x7f51d40a4430 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 client.144488.objecter ms_dispatch 0x7f51d4000ba0 osd_map(698..698 src has 501..698) 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 3 client.144488.objecter handle_osd_map got epochs [698,698] > 0 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 3 client.144488.objecter handle_osd_map decoding full epoch 698 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 20 client.144488.objecter dump_active .. 0 homeless 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 10 monclient(hunting): handle_auth_done global_id 144487 payload 824 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51d409ef70 con 0x7f51d4122f40 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d8900640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f51b8038e10 0x7f51b803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.008 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d8900640 10 monclient: get_auth_request con 0x7f51b8038e10 auth_method 0 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 client.144488.objecter ms_handle_connect 0x7f51d4122f40 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 client.144488.objecter resend_mon_ops 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d9101640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 0x7f51d4123320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51c400f440 tx=0x7f51c400f900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f51c4019030 con 0x7f51d4122f40 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: dump: 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.009 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 client.144488.objecter ms_handle_connect 0x7f51b8038e10 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51d8900640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f51b8038e10 0x7f51b803b2d0 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7f51d4072e50 tx=0x7f51d0009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f51c4014670 con 0x7f51d4122f40 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.010 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 msgr2=0x7f51d4123320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 0x7f51d4123320 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51c400f440 tx=0x7f51c400f900 comp rx=0 tx=0).stop 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 msgr2=0x7f51d4123320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.980+0000 7f51c9ffb640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4122f40 0x7f51d4123320 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f51c400f440 tx=0x7f51c400f900 comp rx=0 tx=0).stop 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51db365640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51db365640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 0x7f51d409da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51db365640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51d4125eb0 con 0x7f51d4123860 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 0x7f51d409da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.011 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient: get_auth_request con 0x7f51d4123860 auth_method 0 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 10 monclient(hunting): handle_auth_done global_id 144496 payload 824 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 1 -- 172.21.15.43:0/1182060337 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51d41262d0 con 0x7f51d4123860 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51d9101640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 0x7f51d409da80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c10 tx=0x7f51c40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51c9ffb640 10 client.144488.objecter ms_handle_connect 0x7f51d4123860 2024-04-19T01:03:08.012 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51c9ffb640 10 client.144488.objecter resend_mon_ops 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f51c401f240 con 0x7f51d4123860 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51c9ffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.988+0000 7f51c9ffb640 10 monclient: dump: 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.013 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.014 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.014 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.014 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.014 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.014 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.014 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f51c401f240 con 0x7f51d4123860 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 msgr2=0x7f51d409da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 0x7f51d409da80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c10 tx=0x7f51c40046d0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 msgr2=0x7f51d409da80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51c9ffb640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f51d4123860 0x7f51d409da80 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51c4000c10 tx=0x7f51c40046d0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.015 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 152624, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:08.016 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 9, 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.017 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.018 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:08.019 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.020 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:08.021 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.022 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:08.023 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.024 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.025 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.026 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.027 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.028 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.029 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.030 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:08.031 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.032 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.033 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:08.034 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.035 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.036 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:08.037 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 10 librados: watch_flush enter 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 10 librados: watch_flush exit 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 20 client.144488.objecter shutdown clearing up homeless session... 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 10 client.144488.objecter successfully canceled tick 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f51b8038e10 msgr2=0x7f51b803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f51b8038e10 0x7f51b803b2d0 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7f51d4072e50 tx=0x7f51d0009190 comp rx=0 tx=0).stop 2024-04-19T01:03:08.039 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 10 monclient: shutdown 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 msgr2=0x7f51d4122a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51d9902640 1 -- 172.21.15.43:0/1182060337 reap_dead start 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d4122a00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f51cc009040 tx=0x7f51cc019690 comp rx=0 tx=0).stop 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 shutdown_connections 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 --2- 172.21.15.43:0/1182060337 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f51d40a4430 0x7f51d4122a00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 >> 172.21.15.43:0/1182060337 conn(0x7f51d40a85a0 msgr2=0x7f51d4071680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 shutdown_connections 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 -- 172.21.15.43:0/1182060337 wait complete. 2024-04-19T01:03:08.040 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:07.992+0000 7f51db365640 1 librados: shutdown 2024-04-19T01:03:08.040 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:08.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T01:03:08.041 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.156+0000 7f5b01cf4640 10 monclient: get_monmap_and_config 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.156+0000 7f5b01cf4640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient: monmap: 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:08.162437+0000 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:08.162437+0000 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.159 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:08.160 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 Processor -- start 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 -- start start 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient: init 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient: _add_conns ranks=[0,2,6,8,1,7,3,5,4,9,10] 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5afc138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5afc0a5b80 0x7f5afc13e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f5afc0a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): picked mon.noname-d con 0x7f5afc1397c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5afc0a3dc0 con 0x7f5afc138cd0 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.163 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5afc0a4410 con 0x7f5afc0a5b80 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f5afc0a40a0 con 0x7f5afc1397c0 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5b01cf4640 10 monclient(hunting): authenticate will time out at 2058.289552s 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afb7fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f5afc138cd0 0x7f5afc1390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47056/0 (socket says 172.21.15.43:47056) 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afb7fe640 1 -- 172.21.15.43:0/4165099002 learned_addr learned my addr 172.21.15.43:0/4165099002 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5afc0a5b80 0x7f5afc13e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): get_auth_request con 0x7f5afc0a5b80 auth_method 0 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): get_auth_request con 0x7f5afc1397c0 auth_method 0 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.164 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient(hunting): handle_auth_done global_id 144497 payload 274 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 1 -- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 msgr2=0x7f5afc1407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afa7fc640 1 -- 172.21.15.43:0/4165099002 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4289073231 0 0) 0x7f5afc0a3dc0 con 0x7f5afc138cd0 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 1 -- 172.21.15.43:0/4165099002 >> v1:172.21.15.43:6789/0 conn(0x7f5afc138cd0 legacy=0x7f5afc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.165 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 1 monclient: found mon.noname-b 2024-04-19T01:03:08.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:08.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afaffd640 1 -- 172.21.15.43:0/4165099002 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc0a2ae0 con 0x7f5afc0a5b80 2024-04-19T01:03:08.166 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.160+0000 7f5afa7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5afc0a5b80 0x7f5afc13e270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5af00097f0 tx=0x7f5af001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a71815c3a29d69 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 1 -- 172.21.15.43:0/4165099002 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5af00b0070 con 0x7f5afc0a5b80 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: dump: 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.173 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.174 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 5 monclient: authenticate success, global_id 144497 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 1 -- 172.21.15.43:0/4165099002 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af00aa810 con 0x7f5afc0a5b80 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 1 -- 172.21.15.43:0/4165099002 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5af00a9300 con 0x7f5afc0a5b80 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 4 set_mon_vals no callback set 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afa7fc640 10 monclient: dump: 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.175 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.176 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: shutdown 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5afc0a5b80 msgr2=0x7f5afc13e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5afc0a5b80 0x7f5afc13e270 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5af00097f0 tx=0x7f5af001c3a0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/4165099002 shutdown_connections 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/4165099002 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5afc0a5b80 0x7f5afc13e270 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/4165099002 >> 172.21.15.43:0/4165099002 conn(0x7f5afc0ab2f0 msgr2=0x7f5afc0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/4165099002 shutdown_connections 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/4165099002 wait complete. 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: monmap: 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.177 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.178 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 librados: starting msgr at 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 librados: starting objecter 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 Processor -- start 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- start start 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 librados: setting wanted keys 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 librados: calling monclient init 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: init 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: _add_conns ranks=[4,8,6,3,5,0,2,7,1] 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5afc0a5b80 0x7f5afc1d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.179 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f5afc0a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5afc138cd0 0x7f5afc1d0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): picked mon.noname-e con 0x7f5afc138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1d1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): picked mon.noname-d con 0x7f5afc1397c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f5afc143730 con 0x7f5afc0a5b80 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f5afc0a20a0 con 0x7f5afc1397c0 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f5afc1438b0 con 0x7f5afc138cd0 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient(hunting): authenticate will time out at 2058.294687s 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5afc0a5b80 0x7f5afc1d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.180 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5afc138cd0 0x7f5afc1d0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1d1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:47558/0 (socket says 172.21.15.43:47558) 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5afc0a5b80 0x7f5afc1d09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47580/0 (socket says 172.21.15.43:47580) 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 -- 172.21.15.43:0/3114959900 learned_addr learned my addr 172.21.15.43:0/3114959900 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): get_auth_request con 0x7f5afc1397c0 auth_method 0 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): get_auth_request con 0x7f5afc0a5b80 auth_method 0 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): get_auth_request con 0x7f5afc138cd0 auth_method 0 2024-04-19T01:03:08.181 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.182 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient(hunting): handle_auth_done global_id 144492 payload 824 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5afc138cd0 msgr2=0x7f5afc1d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5afc138cd0 0x7f5afc1d0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5afc0a5b80 msgr2=0x7f5afc1d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5afc0a5b80 0x7f5afc1d09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 monclient: found mon.noname-d 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5afc0a3040 con 0x7f5afc1397c0 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afb7fe640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5afc0a5b80 0x7f5afc1d09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:08.183 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afaffd640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5afc138cd0 0x7f5afc1d0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5afbfff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1d1420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5af400efc0 tx=0x7f5af400c490 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 client.?.objecter ms_handle_connect 0x7f5afc1397c0 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5af4018020 con 0x7f5afc1397c0 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: dump: 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.184 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af400cd30 con 0x7f5afc1397c0 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 5 monclient: authenticate success, global_id 144492 2024-04-19T01:03:08.185 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: _renew_subs 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5afc0a1960 con 0x7f5afc1397c0 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5af4014d60 con 0x7f5afc1397c0 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5ae3fff640 10 monclient: dump: 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.186 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 client.144492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: _renew_subs 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f5afc0a0990 con 0x7f5afc1397c0 2024-04-19T01:03:08.187 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5b01cf4640 1 librados: init done 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.164+0000 7f5af9ffb640 4 set_mon_vals no callback set 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5b01cf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 0x7f5afc135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5afc06d140 con 0x7f5afc1358b0 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5af401b020 con 0x7f5afc1397c0 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ae8038e40 0x7f5ae803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(698..698 src has 501..698) ==== 6356+0+0 (secure 0 0 0) 0x7f5af404a150 con 0x7f5afc1397c0 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 client.144492.objecter ms_dispatch 0x7f5afc000ba0 osd_map(698..698 src has 501..698) 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 3 client.144492.objecter handle_osd_map got epochs [698,698] > 0 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 3 client.144492.objecter handle_osd_map decoding full epoch 698 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 0x7f5afc135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.188 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 20 client.144492.objecter dump_active .. 0 homeless 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient: get_auth_request con 0x7f5afc1358b0 auth_method 0 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afaffd640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ae8038e40 0x7f5ae803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afaffd640 10 monclient: get_auth_request con 0x7f5ae8038e40 auth_method 0 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_done global_id 164154 payload 824 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5afc0693b0 con 0x7f5afc1358b0 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afb7fe640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 0x7f5afc135c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000f70 tx=0x7f5aec007770 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 client.144492.objecter ms_handle_connect 0x7f5afc1358b0 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 client.144492.objecter resend_mon_ops 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 client.144492.objecter ms_handle_connect 0x7f5ae8038e40 2024-04-19T01:03:08.189 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5afaffd640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ae8038e40 0x7f5ae803b300 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 crypto rx=0x7f5af00ad000 tx=0x7f5af00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5aec00f780 con 0x7f5afc1358b0 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 monclient: dump: 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.190 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5aec018070 con 0x7f5afc1358b0 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 msgr2=0x7f5afc135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 0x7f5afc135c90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000f70 tx=0x7f5aec007770 comp rx=0 tx=0).stop 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 msgr2=0x7f5afc135c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.191 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.168+0000 7f5ae3fff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1358b0 0x7f5afc135c90 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5aec000f70 tx=0x7f5aec007770 comp rx=0 tx=0).stop 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5b01cf4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 0x7f5afc13bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5b01cf4640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5afc068240 con 0x7f5afc1361d0 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 0x7f5afc13bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient: get_auth_request con 0x7f5afc1361d0 auth_method 0 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.193 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 10 monclient(hunting): handle_auth_done global_id 164163 payload 824 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 1 -- 172.21.15.43:0/3114959900 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5afc076190 con 0x7f5afc1361d0 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5afb7fe640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 0x7f5afc13bdd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5aec017000 tx=0x7f5aec000ee0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 client.144492.objecter ms_handle_connect 0x7f5afc1361d0 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 client.144492.objecter resend_mon_ops 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5aec00f800 con 0x7f5afc1361d0 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 monclient: dump: 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.194 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.195 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f5aec004ac0 con 0x7f5afc1361d0 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 msgr2=0x7f5afc13bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 0x7f5afc13bdd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5aec017000 tx=0x7f5aec000ee0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 msgr2=0x7f5afc13bdd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.176+0000 7f5ae3fff640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5afc1361d0 0x7f5afc13bdd0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5aec017000 tx=0x7f5aec000ee0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:08.196 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19543, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 9, 2024-04-19T01:03:08.197 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.198 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.199 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:08.200 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.201 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.202 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.203 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.204 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.205 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.206 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.207 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.208 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.209 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.210 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.211 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.212 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.213 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:08.214 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.215 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.216 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.217 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:08.218 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 10 librados: watch_flush enter 2024-04-19T01:03:08.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 10 librados: watch_flush exit 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 20 client.144492.objecter shutdown clearing up homeless session... 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 10 client.144492.objecter successfully canceled tick 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ae8038e40 msgr2=0x7f5ae803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5ae8038e40 0x7f5ae803b300 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 crypto rx=0x7f5af00ad000 tx=0x7f5af00a6000 comp rx=0 tx=0).stop 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 10 monclient: shutdown 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 msgr2=0x7f5afc1d1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 --2- 172.21.15.43:0/3114959900 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5afc1397c0 0x7f5afc1d1420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5af400efc0 tx=0x7f5af400c490 comp rx=0 tx=0).stop 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5afbfff640 1 -- 172.21.15.43:0/3114959900 reap_dead start 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5afbfff640 1 -- 172.21.15.43:0/3114959900 reap_dead start 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 shutdown_connections 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 >> 172.21.15.43:0/3114959900 conn(0x7f5afc0ab2f0 msgr2=0x7f5afc1376a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 shutdown_connections 2024-04-19T01:03:08.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 -- 172.21.15.43:0/3114959900 wait complete. 2024-04-19T01:03:08.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.180+0000 7f5b01cf4640 1 librados: shutdown 2024-04-19T01:03:08.224 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:08.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T01:03:08.224 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient: get_monmap_and_config 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient: monmap: 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:08.351370+0000 2024-04-19T01:03:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:08.351370+0000 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:08.349 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 Processor -- start 2024-04-19T01:03:08.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 -- start start 2024-04-19T01:03:08.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient: init 2024-04-19T01:03:08.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient: _add_conns ranks=[8,7,0,2,9,1,6,3,4,5,10] 2024-04-19T01:03:08.360 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c20138cd0 0x7f7c201390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): picked mon.noname-e con 0x7f7c20138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c200a5b80 0x7f7c201395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): picked mon.noname-i con 0x7f7c200a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c201358b0 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c200a3dc0 con 0x7f7c201358b0 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f7c200a4410 con 0x7f7c200a5b80 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7c20139b30 con 0x7f7c20138cd0 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c26e08640 10 monclient(hunting): authenticate will time out at 2058.478466s 2024-04-19T01:03:08.361 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c253a5640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f7c201358b0 0x7f7c20135d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:47064/0 (socket says 172.21.15.43:47064) 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c253a5640 1 -- 172.21.15.43:0/3617504218 learned_addr learned my addr 172.21.15.43:0/3617504218 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c20138cd0 0x7f7c201390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): get_auth_request con 0x7f7c20138cd0 auth_method 0 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1ffff640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c200a5b80 0x7f7c201395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.362 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request con 0x7f7c200a5b80 auth_method 0 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_done global_id 144485 payload 274 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 1 -- 172.21.15.43:0/3617504218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c200a5b80 msgr2=0x7f7c201395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c200a5b80 0x7f7c201395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1f7fe640 1 -- 172.21.15.43:0/3617504218 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554224333 0 0) 0x7f7c200a3dc0 con 0x7f7c201358b0 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 1 -- 172.21.15.43:0/3617504218 >> v1:172.21.15.43:6789/0 conn(0x7f7c201358b0 legacy=0x7f7c20135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 1 monclient: found mon.noname-e 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.363 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c24ba4640 1 -- 172.21.15.43:0/3617504218 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c200a40a0 con 0x7f7c20138cd0 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.348+0000 7f7c1f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c20138cd0 0x7f7c201390b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c140097f0 tx=0x7f7c1401c340 comp rx=0 tx=0).ready entity=mon.8 client_cookie=b1432c609d22f869 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 1 -- 172.21.15.43:0/3617504218 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c140b0070 con 0x7f7c20138cd0 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: dump: 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.364 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 1 -- 172.21.15.43:0/3617504218 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c140aa810 con 0x7f7c20138cd0 2024-04-19T01:03:08.365 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 1 -- 172.21.15.43:0/3617504218 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c140a9300 con 0x7f7c20138cd0 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 5 monclient: authenticate success, global_id 144485 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 4 set_mon_vals no callback set 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1f7fe640 10 monclient: dump: 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.366 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: shutdown 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- 172.21.15.43:0/3617504218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c20138cd0 msgr2=0x7f7c201390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c20138cd0 0x7f7c201390b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c140097f0 tx=0x7f7c1401c340 comp rx=0 tx=0).stop 2024-04-19T01:03:08.367 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- 172.21.15.43:0/3617504218 shutdown_connections 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f7c200a5b80 0x7f7c201395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 --2- 172.21.15.43:0/3617504218 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c20138cd0 0x7f7c201390b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- 172.21.15.43:0/3617504218 >> 172.21.15.43:0/3617504218 conn(0x7f7c200ab2f0 msgr2=0x7f7c200ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- 172.21.15.43:0/3617504218 shutdown_connections 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- 172.21.15.43:0/3617504218 wait complete. 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: monmap: 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:08.368 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 librados: starting msgr at 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 librados: starting objecter 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 Processor -- start 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- start start 2024-04-19T01:03:08.369 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 librados: setting wanted keys 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 librados: calling monclient init 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: init 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient: _add_conns ranks=[0,8,4,7,3,6,5,2,1] 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c200a5b80 0x7f7c2012aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): picked mon.noname-a con 0x7f7c200a5b80 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c201358b0 0x7f7c2012afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): picked mon.noname-e con 0x7f7c201358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c20138cd0 0x7f7c201324f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): picked mon.noname-c con 0x7f7c20138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f7c20143ab0 con 0x7f7c200a5b80 2024-04-19T01:03:08.370 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f7c200a20a0 con 0x7f7c20138cd0 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f7c20143c30 con 0x7f7c201358b0 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c26e08640 10 monclient(hunting): authenticate will time out at 2058.483853s 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c201358b0 0x7f7c2012afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c24ba4640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c200a5b80 0x7f7c2012aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c201358b0 0x7f7c2012afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52814/0 (socket says 172.21.15.43:52814) 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c24ba4640 10 monclient(hunting): get_auth_request con 0x7f7c200a5b80 auth_method 0 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c1ffff640 1 -- 172.21.15.43:0/202070844 learned_addr learned my addr 172.21.15.43:0/202070844 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c24ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.371 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.352+0000 7f7c253a5640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c20138cd0 0x7f7c201324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request con 0x7f7c201358b0 auth_method 0 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): get_auth_request con 0x7f7c20138cd0 auth_method 0 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient(hunting): handle_auth_done global_id 144494 payload 824 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c20138cd0 msgr2=0x7f7c201324f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f7c20138cd0 0x7f7c201324f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c200a5b80 msgr2=0x7f7c2012aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f7c200a5b80 0x7f7c2012aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 monclient: found mon.noname-e 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.373 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c200a3040 con 0x7f7c201358b0 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1ffff640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c201358b0 0x7f7c2012afa0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000d7d0 tx=0x7f7c1000dc90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 client.?.objecter ms_handle_connect 0x7f7c201358b0 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c1000bb30 con 0x7f7c201358b0 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: dump: 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.374 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c1000bcf0 con 0x7f7c201358b0 2024-04-19T01:03:08.375 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 5 monclient: authenticate success, global_id 144494 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 monclient: _renew_subs 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c20133f10 con 0x7f7c201358b0 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c100148a0 con 0x7f7c201358b0 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 client.144494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: dump: 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.376 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1e7fc640 4 set_mon_vals no callback set 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 monclient: _renew_subs 2024-04-19T01:03:08.377 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c2009be80 con 0x7f7c201358b0 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 1 librados: init done 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 0x7f7c20132a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7c2006d140 con 0x7f7c20134a70 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f7c1001b030 con 0x7f7c201358b0 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c0c038e10 0x7f7c0c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 0x7f7c20132a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(698..698 src has 501..698) ==== 6356+0+0 (secure 0 0 0) 0x7f7c10019020 con 0x7f7c201358b0 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 client.144494.objecter ms_dispatch 0x7f7c20000ba0 osd_map(698..698 src has 501..698) 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 3 client.144494.objecter handle_osd_map got epochs [698,698] > 0 2024-04-19T01:03:08.378 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 3 client.144494.objecter handle_osd_map decoding full epoch 698 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient: get_auth_request con 0x7f7c20134a70 auth_method 0 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 20 client.144494.objecter dump_active .. 0 homeless 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c0c038e10 0x7f7c0c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 10 monclient: get_auth_request con 0x7f7c0c038e10 auth_method 0 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_done global_id 75029 payload 824 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c200693b0 con 0x7f7c20134a70 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 0x7f7c20132a30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7c200a4380 tx=0x7f7c14019140 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.379 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 client.144494.objecter ms_handle_connect 0x7f7c20134a70 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 client.144494.objecter resend_mon_ops 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 client.144494.objecter ms_handle_connect 0x7f7c0c038e10 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c253a5640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c0c038e10 0x7f7c0c03b2d0 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7f7c18004fd0 tx=0x7f7c18001850 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c140b0040 con 0x7f7c20134a70 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.356+0000 7f7c1d7fa640 10 monclient: dump: 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.380 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7c140b1040 con 0x7f7c20134a70 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 msgr2=0x7f7c20132a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 0x7f7c20132a30 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7c200a4380 tx=0x7f7c14019140 comp rx=0 tx=0).stop 2024-04-19T01:03:08.381 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 msgr2=0x7f7c20132a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.360+0000 7f7c1d7fa640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20134a70 0x7f7c20132a30 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c26e08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:08.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c26e08640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 0x7f7c20133380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c26e08640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f7c20068240 con 0x7f7c20132f70 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 0x7f7c20133380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient: get_auth_request con 0x7f7c20132f70 auth_method 0 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 10 monclient(hunting): handle_auth_done global_id 75038 payload 824 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 1 -- 172.21.15.43:0/202070844 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c201338c0 con 0x7f7c20132f70 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c24ba4640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 0x7f7c20133380 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7c1401aa50 tx=0x7f7c140a8000 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c1d7fa640 10 client.144494.objecter ms_handle_connect 0x7f7c20132f70 2024-04-19T01:03:08.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.364+0000 7f7c1d7fa640 10 client.144494.objecter resend_mon_ops 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f7c140abdd0 con 0x7f7c20132f70 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 10 monclient: dump: 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.385 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f7c140c0750 con 0x7f7c20132f70 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 msgr2=0x7f7c20133380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 0x7f7c20133380 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7c1401aa50 tx=0x7f7c140a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 msgr2=0x7f7c20133380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c1d7fa640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f7c20132f70 0x7f7c20133380 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 567985, 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:08.387 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 9, 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.388 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.389 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:08.390 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.391 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.392 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.393 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:08.394 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.395 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.396 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.396 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.396 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.396 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.397 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.397 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.397 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:08.397 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.398 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.399 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.400 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:08.401 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.402 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.403 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.404 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.405 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.406 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.407 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.408 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:08.409 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:08.410 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 10 librados: watch_flush enter 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 10 librados: watch_flush exit 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 20 client.144494.objecter shutdown clearing up homeless session... 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 10 client.144494.objecter successfully canceled tick 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c0c038e10 msgr2=0x7f7c0c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f7c0c038e10 0x7f7c0c03b2d0 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7f7c18004fd0 tx=0x7f7c18001850 comp rx=0 tx=0).stop 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 10 monclient: shutdown 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c201358b0 msgr2=0x7f7c2012afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 --2- 172.21.15.43:0/202070844 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f7c201358b0 0x7f7c2012afa0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c1000d7d0 tx=0x7f7c1000dc90 comp rx=0 tx=0).stop 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c253a5640 1 -- 172.21.15.43:0/202070844 reap_dead start 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c253a5640 1 -- 172.21.15.43:0/202070844 reap_dead start 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 shutdown_connections 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 >> 172.21.15.43:0/202070844 conn(0x7f7c200ab2f0 msgr2=0x7f7c200a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 shutdown_connections 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 -- 172.21.15.43:0/202070844 wait complete. 2024-04-19T01:03:08.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.368+0000 7f7c26e08640 1 librados: shutdown 2024-04-19T01:03:08.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:08.414 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T01:03:08.414 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient: get_monmap_and_config 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient: monmap: 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:08.535248+0000 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:08.535248+0000 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:08.533 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 Processor -- start 2024-04-19T01:03:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 -- start start 2024-04-19T01:03:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient: init 2024-04-19T01:03:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient: _add_conns ranks=[7,4,9,0,10,1,8,6,3,2,5] 2024-04-19T01:03:08.536 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a440a1e00 0x7f4a440a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): picked mon.noname-i con 0x7f4a440a1e00 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 0x7f4a440a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): picked mon.noname-c con 0x7f4a440a1190 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4a440a4350 0x7f4a4413fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): picked mon.noname-a con 0x7f4a440a4350 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4406c690 con 0x7f4a440a4350 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f4a440735a0 con 0x7f4a440a1190 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f4a4409fff0 con 0x7f4a440a1e00 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a490f7640 10 monclient(hunting): authenticate will time out at 2058.662975s 2024-04-19T01:03:08.537 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 0x7f4a440a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f4a440a4350 0x7f4a4413fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 0x7f4a440a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:47598/0 (socket says 172.21.15.43:47598) 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 1 -- 172.21.15.43:0/407079831 learned_addr learned my addr 172.21.15.43:0/407079831 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): get_auth_request con 0x7f4a440a1190 auth_method 0 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): get_auth_request con 0x7f4a440a4350 auth_method 0 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.538 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a440a1e00 0x7f4a440a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): get_auth_request con 0x7f4a440a1e00 auth_method 0 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.539 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient(hunting): handle_auth_done global_id 65790 payload 274 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 -- 172.21.15.43:0/407079831 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a440a1e00 msgr2=0x7f4a440a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a440a1e00 0x7f4a440a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 -- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 msgr2=0x7f4a440a3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 0x7f4a440a3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 monclient: found mon.noname-a 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4359e640 1 -- 172.21.15.43:0/407079831 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4406c330 con 0x7f4a440a4350 2024-04-19T01:03:08.541 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.532+0000 7f4a4259c640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 0x7f4a440a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:08.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a4359e640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a4350 0x7f4a4413fac0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009d90 tx=0x7f4a3c01c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad37a3fb8b0d624 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 1 -- 172.21.15.43:0/407079831 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a3c0a6980 con 0x7f4a440a4350 2024-04-19T01:03:08.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.548 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: dump: 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 1 -- 172.21.15.43:0/407079831 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c0a6b40 con 0x7f4a440a4350 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a490f7640 5 monclient: authenticate success, global_id 65790 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a490f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a490f7640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 1 -- 172.21.15.43:0/407079831 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a3c0a6d30 con 0x7f4a440a4350 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.550 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.536+0000 7f4a490f7640 4 set_mon_vals no callback set 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a41d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a41d9b640 10 monclient: dump: 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.551 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: shutdown 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a4350 msgr2=0x7f4a4413fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a4350 0x7f4a4413fac0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009d90 tx=0x7f4a3c01c8c0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/407079831 shutdown_connections 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a4350 0x7f4a4413fac0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f4a440a1190 0x7f4a440a3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- 172.21.15.43:0/407079831 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a440a1e00 0x7f4a440a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/407079831 >> 172.21.15.43:0/407079831 conn(0x7f4a440a84c0 msgr2=0x7f4a440a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/407079831 shutdown_connections 2024-04-19T01:03:08.552 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/407079831 wait complete. 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: monmap: 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.553 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 librados: starting msgr at 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 librados: starting objecter 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 Processor -- start 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- start start 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 librados: setting wanted keys 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 librados: calling monclient init 2024-04-19T01:03:08.554 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: init 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: _add_conns ranks=[0,5,1,2,7,3,6,4,8] 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a1190 0x7f4a441d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): picked mon.noname-a con 0x7f4a440a1190 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4a440a1e00 0x7f4a4413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): picked mon.noname-h con 0x7f4a440a1e00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a440a4350 0x7f4a44122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): picked mon.noname-f con 0x7f4a440a4350 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4a44142820 con 0x7f4a440a1190 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4a440a2430 con 0x7f4a440a4350 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f4a441429a0 con 0x7f4a440a1e00 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient(hunting): authenticate will time out at 2058.670347s 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a1190 0x7f4a441d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a1190 0x7f4a441d0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:45054/0 (socket says 172.21.15.43:45054) 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 -- 172.21.15.43:0/704852976 learned_addr learned my addr 172.21.15.43:0/704852976 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): get_auth_request con 0x7f4a440a1190 auth_method 0 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4359e640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a440a4350 0x7f4a44122920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4259c640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4a440a1e00 0x7f4a4413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4359e640 10 monclient(hunting): get_auth_request con 0x7f4a440a4350 auth_method 0 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4359e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4359e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient(hunting): handle_auth_done global_id 65799 payload 824 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4a440a1e00 msgr2=0x7f4a4413aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4a440a1e00 0x7f4a4413aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a440a4350 msgr2=0x7f4a44122920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4a440a4350 0x7f4a44122920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 monclient: found mon.noname-a 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a441263d0 con 0x7f4a440a1190 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4259c640 10 monclient: get_auth_request con 0x7f4a440a1e00 auth_method 0 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4259c640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f4a440a1e00 0x7f4a4413aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a42d9d640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a1190 0x7f4a441d0930 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800e930 tx=0x7f4a3800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f4a440a1190 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a38017070 con 0x7f4a440a1190 2024-04-19T01:03:08.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: dump: 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.559 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 5 monclient: authenticate success, global_id 65799 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: _renew_subs 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a38013470 con 0x7f4a440a1190 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a44073270 con 0x7f4a440a1190 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a38013a90 con 0x7f4a440a1190 2024-04-19T01:03:08.560 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 client.65799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a2b7fe640 10 monclient: dump: 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.561 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a4159a640 4 set_mon_vals no callback set 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: _renew_subs 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a44124070 con 0x7f4a440a1190 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.540+0000 7f4a490f7640 1 librados: init done 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a490f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a490f7640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 0x7f4a44122e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f4a44125d70 con 0x7f4a4413af50 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4a3801a070 con 0x7f4a440a1190 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 0x7f4a44122e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4a30038e40 0x7f4a3003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient: get_auth_request con 0x7f4a4413af50 auth_method 0 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(698..698 src has 501..698) ==== 6356+0+0 (secure 0 0 0) 0x7f4a3800f040 con 0x7f4a440a1190 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 client.65799.objecter ms_dispatch 0x7f4a44000ba0 osd_map(698..698 src has 501..698) 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 3 client.65799.objecter handle_osd_map got epochs [698,698] > 0 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 3 client.65799.objecter handle_osd_map decoding full epoch 698 2024-04-19T01:03:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 20 client.65799.objecter dump_active .. 0 homeless 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4359e640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4a30038e40 0x7f4a3003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4359e640 10 monclient: get_auth_request con 0x7f4a30038e40 auth_method 0 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 10 monclient(hunting): handle_auth_done global_id 164176 payload 824 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a44125b30 con 0x7f4a4413af50 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4259c640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 0x7f4a44122e60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a34010790 tx=0x7f4a34010c50 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 client.65799.objecter ms_handle_connect 0x7f4a4413af50 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 client.65799.objecter resend_mon_ops 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 client.65799.objecter ms_handle_connect 0x7f4a30038e40 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a4359e640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4a30038e40 0x7f4a3003b300 secure :-1 s=READY pgs=3647 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0af000 tx=0x7f4a3c0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a3400f040 con 0x7f4a4413af50 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 monclient: dump: 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.565 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4a3401c400 con 0x7f4a4413af50 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 msgr2=0x7f4a44122e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 0x7f4a44122e60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a34010790 tx=0x7f4a34010c50 comp rx=0 tx=0).stop 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 msgr2=0x7f4a44122e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.566 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.544+0000 7f4a2b7fe640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a4413af50 0x7f4a44122e60 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4a34010790 tx=0x7f4a34010c50 comp rx=0 tx=0).stop 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 0x7f4a4413bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f4a441237b0 con 0x7f4a441233a0 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 0x7f4a4413bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient: get_auth_request con 0x7f4a441233a0 auth_method 0 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.568 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 10 monclient(hunting): handle_auth_done global_id 164185 payload 824 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 1 -- 172.21.15.43:0/704852976 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a44073050 con 0x7f4a441233a0 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4259c640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 0x7f4a4413bd50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a3401a000 tx=0x7f4a340046d0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 client.65799.objecter ms_handle_connect 0x7f4a441233a0 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 client.65799.objecter resend_mon_ops 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4a3401de20 con 0x7f4a441233a0 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 monclient: dump: 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.569 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.570 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f4a340059a0 con 0x7f4a441233a0 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 msgr2=0x7f4a4413bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 0x7f4a4413bd50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a3401a000 tx=0x7f4a340046d0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 msgr2=0x7f4a4413bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a2b7fe640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f4a441233a0 0x7f4a4413bd50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 118, 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 19919, 2024-04-19T01:03:08.572 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 9, 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:08.573 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.574 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.575 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:08.576 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:08.577 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.578 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.579 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.580 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.581 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:08.582 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.583 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.584 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.585 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.586 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.587 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.588 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:08.589 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.590 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:08.591 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.592 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:08.593 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:08.594 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:08.595 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 10 librados: watch_flush enter 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 10 librados: watch_flush exit 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 20 client.65799.objecter shutdown clearing up homeless session... 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 10 client.65799.objecter successfully canceled tick 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4a30038e40 msgr2=0x7f4a3003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4a30038e40 0x7f4a3003b300 secure :-1 s=READY pgs=3647 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0af000 tx=0x7f4a3c0a7000 comp rx=0 tx=0).stop 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 10 monclient: shutdown 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a1190 msgr2=0x7f4a441d0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 --2- 172.21.15.43:0/704852976 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4a440a1190 0x7f4a441d0930 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800e930 tx=0x7f4a3800edf0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4359e640 1 -- 172.21.15.43:0/704852976 reap_dead start 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a4359e640 1 -- 172.21.15.43:0/704852976 reap_dead start 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 shutdown_connections 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 >> 172.21.15.43:0/704852976 conn(0x7f4a440a84c0 msgr2=0x7f4a44071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 shutdown_connections 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 -- 172.21.15.43:0/704852976 wait complete. 2024-04-19T01:03:08.598 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.552+0000 7f4a490f7640 1 librados: shutdown 2024-04-19T01:03:08.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:03:08.599 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a', 'i', 'c', 'd'] 2024-04-19T01:03:08.599 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-04-19T01:03:08.599 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-19T01:03:08.599 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T01:03:08.599 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T01:03:08.599 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:03:08.619 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T01:03:08.620 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i 2024-04-19T01:03:08.620 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.i store 2024-04-19T01:03:08.620 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i sync_force --yes-i-really-mean-it 2024-04-19T01:03:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient: get_monmap_and_config 2024-04-19T01:03:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient: monmap: 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:08.792653+0000 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:08.792653+0000 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.790 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.791 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.791 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:08.791 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:08.791 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 Processor -- start 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 -- start start 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient: init 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient: _add_conns ranks=[9,8,5,0,7,10,3,6,2,1,4] 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe620072910 0x7fe6201391e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): picked mon.noname-a con 0x7fe620072910 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 0x7fe620073150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): picked mon.noname-e con 0x7fe620072cf0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe6200a4b10 0x7fe6200a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): picked mon.noname-h con 0x7fe6200a4b10 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:08.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fe62006dc10 con 0x7fe620072910 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fe620073950 con 0x7fe6200a4b10 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe62006f2d0 con 0x7fe620072cf0 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe627f77640 10 monclient(hunting): authenticate will time out at 2058.919832s 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe625d13640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7fe620072910 msgr2=0x7fe6201391e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe625d13640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fe620072910 0x7fe6201391e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:08.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.788+0000 7fe625512640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 0x7fe620073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 0x7fe620073150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52830/0 (socket says 172.21.15.43:52830) 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 -- 172.21.15.43:0/832064712 learned_addr learned my addr 172.21.15.43:0/832064712 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): get_auth_request con 0x7fe620072cf0 auth_method 0 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe626514640 1 --2- 172.21.15.43:0/832064712 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe6200a4b10 0x7fe6200a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe626514640 10 monclient(hunting): get_auth_request con 0x7fe6200a4b10 auth_method 0 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe626514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe626514640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe626514640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient(hunting): handle_auth_done global_id 144503 payload 274 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 -- 172.21.15.43:0/832064712 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe6200a4b10 msgr2=0x7fe6200a4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 --2- 172.21.15.43:0/832064712 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe6200a4b10 0x7fe6200a4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 -- 172.21.15.43:0/832064712 >> v2:172.21.15.43:3300/0 conn(0x7fe620072910 msgr2=0x7fe6201391e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 --2- 172.21.15.43:0/832064712 >> v2:172.21.15.43:3300/0 conn(0x7fe620072910 0x7fe6201391e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 monclient: found mon.noname-e 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 -- 172.21.15.43:0/832064712 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62006f5b0 con 0x7fe620072cf0 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe625512640 1 --2- 172.21.15.43:0/832064712 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 0x7fe620073150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe614009870 tx=0x7fe61401c440 comp rx=0 tx=0).ready entity=mon.8 client_cookie=bc4a4a59f1885a58 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 1 -- 172.21.15.43:0/832064712 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6140b0070 con 0x7fe620072cf0 2024-04-19T01:03:08.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: dump: 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.806 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.807 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.810 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 1 -- 172.21.15.43:0/832064712 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6140a4e50 con 0x7fe620072cf0 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 5 monclient: authenticate success, global_id 144503 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 1 -- 172.21.15.43:0/832064712 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe6140aac80 con 0x7fe620072cf0 2024-04-19T01:03:08.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 4 set_mon_vals no callback set 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe624d11640 10 monclient: dump: 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.812 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 10 monclient: shutdown 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 -- 172.21.15.43:0/832064712 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 msgr2=0x7fe620073150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 --2- 172.21.15.43:0/832064712 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 0x7fe620073150 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe614009870 tx=0x7fe61401c440 comp rx=0 tx=0).stop 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 -- 172.21.15.43:0/832064712 shutdown_connections 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 --2- 172.21.15.43:0/832064712 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fe6200a4b10 0x7fe6200a4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 --2- 172.21.15.43:0/832064712 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe620072cf0 0x7fe620073150 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 --2- 172.21.15.43:0/832064712 >> v2:172.21.15.43:3300/0 conn(0x7fe620072910 0x7fe6201391e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 -- 172.21.15.43:0/832064712 >> 172.21.15.43:0/832064712 conn(0x7fe6200a99e0 msgr2=0x7fe6200a5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 -- 172.21.15.43:0/832064712 shutdown_connections 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 1 -- 172.21.15.43:0/832064712 wait complete. 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 10 monclient: build_initial_monmap 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.792+0000 7fe627f77640 10 monclient: monmap: 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:08.814 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:08.815 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 librados: starting msgr at 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 librados: starting objecter 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 Processor -- start 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- start start 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 librados: setting wanted keys 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 librados: calling monclient init 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: init 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: _add_conns ranks=[4,0,8,6,3,5,7,2,1] 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe620072910 0x7fe62012c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): picked mon.noname-c con 0x7fe620072910 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe620072cf0 0x7fe62012cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): picked mon.noname-a con 0x7fe620072cf0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe6200a4b10 0x7fe62012d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): picked mon.noname-e con 0x7fe6200a4b10 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:08.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fe620142bf0 con 0x7fe620072cf0 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fe620142a70 con 0x7fe620072910 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fe620142d70 con 0x7fe6200a4b10 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): authenticate will time out at 2058.924784s 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625512640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe620072cf0 msgr2=0x7fe62012cd60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625512640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe620072cf0 0x7fe62012cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe620072910 0x7fe62012c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe6200a4b10 0x7fe62012d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe6200a4b10 0x7fe62012d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:52834/0 (socket says 172.21.15.43:52834) 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 -- 172.21.15.43:0/1044147331 learned_addr learned my addr 172.21.15.43:0/1044147331 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): get_auth_request con 0x7fe620072910 auth_method 0 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): get_auth_request con 0x7fe6200a4b10 auth_method 0 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient(hunting): handle_auth_done global_id 144512 payload 824 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe620072910 msgr2=0x7fe62012c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe620072910 0x7fe62012c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe620072cf0 msgr2=0x7fe62012cd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fe620072cf0 0x7fe62012cd60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient: _finish_hunting 0 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 monclient: found mon.noname-e 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe620136300 con 0x7fe6200a4b10 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fe620072910 0x7fe62012c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe626514640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe6200a4b10 0x7fe62012d2a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe61000d5c0 tx=0x7fe61000da80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 client.?.objecter ms_handle_connect 0x7fe6200a4b10 2024-04-19T01:03:08.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe610012a60 con 0x7fe6200a4b10 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: dump: 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.822 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe610019070 con 0x7fe6200a4b10 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 5 monclient: authenticate success, global_id 144512 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: _renew_subs 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6200a19c0 con 0x7fe6200a4b10 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe610012c60 con 0x7fe6200a4b10 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 monclient: dump: 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61ffff640 4 set_mon_vals no callback set 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 client.144512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: _renew_subs 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6200a16e0 con 0x7fe6200a4b10 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 librados: init done 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 0x7fe620134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe6200a3490 con 0x7fe6200a2730 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fe6100125f0 con 0x7fe6200a4b10 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe60c038e10 0x7fe60c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(698..698 src has 501..698) ==== 6356+0+0 (secure 0 0 0) 0x7fe61000f040 con 0x7fe6200a4b10 2024-04-19T01:03:08.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 client.144512.objecter ms_dispatch 0x7fe620000ba0 osd_map(698..698 src has 501..698) 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 0x7fe620134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 3 client.144512.objecter handle_osd_map got epochs [698,698] > 0 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 3 client.144512.objecter handle_osd_map decoding full epoch 698 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient: get_auth_request con 0x7fe6200a2730 auth_method 0 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 20 client.144512.objecter dump_active .. 0 homeless 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625512640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe60c038e10 0x7fe60c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625512640 10 monclient: get_auth_request con 0x7fe60c038e10 auth_method 0 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 10 monclient(hunting): handle_auth_done global_id 164194 payload 824 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe620136700 con 0x7fe6200a2730 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 client.144512.objecter ms_handle_connect 0x7fe6200a2730 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe61e7fc640 10 client.144512.objecter resend_mon_ops 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.796+0000 7fe625d13640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 0x7fe620134740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe618003d10 tx=0x7fe6180079c0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 10 client.144512.objecter ms_handle_connect 0x7fe60c038e10 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe625512640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe60c038e10 0x7fe60c03b2d0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 crypto rx=0x7fe6140ad000 tx=0x7fe614008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe61801a070 con 0x7fe6200a2730 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 10 monclient: dump: 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.829 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.830 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.830 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.830 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe61800a6b0 con 0x7fe6200a2730 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 msgr2=0x7fe620134740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 0x7fe620134740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe618003d10 tx=0x7fe6180079c0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 msgr2=0x7fe620134740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.800+0000 7fe61e7fc640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe6200a2730 0x7fe620134740 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe618003d10 tx=0x7fe6180079c0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.804+0000 7fe627f77640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.804+0000 7fe627f77640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 0x7fe620135060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.804+0000 7fe627f77640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fe62006d140 con 0x7fe620134c80 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 0x7fe620135060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient: get_auth_request con 0x7fe620134c80 auth_method 0 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:08.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 10 monclient(hunting): handle_auth_done global_id 164203 payload 824 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 1 -- 172.21.15.43:0/1044147331 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7fe6201355a0 con 0x7fe620134c80 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe625d13640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 0x7fe620135060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe618017000 tx=0x7fe618005d60 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 client.144512.objecter ms_handle_connect 0x7fe620134c80 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 client.144512.objecter resend_mon_ops 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fe61801a030 con 0x7fe620134c80 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 monclient: dump: 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:08.835 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7fe61801c830 con 0x7fe620134c80 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-04-19T01:03:08.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 msgr2=0x7fe620135060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 0x7fe620135060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe618017000 tx=0x7fe618005d60 comp rx=0 tx=0).stop 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 msgr2=0x7fe620135060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe61e7fc640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fe620134c80 0x7fe620135060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stdout: "ret": 0, 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-04-19T01:03:08.837 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 10 librados: watch_flush enter 2024-04-19T01:03:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 10 librados: watch_flush exit 2024-04-19T01:03:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 20 client.144512.objecter shutdown clearing up homeless session... 2024-04-19T01:03:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 10 client.144512.objecter successfully canceled tick 2024-04-19T01:03:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe60c038e10 msgr2=0x7fe60c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fe60c038e10 0x7fe60c03b2d0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 crypto rx=0x7fe6140ad000 tx=0x7fe614008040 comp rx=0 tx=0).stop 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 10 monclient: shutdown 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe6200a4b10 msgr2=0x7fe62012d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 --2- 172.21.15.43:0/1044147331 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fe6200a4b10 0x7fe62012d2a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe61000d5c0 tx=0x7fe61000da80 comp rx=0 tx=0).stop 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe626514640 1 -- 172.21.15.43:0/1044147331 reap_dead start 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe626514640 1 -- 172.21.15.43:0/1044147331 reap_dead start 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 shutdown_connections 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 >> 172.21.15.43:0/1044147331 conn(0x7fe6200a99e0 msgr2=0x7fe6200717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 shutdown_connections 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 -- 172.21.15.43:0/1044147331 wait complete. 2024-04-19T01:03:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:08.808+0000 7fe627f77640 1 librados: shutdown 2024-04-19T01:03:08.841 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-04-19T01:03:08.841 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-04-19T01:03:08.841 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:03:08.858 INFO:tasks.ceph.mon.i:Stopped 2024-04-19T01:03:08.859 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-19T01:03:08.859 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T01:03:08.859 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T01:03:08.859 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:03:08.876 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T01:03:08.876 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.d 2024-04-19T01:03:08.876 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T01:03:08.876 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-04-19T01:03:08.876 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:03:08.933 INFO:tasks.ceph.mon.d:Stopped 2024-04-19T01:03:08.933 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-04-19T01:03:08.934 INFO:tasks.ceph.mon.b:Sent signal 19 2024-04-19T01:03:08.934 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T01:03:23.935 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-04-19T01:03:23.936 INFO:tasks.ceph.mon.b:Sent signal 18 2024-04-19T01:03:23.936 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 5 2024-04-19T01:03:23.936 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:03:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient: get_monmap_and_config 2024-04-19T01:03:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient: build_initial_monmap 2024-04-19T01:03:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient: monmap: 2024-04-19T01:03:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:24.069 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:24.072191+0000 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:24.072191+0000 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:24.070 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:24.071 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 Processor -- start 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 -- start start 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient: init 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient: _add_conns ranks=[2,8,10,7,9,5,1,3,0,6,4] 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): picked mon.noname-b con 0x7efccc138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): picked mon.noname-e con 0x7efccc0a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:24.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): picked mon.noname-f con 0x7efccc1358b0 addr v2:172.21.15.161:3300/0 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efccc1358b0 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efccc139b30 con 0x7efccc138cd0 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7efccc0a3dc0 con 0x7efccc0a5b80 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.067+0000 7efcd0f63640 10 monclient(hunting): authenticate will time out at 2074.199906s 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:44724/0 (socket says 172.21.15.43:44724) 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 learned_addr learned my addr 172.21.15.43:0/3858515310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 10 monclient(hunting): get_auth_request con 0x7efccc138cd0 auth_method 0 2024-04-19T01:03:24.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efccc0a5b80 auth_method 0 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:24.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efccc1358b0 auth_method 0 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:24.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 10 monclient(hunting): get_auth_request con 0x7efccc138cd0 auth_method 0 2024-04-19T01:03:24.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efccc0a5b80 auth_method 0 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efccc1358b0 auth_method 0 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:24.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 10 monclient(hunting): get_auth_request con 0x7efccc138cd0 auth_method 0 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efccc0a5b80 auth_method 0 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efccc1358b0 auth_method 0 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:24.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:24.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:24.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 10 monclient(hunting): get_auth_request con 0x7efccc138cd0 auth_method 0 2024-04-19T01:03:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:25.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efccc0a5b80 auth_method 0 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efccc1358b0 auth_method 0 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:25.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:25.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:25.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:27.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.067+0000 7efcc3fff640 10 monclient(hunting): tick 2024-04-19T01:03:27.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.067+0000 7efcc3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:03:27.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.067+0000 7efcc3fff640 1 monclient(hunting): continuing hunt 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.067+0000 7efcc3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.067+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 msgr2=0x7efccc1395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efccc0a5b80 0x7efccc1395f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 msgr2=0x7efccc1390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efccc138cd0 0x7efccc1390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 msgr2=0x7efccc135d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.161:3300/0 conn(0x7efccc1358b0 0x7efccc135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient: _add_conns ranks=[1,3,5,0,6,4,7,9] 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7efcb0003800 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-g con 0x7efcb00073f0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-h con 0x7efcb0009ee0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:27.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efccc0a40a0 con 0x7efcb0003800 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efcb00073f0 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7efccc139b30 con 0x7efcb0009ee0 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc3fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efcb0009ee0 auth_method 0 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efcb00073f0 auth_method 0 2024-04-19T01:03:27.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:27.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:27.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:27.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:27.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efcb0009ee0 auth_method 0 2024-04-19T01:03:27.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efcb00073f0 auth_method 0 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:27.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:27.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:27.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efcb0009ee0 auth_method 0 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efcb00073f0 auth_method 0 2024-04-19T01:03:27.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:27.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:27.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:28.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:28.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efcb0009ee0 auth_method 0 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efcb00073f0 auth_method 0 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-19T01:03:28.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-19T01:03:28.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-19T01:03:28.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:28.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:30.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): tick 2024-04-19T01:03:30.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:03:30.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 monclient(hunting): continuing hunt 2024-04-19T01:03:30.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 msgr2=0x7efcb000c3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efcb0009ee0 0x7efcb000c3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 msgr2=0x7efcb0009880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb00073f0 0x7efcb0009880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.161:6789/0 conn(0x7efcb0003800 legacy=0x7efcb0005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 reap_dead start 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient: _add_conns ranks=[0,4,6,7,9] 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcb000ccc0 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 0x7efcb000b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-c con 0x7efcb0009880 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 0x7efcb000bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-d con 0x7efcb000b8e0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:30.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb000dc90 con 0x7efcb000ccc0 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efcb0009880 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7efccc139b30 con 0x7efcb000b8e0 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc3fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000ccc0 legacy=0x7efcb000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 msgr2=0x7efcb000bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 msgr2=0x7efcb000b3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 0x7efcb000bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:30.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 0x7efcb000b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:30.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 msgr2=0x7efcb000b3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:30.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.275+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 msgr2=0x7efcb000bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:30.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.275+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000ccc0 legacy=0x7efcb000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T01:03:30.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 0x7efcb000b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:30.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 0x7efcb000bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.679+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000ccc0 legacy=0x7efcb000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T01:03:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 msgr2=0x7efcb000b3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.679+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 msgr2=0x7efcb000bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 0x7efcb000bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:30.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:30.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 0x7efcb000b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:31.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:31.483+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000ccc0 legacy=0x7efcb000d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T01:03:31.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:31.483+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 msgr2=0x7efcb000bd00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:31.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:31.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 msgr2=0x7efcb000b3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:31.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 0x7efcb000bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:31.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:31.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 0x7efcb000b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): tick 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 monclient(hunting): continuing hunt 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 msgr2=0x7efcb000bd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb000b8e0 0x7efcb000bd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 msgr2=0x7efcb000b3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb0009880 0x7efcb000b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000ccc0 legacy=0x7efcb000d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient: _add_conns ranks=[7,9] 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 0x7efcb00143d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:33.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-i con 0x7efcb0011fe0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 0x7efcb0016d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-a con 0x7efcb0014910 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efcb0014910 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7efccc139b30 con 0x7efcb0011fe0 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc3fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:33.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.071+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:33.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:33.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.275+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:33.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.275+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:33.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:33.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.679+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:33.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:33.679+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:34.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:34.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:34.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:34.483+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:34.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:34.483+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:36.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): tick 2024-04-19T01:03:36.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 monclient(hunting): continuing hunt 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0011fe0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb0014910 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 reap_dead start 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 reap_dead start 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient: _add_conns ranks=[7,9,4,6,5,10,1,3,0,2,8] 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 0x7efcb000bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-i con 0x7efcb0003800 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 0x7efcb0016d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:36.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-a con 0x7efcb000d0a0 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 0x7efcb00143d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-c con 0x7efcb00063c0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efcb000d0a0 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7efccc139b30 con 0x7efcb00063c0 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7efccc0a3dc0 con 0x7efcb0003800 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc3fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:36.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 msgr2=0x7efcb000bce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:36.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.071+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 0x7efcb000bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:36.275 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.275+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:36.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.275+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:36.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.275+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:36.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.275+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:36.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.275+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 msgr2=0x7efcb000bce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:36.276 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.275+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 0x7efcb000bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-19T01:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.679+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.679+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:36.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.679+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:36.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.679+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:36.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.679+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 msgr2=0x7efcb000bce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:36.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:36.679+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 0x7efcb000bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-19T01:03:37.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:37.483+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:37.483+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:37.483+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:37.483+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:37.483+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 msgr2=0x7efcb000bce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:37.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:37.483+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 0x7efcb000bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-19T01:03:39.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): tick 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): _check_auth_tickets 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 monclient(hunting): continuing hunt 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 msgr2=0x7efcb000bce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efcb0003800 0x7efcb000bce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 msgr2=0x7efcb00143d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7efcb00063c0 0x7efcb00143d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 msgr2=0x7efcb0016d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> v2:172.21.15.43:3300/0 conn(0x7efcb000d0a0 0x7efcb0016d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient: _add_conns ranks=[1,0,6,3,5,2,8,10] 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7efcb0006800 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:39.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcb000b340 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0010fa0 0x7efcb00113d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): picked mon.noname-d con 0x7efcb0010fa0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb000c330 con 0x7efcb000b340 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb000dc90 con 0x7efcb0006800 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efcb0010fa0 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc3fff640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc9d9b640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000b340 legacy=0x7efcb0006c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0010fa0 msgr2=0x7efcb00113d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:39.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0010fa0 0x7efcb00113d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2862313019 0 0) 0x7efcb000dc90 con 0x7efcb0006800 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient(hunting): _negotiate 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient(hunting): my global_id is 144514 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcb4003530 con 0x7efcb0006800 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3017043199 0 0) 0x7efcb4003530 con 0x7efcb0006800 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0010fa0 msgr2=0x7efcb00113d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0010fa0 0x7efcb00113d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.43:6789/0 conn(0x7efcb000b340 legacy=0x7efcb0006c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 monclient: found mon.noname-f-legacy 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: _send_mon_message to mon.noname-f-legacy at v1:172.21.15.161:6789/0 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb0011a90 con 0x7efcb0006800 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 reap_dead start 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7efcbc00f040 con 0x7efcb0006800 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: got monmap 1 from mon.noname-f-legacy (according to old e1) 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 10 monclient: dump: 2024-04-19T01:03:39.081 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.082 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.082 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.082 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.083 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.071+0000 7efcc959a640 1 monclient: mon.f has (v2) addrs [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] but i'm connected to v1:172.21.15.161:6789/0, reconnecting 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 >> v1:172.21.15.161:6789/0 conn(0x7efcb0006800 legacy=0x7efcb0005c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient: _add_conns ranks=[1,3,0,2,6,4,7,5,8] 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efcb4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): picked mon.f con 0x7efcb4004210 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb4004f50 0x7efcb40099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): picked mon.g con 0x7efcb4004f50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcb4009f00 0x7efcb400c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): picked mon.a con 0x7efcb4009f00 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7efccc0a4410 con 0x7efcb4009f00 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efcb400cb30 con 0x7efcb4004210 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7efcb400cce0 con 0x7efcb4004f50 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc959a640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efccad9d640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcb4009f00 msgr2=0x7efcb400c2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efccad9d640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcb4009f00 0x7efcb400c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efcb4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb4004f50 0x7efcb40099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): get_auth_request con 0x7efcb4004210 auth_method 0 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.085 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efcb4004f50 auth_method 0 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcc9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient(hunting): handle_auth_done global_id 144514 payload 293 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb4004f50 msgr2=0x7efcb40099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb4004f50 0x7efcb40099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.086 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcb4009f00 msgr2=0x7efcb400c2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcb4009f00 0x7efcb400c2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 monclient: found mon.f 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient: _send_mon_message to mon.f at v2:172.21.15.161:3300/0 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 1 -- 172.21.15.43:0/3858515310 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb4003530 con 0x7efcb4004210 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcca59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcd0f63640 5 monclient: authenticate success, global_id 144514 2024-04-19T01:03:39.087 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.075+0000 7efcd0f63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcca59c640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efcb4004810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efcb000a2c0 tx=0x7efcbc01d670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd62f28c80bb6547 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcbc0b4070 con 0x7efcb4004210 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: dump: 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.095 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcbc0ad7d0 con 0x7efcb4004210 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 1 -- 172.21.15.43:0/3858515310 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcbc00c9f0 con 0x7efcb4004210 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcd0f63640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcd0f63640 4 set_mon_vals no callback set 2024-04-19T01:03:39.096 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.079+0000 7efcc959a640 10 monclient: dump: 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.097 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 10 monclient: shutdown 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 -- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 msgr2=0x7efcb4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efcb4004810 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7efcb000a2c0 tx=0x7efcbc01d670 comp rx=0 tx=0).stop 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 -- 172.21.15.43:0/3858515310 shutdown_connections 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7efcb4009f00 0x7efcb400c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7efcb4004f50 0x7efcb40099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 --2- 172.21.15.43:0/3858515310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efcb4004810 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 -- 172.21.15.43:0/3858515310 >> 172.21.15.43:0/3858515310 conn(0x7efccc0ab2f0 msgr2=0x7efccc0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 -- 172.21.15.43:0/3858515310 shutdown_connections 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 1 -- 172.21.15.43:0/3858515310 wait complete. 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.083+0000 7efcd0f63640 10 monclient: monmap: 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:39.099 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 librados: starting msgr at 2024-04-19T01:03:39.100 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 librados: starting objecter 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 Processor -- start 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 -- start start 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 librados: setting wanted keys 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 librados: calling monclient init 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient: init 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient: _add_conns ranks=[6,1,8,5,3,2,7,4,0] 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0006800 0x7efccc1d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): picked mon.noname-d con 0x7efcb0006800 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efccc131860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): picked mon.noname-f con 0x7efcb4004210 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 0x7efccc09f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcca59c640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0006800 msgr2=0x7efccc1d0240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): picked mon.noname-e con 0x7efcb4004f50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:39.101 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efccc135cc0 con 0x7efcb4004210 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcca59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0006800 0x7efccc1d0240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7efccc0a20a0 con 0x7efcb0006800 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7efccc135e40 con 0x7efcb4004f50 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient(hunting): authenticate will time out at 2089.218476s 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 0x7efccc09f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 0x7efccc09f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:43224/0 (socket says 172.21.15.43:43224) 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 -- 172.21.15.43:0/1496761570 learned_addr learned my addr 172.21.15.43:0/1496761570 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): get_auth_request con 0x7efcb4004f50 auth_method 0 2024-04-19T01:03:39.102 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efccc131860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request con 0x7efcb4004210 auth_method 0 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient(hunting): handle_auth_done global_id 144521 payload 824 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 -- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0006800 msgr2=0x7efccc1d0240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0006800 0x7efccc1d0240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 -- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 msgr2=0x7efccc131860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efccc131860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 monclient: found mon.noname-e 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 -- 172.21.15.43:0/1496761570 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc0a3590 con 0x7efcb4004f50 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efccad9d640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 0x7efccc09f820 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efccc0a76b0 tx=0x7efcc4004640 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 client.?.objecter ms_handle_connect 0x7efcb4004f50 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc4018de0 con 0x7efcb4004f50 2024-04-19T01:03:39.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: dump: 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.105 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc4024480 con 0x7efcb4004f50 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efcc4024aa0 con 0x7efcb4004f50 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 5 monclient: authenticate success, global_id 144521 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient: _renew_subs 2024-04-19T01:03:39.106 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc8d99640 4 set_mon_vals no callback set 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efccc0a14a0 con 0x7efcb4004f50 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.087+0000 7efcc2ffd640 10 monclient: dump: 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.107 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 10 client.144521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 10 monclient: _renew_subs 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.108 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7efccc123260 con 0x7efcb4004f50 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 1 librados: init done 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7efcc4015040 con 0x7efcb4004f50 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efccc121b30 con 0x7efcb4004f50 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efcb003bb70 0x7efcb003e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(705..705 src has 501..705) ==== 5937+0+0 (secure 0 0 0) 0x7efcc404ada0 con 0x7efcb4004f50 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 10 client.144521.objecter ms_dispatch 0x7efccc000ba0 osd_map(705..705 src has 501..705) 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 3 client.144521.objecter handle_osd_map got epochs [705,705] > 0 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 3 client.144521.objecter handle_osd_map decoding full epoch 705 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 20 client.144521.objecter dump_active .. 0 homeless 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcca59c640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efcb003bb70 0x7efcb003e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.109 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcca59c640 10 monclient: get_auth_request con 0x7efcb003bb70 auth_method 0 2024-04-19T01:03:39.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcca59c640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efcb003bb70 0x7efcb003e030 secure :-1 s=READY pgs=3649 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c6d0 tx=0x7efcbc0b1040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.091+0000 7efcc2ffd640 10 client.144521.objecter ms_handle_connect 0x7efcb003bb70 2024-04-19T01:03:39.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.099+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efcc4010040 con 0x7efcb4004f50 2024-04-19T01:03:39.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.099+0000 7efcc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.110 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.099+0000 7efcc2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcd0f63640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:03:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcd0f63640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:03:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcd0f63640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.329 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efccc0693b0 con 0x7efcb4004f50 2024-04-19T01:03:39.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcc2ffd640 1 -- 172.21.15.43:0/1496761570 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3169 (secure 0 0 0) 0x7efcc4026d30 con 0x7efcb4004f50 2024-04-19T01:03:39.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:03:39.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.327+0000 7efcc2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:39.330 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":126,"quorum":[1,2,3,5,8],"quorum_names":["f","b","g","h","e"],"quorum_leader_name":"g","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 10 librados: watch_flush enter 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 10 librados: watch_flush exit 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 20 client.144521.objecter shutdown clearing up homeless session... 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 10 client.144521.objecter successfully canceled tick 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efcb003bb70 msgr2=0x7efcb003e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efcb003bb70 0x7efcb003e030 secure :-1 s=READY pgs=3649 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c6d0 tx=0x7efcbc0b1040 comp rx=0 tx=0).stop 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 10 monclient: shutdown 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 msgr2=0x7efccc09f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 0x7efccc09f820 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7efccc0a76b0 tx=0x7efcc4004640 comp rx=0 tx=0).stop 2024-04-19T01:03:39.333 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 shutdown_connections 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efcb003bb70 0x7efcb003e030 unknown :-1 s=CLOSED pgs=3649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efcb4004f50 0x7efccc09f820 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efcb4004210 0x7efccc131860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 --2- 172.21.15.43:0/1496761570 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efcb0006800 0x7efccc1d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 >> 172.21.15.43:0/1496761570 conn(0x7efccc0ab2f0 msgr2=0x7efccc0a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 shutdown_connections 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 -- 172.21.15.43:0/1496761570 wait complete. 2024-04-19T01:03:39.334 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.331+0000 7efcd0f63640 1 librados: shutdown 2024-04-19T01:03:39.350 INFO:tasks.mon_thrash.ceph_manager:quorum is size 5 2024-04-19T01:03:39.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon b 2024-04-19T01:03:39.350 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.463+0000 7f24ce693640 10 monclient: get_monmap_and_config 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.463+0000 7f24ce693640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.463+0000 7f24ce693640 10 monclient: monmap: 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:39.470061+0000 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:39.470061+0000 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.467 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:39.468 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 Processor -- start 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 -- start start 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient: init 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient: _add_conns ranks=[3,6,1,4,7,2,10,5,0,8,9] 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c80a51b0 0x7f24c809e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): picked mon.noname-g con 0x7f24c80a51b0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c809eba0 0x7f24c80a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): picked mon.noname-d con 0x7f24c809eba0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f24c80a3ef0 addr v1:172.21.15.161:6789/0 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24c809ef80 con 0x7f24c80a3ef0 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f24c80a5590 con 0x7f24c80a51b0 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f24c80a1c10 con 0x7f24c809eba0 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ce693640 10 monclient(hunting): authenticate will time out at 2089.597270s 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c77fe640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c809eba0 msgr2=0x7f24c80a39b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c809eba0 0x7f24c80a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ccc30640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f24c80a3ef0 0x7f24c813db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.43:48100/0 (socket says 172.21.15.43:48100) 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24ccc30640 1 -- 172.21.15.43:0/1042785934 learned_addr learned my addr 172.21.15.43:0/1042785934 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c80a51b0 0x7f24c809e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): get_auth_request con 0x7f24c80a51b0 auth_method 0 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient(hunting): handle_auth_done global_id 164172 payload 274 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c6ffd640 1 -- 172.21.15.43:0/1042785934 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253140642 0 0) 0x7f24c809ef80 con 0x7f24c80a3ef0 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 1 -- 172.21.15.43:0/1042785934 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c809eba0 msgr2=0x7f24c80a39b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 1 --2- 172.21.15.43:0/1042785934 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c809eba0 0x7f24c80a39b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 1 -- 172.21.15.43:0/1042785934 >> v1:172.21.15.161:6789/0 conn(0x7f24c80a3ef0 legacy=0x7f24c813db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 1 monclient: found mon.noname-g 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c7fff640 1 -- 172.21.15.43:0/1042785934 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c8137070 con 0x7f24c80a51b0 2024-04-19T01:03:39.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.467+0000 7f24c6ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:03:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 1 --2- 172.21.15.43:0/1042785934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c80a51b0 0x7f24c809e660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f24b00097f0 tx=0x7f24b001c390 comp rx=0 tx=0).ready entity=mon.3 client_cookie=55fb9710990b4d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 1 -- 172.21.15.43:0/1042785934 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24b00b0070 con 0x7f24c80a51b0 2024-04-19T01:03:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T01:03:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: dump: 2024-04-19T01:03:39.480 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.481 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 1 -- 172.21.15.43:0/1042785934 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24b00044a0 con 0x7f24c80a51b0 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 1 -- 172.21.15.43:0/1042785934 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24b0003a90 con 0x7f24c80a51b0 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 5 monclient: authenticate success, global_id 164172 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 4 set_mon_vals no callback set 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c6ffd640 10 monclient: dump: 2024-04-19T01:03:39.482 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.483 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: shutdown 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- 172.21.15.43:0/1042785934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c80a51b0 msgr2=0x7f24c809e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 --2- 172.21.15.43:0/1042785934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c80a51b0 0x7f24c809e660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f24b00097f0 tx=0x7f24b001c390 comp rx=0 tx=0).stop 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- 172.21.15.43:0/1042785934 shutdown_connections 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 --2- 172.21.15.43:0/1042785934 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c809eba0 0x7f24c80a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 --2- 172.21.15.43:0/1042785934 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c80a51b0 0x7f24c809e660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- 172.21.15.43:0/1042785934 >> 172.21.15.43:0/1042785934 conn(0x7f24c80a85a0 msgr2=0x7f24c80a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- 172.21.15.43:0/1042785934 shutdown_connections 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- 172.21.15.43:0/1042785934 wait complete. 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: monmap: 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.484 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.485 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 librados: starting msgr at 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 librados: starting objecter 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 Processor -- start 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- start start 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 librados: setting wanted keys 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 librados: calling monclient init 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: init 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient: _add_conns ranks=[3,6,8,2,0,7,1,5,4] 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c809eba0 0x7f24c812e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): picked mon.noname-g con 0x7f24c809eba0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c80a3ef0 0x7f24c812ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): picked mon.noname-d con 0x7f24c80a3ef0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f24c80a51b0 0x7f24c812f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.486 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): picked mon.noname-e con 0x7f24c80a51b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f24c80a2010 con 0x7f24c809eba0 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f24c8136d60 con 0x7f24c80a3ef0 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f24c8073220 con 0x7f24c80a51b0 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ce693640 10 monclient(hunting): authenticate will time out at 2089.603079s 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c77fe640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c80a3ef0 msgr2=0x7f24c812ee40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c77fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c80a3ef0 0x7f24c812ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f24c80a51b0 0x7f24c812f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f24c80a51b0 0x7f24c812f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:43240/0 (socket says 172.21.15.43:43240) 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 -- 172.21.15.43:0/277686847 learned_addr learned my addr 172.21.15.43:0/277686847 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:39.487 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): get_auth_request con 0x7f24c80a51b0 auth_method 0 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c809eba0 0x7f24c812e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): get_auth_request con 0x7f24c809eba0 auth_method 0 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.488 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient(hunting): handle_auth_done global_id 144530 payload 824 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c80a3ef0 msgr2=0x7f24c812ee40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f24c80a3ef0 0x7f24c812ee40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c809eba0 msgr2=0x7f24c812e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f24c809eba0 0x7f24c812e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 monclient: found mon.noname-e 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c813d2b0 con 0x7f24c80a51b0 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24ccc30640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f24c80a51b0 0x7f24c812f380 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c7d0 tx=0x7f24c000cc90 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c4ff9640 10 client.?.objecter ms_handle_connect 0x7f24c80a51b0 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.471+0000 7f24c4ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24c0018070 con 0x7f24c80a51b0 2024-04-19T01:03:39.489 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: dump: 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.490 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c0004d80 con 0x7f24c80a51b0 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24c00070d0 con 0x7f24c80a51b0 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: dump: 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.491 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c67fc640 4 set_mon_vals no callback set 2024-04-19T01:03:39.492 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 5 monclient: authenticate success, global_id 144530 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 monclient: _renew_subs 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c813ea80 con 0x7f24c80a51b0 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 client.144530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 monclient: _renew_subs 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c806bb00 con 0x7f24c80a51b0 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 1 librados: init done 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 0x7f24c812d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f24c80735c0 con 0x7f24c813dd70 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 0x7f24c812d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.493 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f24c0007270 con 0x7f24c80a51b0 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient: get_auth_request con 0x7f24c813dd70 auth_method 0 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f24a8038e10 0x7f24a803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 10 monclient(hunting): handle_auth_done global_id 144506 payload 824 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(705..705 src has 501..705) ==== 5937+0+0 (secure 0 0 0) 0x7f24c0048780 con 0x7f24c80a51b0 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 client.144530.objecter ms_dispatch 0x7f24c8000ba0 osd_map(705..705 src has 501..705) 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 3 client.144530.objecter handle_osd_map got epochs [705,705] > 0 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 3 client.144530.objecter handle_osd_map decoding full epoch 705 2024-04-19T01:03:39.494 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24c8130620 con 0x7f24c813dd70 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 20 client.144530.objecter dump_active .. 0 homeless 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 client.144530.objecter ms_handle_connect 0x7f24c813dd70 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c7fff640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 0x7f24c812d6e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24b00ad000 tx=0x7f24b0019130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 client.144530.objecter resend_mon_ops 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c77fe640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f24a8038e10 0x7f24a803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c77fe640 10 monclient: get_auth_request con 0x7f24a8038e10 auth_method 0 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24b00b0040 con 0x7f24c813dd70 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: dump: 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.495 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.496 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f24b00036e0 con 0x7f24c813dd70 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 msgr2=0x7f24c812d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 0x7f24c812d6e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24b00ad000 tx=0x7f24b0019130 comp rx=0 tx=0).stop 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 msgr2=0x7f24c812d6e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c813dd70 0x7f24c812d6e0 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24b00ad000 tx=0x7f24b0019130 comp rx=0 tx=0).stop 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c4ff9640 10 client.144530.objecter ms_handle_connect 0x7f24a8038e10 2024-04-19T01:03:39.497 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.475+0000 7f24c77fe640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f24a8038e10 0x7f24a803b2d0 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f24c81309a0 tx=0x7f24b8006cc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:39.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 0x7f24c812e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.499 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f24c812e570 con 0x7f24c812dc20 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 0x7f24c812e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient: get_auth_request con 0x7f24c812dc20 auth_method 0 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 10 monclient(hunting): handle_auth_done global_id 144515 payload 824 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 1 -- 172.21.15.43:0/277686847 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f24c80766e0 con 0x7f24c812dc20 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c7fff640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 0x7f24c812e030 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24b001cc30 tx=0x7f24b00a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 client.144530.objecter ms_handle_connect 0x7f24c812dc20 2024-04-19T01:03:39.500 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 client.144530.objecter resend_mon_ops 2024-04-19T01:03:39.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f24b00b0040 con 0x7f24c812dc20 2024-04-19T01:03:39.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:39.501 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 monclient: dump: 2024-04-19T01:03:39.501 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.501 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.502 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9570 (secure 0 0 0) 0x7f24b00ab440 con 0x7f24c812dc20 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 msgr2=0x7f24c812e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 0x7f24c812e030 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24b001cc30 tx=0x7f24b00a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 msgr2=0x7f24c812e030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24c4ff9640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f24c812dc20 0x7f24c812e030 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f24b001cc30 tx=0x7f24b00a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:39.503 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 126, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 184113, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 6, 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:39.504 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:39.505 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:39.506 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:39.507 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:39.508 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.509 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.510 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:39.511 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.512 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.513 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.514 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.515 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.516 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:39.517 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:39.518 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.519 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.520 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.521 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.522 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.523 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.524 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.525 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.526 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:39.526 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 10 librados: watch_flush enter 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 10 librados: watch_flush exit 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 20 client.144530.objecter shutdown clearing up homeless session... 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 10 client.144530.objecter successfully canceled tick 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f24a8038e10 msgr2=0x7f24a803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f24a8038e10 0x7f24a803b2d0 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f24c81309a0 tx=0x7f24b8006cc0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 10 monclient: shutdown 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f24c80a51b0 msgr2=0x7f24c812f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 --2- 172.21.15.43:0/277686847 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f24c80a51b0 0x7f24c812f380 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c7d0 tx=0x7f24c000cc90 comp rx=0 tx=0).stop 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ccc30640 1 -- 172.21.15.43:0/277686847 reap_dead start 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ccc30640 1 -- 172.21.15.43:0/277686847 reap_dead start 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 shutdown_connections 2024-04-19T01:03:39.528 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 >> 172.21.15.43:0/277686847 conn(0x7f24c80a85a0 msgr2=0x7f24c80a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 shutdown_connections 2024-04-19T01:03:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 -- 172.21.15.43:0/277686847 wait complete. 2024-04-19T01:03:39.529 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.483+0000 7f24ce693640 1 librados: shutdown 2024-04-19T01:03:39.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T01:03:39.529 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon e 2024-04-19T01:03:39.529 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:03:39.656 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient: get_monmap_and_config 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient: monmap: 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:39.659765+0000 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:39.659765+0000 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.657 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.658 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.658 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.658 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.658 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:39.658 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:39.658 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 Processor -- start 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 -- start start 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient: init 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient: _add_conns ranks=[5,10,2,6,3,1,8,7,4,0,9] 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fde500a51b0 0x7fde5009e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): picked mon.noname-h con 0x7fde500a51b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fde5009eba0 0x7fde500a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.660 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): picked mon.noname-f con 0x7fde5009eba0 addr v2:172.21.15.161:3300/0 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 0x7fde500a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): picked mon.noname-b con 0x7fde500a3ef0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fde500a5590 con 0x7fde5009eba0 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fde5009ef80 con 0x7fde500a3ef0 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fde500a1c10 con 0x7fde500a51b0 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde55772640 10 monclient(hunting): authenticate will time out at 2089.787079s 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 0x7fde500a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 0x7fde500a43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:39720/0 (socket says 172.21.15.43:39720) 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 -- 172.21.15.43:0/1836783544 learned_addr learned my addr 172.21.15.43:0/1836783544 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): get_auth_request con 0x7fde500a3ef0 auth_method 0 2024-04-19T01:03:39.661 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4effd640 1 --2- 172.21.15.43:0/1836783544 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fde500a51b0 0x7fde5009e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_done global_id 144524 payload 274 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 -- 172.21.15.43:0/1836783544 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fde500a51b0 msgr2=0x7fde5009e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 --2- 172.21.15.43:0/1836783544 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fde500a51b0 0x7fde5009e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 -- 172.21.15.43:0/1836783544 >> v2:172.21.15.161:3300/0 conn(0x7fde5009eba0 msgr2=0x7fde500a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 --2- 172.21.15.43:0/1836783544 >> v2:172.21.15.161:3300/0 conn(0x7fde5009eba0 0x7fde500a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 monclient: found mon.noname-b 2024-04-19T01:03:39.662 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:39.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.655+0000 7fde4f7fe640 1 -- 172.21.15.43:0/1836783544 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde50137070 con 0x7fde500a3ef0 2024-04-19T01:03:39.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4f7fe640 1 --2- 172.21.15.43:0/1836783544 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 0x7fde500a43a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde48009e80 tx=0x7fde4801c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2232b1195347a29a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.669 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 1 -- 172.21.15.43:0/1836783544 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde480090d0 con 0x7fde500a3ef0 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: dump: 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.670 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 1 -- 172.21.15.43:0/1836783544 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde480a5b50 con 0x7fde500a3ef0 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 1 -- 172.21.15.43:0/1836783544 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde480a5d40 con 0x7fde500a3ef0 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.671 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde4dffb640 10 monclient: dump: 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.672 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 5 monclient: authenticate success, global_id 144524 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 4 set_mon_vals no callback set 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 10 monclient: shutdown 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 -- 172.21.15.43:0/1836783544 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 msgr2=0x7fde500a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 --2- 172.21.15.43:0/1836783544 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 0x7fde500a43a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde48009e80 tx=0x7fde4801c8a0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 -- 172.21.15.43:0/1836783544 shutdown_connections 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 --2- 172.21.15.43:0/1836783544 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fde500a3ef0 0x7fde500a43a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 --2- 172.21.15.43:0/1836783544 >> v2:172.21.15.161:3300/0 conn(0x7fde5009eba0 0x7fde500a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 --2- 172.21.15.43:0/1836783544 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fde500a51b0 0x7fde5009e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.673 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 -- 172.21.15.43:0/1836783544 >> 172.21.15.43:0/1836783544 conn(0x7fde500a85a0 msgr2=0x7fde500a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 -- 172.21.15.43:0/1836783544 shutdown_connections 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 -- 172.21.15.43:0/1836783544 wait complete. 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 10 monclient: monmap: 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.674 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 librados: starting msgr at 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 librados: starting objecter 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 Processor -- start 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 -- start start 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 librados: setting wanted keys 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 1 librados: calling monclient init 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.659+0000 7fde55772640 10 monclient: init 2024-04-19T01:03:39.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient: _add_conns ranks=[1,6,8,3,2,4,0,7,5] 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde5009eba0 0x7fde5013cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): picked mon.noname-f con 0x7fde5009eba0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fde500a3ef0 0x7fde50135cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): picked mon.noname-d con 0x7fde500a3ef0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde500a51b0 0x7fde501d0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): picked mon.noname-e con 0x7fde500a51b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fde50073220 con 0x7fde5009eba0 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fde500a2010 con 0x7fde500a3ef0 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fde500733a0 con 0x7fde500a51b0 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): authenticate will time out at 2089.792291s 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4e7fc640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fde500a3ef0 msgr2=0x7fde50135cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4e7fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fde500a3ef0 0x7fde50135cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde500a51b0 0x7fde501d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde500a51b0 0x7fde501d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:43256/0 (socket says 172.21.15.43:43256) 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 -- 172.21.15.43:0/547600707 learned_addr learned my addr 172.21.15.43:0/547600707 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): get_auth_request con 0x7fde500a51b0 auth_method 0 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde5009eba0 0x7fde5013cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.677 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient(hunting): handle_auth_done global_id 144539 payload 824 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fde500a3ef0 msgr2=0x7fde50135cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fde500a3ef0 0x7fde50135cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde5009eba0 msgr2=0x7fde5013cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde5009eba0 0x7fde5013cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 monclient: found mon.noname-e 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5013d690 con 0x7fde500a51b0 2024-04-19T01:03:39.678 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient: get_auth_request con 0x7fde5009eba0 auth_method 0 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fde5009eba0 0x7fde5013cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4f7fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde500a51b0 0x7fde501d0960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fde4801c6d0 tx=0x7fde48019640 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 client.?.objecter ms_handle_connect 0x7fde500a51b0 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde480bc020 con 0x7fde500a51b0 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: dump: 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.679 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde48003e10 con 0x7fde500a51b0 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde480044b0 con 0x7fde500a51b0 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4d7fa640 4 set_mon_vals no callback set 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: dump: 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.681 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 5 monclient: authenticate success, global_id 144539 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient: _renew_subs 2024-04-19T01:03:39.682 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde50122de0 con 0x7fde500a51b0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 client.144539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient: _renew_subs 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fde500a06b0 con 0x7fde500a51b0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 librados: init done 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 0x7fde5002c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fde5006d140 con 0x7fde50068240 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fde480bf020 con 0x7fde500a51b0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 0x7fde5002c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient: get_auth_request con 0x7fde50068240 auth_method 0 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.683 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde30038e10 0x7fde3003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 10 monclient(hunting): handle_auth_done global_id 144548 payload 824 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde500a71c0 con 0x7fde50068240 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(705..705 src has 501..705) ==== 5937+0+0 (secure 0 0 0) 0x7fde480e4310 con 0x7fde500a51b0 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 client.144539.objecter ms_dispatch 0x7fde50000ba0 osd_map(705..705 src has 501..705) 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 3 client.144539.objecter handle_osd_map got epochs [705,705] > 0 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 3 client.144539.objecter handle_osd_map decoding full epoch 705 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 20 client.144539.objecter dump_active .. 0 homeless 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4effd640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 0x7fde5002c450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde3800d790 tx=0x7fde3800dc50 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.684 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 client.144539.objecter ms_handle_connect 0x7fde50068240 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 client.144539.objecter resend_mon_ops 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde38016070 con 0x7fde50068240 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: dump: 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.685 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4e7fc640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde30038e10 0x7fde3003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4e7fc640 10 monclient: get_auth_request con 0x7fde30038e10 auth_method 0 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fde38008700 con 0x7fde50068240 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 msgr2=0x7fde5002c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 0x7fde5002c450 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fde3800d790 tx=0x7fde3800dc50 comp rx=0 tx=0).stop 2024-04-19T01:03:39.686 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 msgr2=0x7fde5002c450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:39.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde50068240 0x7fde5002c450 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde477fe640 10 client.144539.objecter ms_handle_connect 0x7fde30038e10 2024-04-19T01:03:39.687 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.663+0000 7fde4e7fc640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde30038e10 0x7fde3003b2d0 secure :-1 s=READY pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7fde40007fd0 tx=0x7fde40007ac0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde55772640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde55772640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 0x7fde5002cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde55772640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fde50139050 con 0x7fde5002c990 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 0x7fde5002cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient: get_auth_request con 0x7fde5002c990 auth_method 0 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 10 monclient(hunting): handle_auth_done global_id 144557 payload 824 2024-04-19T01:03:39.689 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 1 -- 172.21.15.43:0/547600707 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde50136840 con 0x7fde5002c990 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde4effd640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 0x7fde5002cda0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde38000c10 tx=0x7fde38004740 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde477fe640 10 client.144539.objecter ms_handle_connect 0x7fde5002c990 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde477fe640 10 client.144539.objecter resend_mon_ops 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fde38012e30 con 0x7fde5002c990 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.671+0000 7fde477fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 10 monclient: dump: 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.690 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9230 (secure 0 0 0) 0x7fde38012e30 con 0x7fde5002c990 2024-04-19T01:03:39.691 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 msgr2=0x7fde5002cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 0x7fde5002cda0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde38000c10 tx=0x7fde38004740 comp rx=0 tx=0).stop 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 msgr2=0x7fde5002cda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde477fe640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde5002c990 0x7fde5002cda0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fde38000c10 tx=0x7fde38004740 comp rx=0 tx=0).stop 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:39.692 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 126, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 184305, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 6, 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:39.693 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.694 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.695 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:39.696 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.697 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.698 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:39.699 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.700 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:39.701 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.702 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.703 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.704 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.705 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.706 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:39.707 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.708 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:39.709 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.710 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:39.711 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.712 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 3 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:39.713 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 10 librados: watch_flush enter 2024-04-19T01:03:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 10 librados: watch_flush exit 2024-04-19T01:03:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 20 client.144539.objecter shutdown clearing up homeless session... 2024-04-19T01:03:39.715 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 10 client.144539.objecter successfully canceled tick 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde30038e10 msgr2=0x7fde3003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fde30038e10 0x7fde3003b2d0 secure :-1 s=READY pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7fde40007fd0 tx=0x7fde40007ac0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 10 monclient: shutdown 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde500a51b0 msgr2=0x7fde501d0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 --2- 172.21.15.43:0/547600707 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fde500a51b0 0x7fde501d0960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fde4801c6d0 tx=0x7fde48019640 comp rx=0 tx=0).stop 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde4f7fe640 1 -- 172.21.15.43:0/547600707 reap_dead start 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde4f7fe640 1 -- 172.21.15.43:0/547600707 reap_dead start 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 shutdown_connections 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 >> 172.21.15.43:0/547600707 conn(0x7fde500a85a0 msgr2=0x7fde500a76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 shutdown_connections 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 -- 172.21.15.43:0/547600707 wait complete. 2024-04-19T01:03:39.716 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.675+0000 7fde55772640 1 librados: shutdown 2024-04-19T01:03:39.717 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T01:03:39.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon f 2024-04-19T01:03:39.717 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:03:39.839 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient: get_monmap_and_config 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient: monmap: 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:39.842694+0000 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:39.842694+0000 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.840 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:39.841 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 Processor -- start 2024-04-19T01:03:39.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 -- start start 2024-04-19T01:03:39.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient: init 2024-04-19T01:03:39.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient: _add_conns ranks=[3,6,0,7,4,10,1,5,2,8,9] 2024-04-19T01:03:39.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 0x7fa078138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): picked mon.noname-g con 0x7fa078138a50 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa0780a5bb0 0x7fa078139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): picked mon.noname-d con 0x7fa0780a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa078137960 addr v1:172.21.15.43:6789/0 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0780a0d40 con 0x7fa078137960 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa0780a1020 con 0x7fa078138a50 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa0781398b0 con 0x7fa0780a5bb0 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07cf63640 10 monclient(hunting): authenticate will time out at 2089.970018s 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa075d9b640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa0780a5bb0 msgr2=0x7fa078139370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa075d9b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa0780a5bb0 0x7fa078139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa076d9d640 1 -- >> v1:172.21.15.43:6789/0 conn(0x7fa078137960 legacy=0x7fa078137dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.43:6789/0 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 0x7fa078138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 0x7fa078138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3301/0 says I am v2:172.21.15.43:40994/0 (socket says 172.21.15.43:40994) 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 -- 172.21.15.43:0/2920844252 learned_addr learned my addr 172.21.15.43:0/2920844252 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): get_auth_request con 0x7fa078138a50 auth_method 0 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient(hunting): handle_auth_done global_id 164190 payload 274 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 -- 172.21.15.43:0/2920844252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa0780a5bb0 msgr2=0x7fa078139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 --2- 172.21.15.43:0/2920844252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa0780a5bb0 0x7fa078139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 -- 172.21.15.43:0/2920844252 >> v1:172.21.15.43:6789/0 conn(0x7fa078137960 legacy=0x7fa078137dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 monclient: found mon.noname-g 2024-04-19T01:03:39.846 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 10 monclient: _send_mon_message to mon.noname-g at v2:172.21.15.161:3301/0 2024-04-19T01:03:39.847 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.839+0000 7fa07659c640 1 -- 172.21.15.43:0/2920844252 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0780a0a60 con 0x7fa078138a50 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07659c640 1 --2- 172.21.15.43:0/2920844252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 0x7fa078138e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0600098a0 tx=0x7fa06001c400 comp rx=0 tx=0).ready entity=mon.3 client_cookie=433f92fcf899c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 1 -- 172.21.15.43:0/2920844252 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0600b0070 con 0x7fa078138a50 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: got monmap 1 from mon.noname-g (according to old e1) 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: dump: 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.854 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 1 -- 172.21.15.43:0/2920844252 <== mon.3 v2:172.21.15.161:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0600aa810 con 0x7fa078138a50 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 1 -- 172.21.15.43:0/2920844252 <== mon.3 v2:172.21.15.161:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0600a9300 con 0x7fa078138a50 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07559a640 10 monclient: dump: 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.856 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 5 monclient: authenticate success, global_id 164190 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 4 set_mon_vals no callback set 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: shutdown 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- 172.21.15.43:0/2920844252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 msgr2=0x7fa078138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 --2- 172.21.15.43:0/2920844252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 0x7fa078138e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0600098a0 tx=0x7fa06001c400 comp rx=0 tx=0).stop 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- 172.21.15.43:0/2920844252 shutdown_connections 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 --2- 172.21.15.43:0/2920844252 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa0780a5bb0 0x7fa078139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 --2- 172.21.15.43:0/2920844252 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa078138a50 0x7fa078138e30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- 172.21.15.43:0/2920844252 >> 172.21.15.43:0/2920844252 conn(0x7fa0780a7cb0 msgr2=0x7fa0780a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- 172.21.15.43:0/2920844252 shutdown_connections 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- 172.21.15.43:0/2920844252 wait complete. 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: build_initial_monmap 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: monmap: 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:39.858 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 librados: starting msgr at 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 librados: starting objecter 2024-04-19T01:03:39.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 Processor -- start 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- start start 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 librados: setting wanted keys 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 librados: calling monclient init 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: init 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient: _add_conns ranks=[5,7,8,1,3,2,4,6,0] 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0780a5bb0 0x7fa0780a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient(hunting): picked mon.noname-h con 0x7fa0780a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa078137960 0x7fa0780a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient(hunting): picked mon.noname-i con 0x7fa078137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa078138a50 0x7fa0780a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient(hunting): picked mon.noname-e con 0x7fa078138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa078143bb0 con 0x7fa0780a5bb0 2024-04-19T01:03:39.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fa078143a30 con 0x7fa078137960 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.843+0000 7fa07cf63640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa078143d30 con 0x7fa078138a50 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient(hunting): authenticate will time out at 2089.976021s 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa078138a50 0x7fa0780a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa075d9b640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa078137960 msgr2=0x7fa0780a2af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa075d9b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa078137960 0x7fa0780a2af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa078138a50 0x7fa0780a3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:43292/0 (socket says 172.21.15.43:43292) 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 -- 172.21.15.43:0/1771101868 learned_addr learned my addr 172.21.15.43:0/1771101868 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): get_auth_request con 0x7fa078138a50 auth_method 0 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0780a5bb0 0x7fa0780a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): get_auth_request con 0x7fa0780a5bb0 auth_method 0 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient(hunting): handle_auth_done global_id 144566 payload 824 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa078137960 msgr2=0x7fa0780a2af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fa078137960 0x7fa0780a2af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0780a5bb0 msgr2=0x7fa0780a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa0780a5bb0 0x7fa0780a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:39.862 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 monclient: found mon.noname-e 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0780a4b30 con 0x7fa078138a50 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa076d9d640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa078138a50 0x7fa0780a3030 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa07000be90 tx=0x7fa07000bf80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 client.?.objecter ms_handle_connect 0x7fa078138a50 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa07000cd20 con 0x7fa078138a50 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: dump: 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.863 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: _finish_auth 0 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa070010030 con 0x7fa078138a50 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa070015cc0 con 0x7fa078138a50 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 monclient: dump: 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.865 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa074d99640 4 set_mon_vals no callback set 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 5 monclient: authenticate success, global_id 144566 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient: _renew_subs 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.866 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0780a4e10 con 0x7fa078138a50 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 client.144566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient: _renew_subs 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa07809dfe0 con 0x7fa078138a50 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 1 librados: init done 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 0x7fa07812cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0780a3cd0 con 0x7fa07812c8b0 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa0700153e0 con 0x7fa078138a50 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa058038e10 0x7fa05803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(706..706 src has 501..706) ==== 5607+0+0 (secure 0 0 0) 0x7fa0700496d0 con 0x7fa078138a50 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 0x7fa07812cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 client.144566.objecter ms_dispatch 0x7fa078000ba0 osd_map(706..706 src has 501..706) 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 3 client.144566.objecter handle_osd_map got epochs [706,706] > 0 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 3 client.144566.objecter handle_osd_map decoding full epoch 706 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient: get_auth_request con 0x7fa07812c8b0 auth_method 0 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 20 client.144566.objecter dump_active .. 0 homeless 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa075d9b640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa058038e10 0x7fa05803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa075d9b640 10 monclient: get_auth_request con 0x7fa058038e10 auth_method 0 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 10 monclient(hunting): handle_auth_done global_id 144541 payload 824 2024-04-19T01:03:39.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa07809eef0 con 0x7fa07812c8b0 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa075d9b640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa058038e10 0x7fa05803b2d0 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7fa07809e3c0 tx=0x7fa068006d10 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 client.144566.objecter ms_handle_connect 0x7fa058038e10 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 client.144566.objecter ms_handle_connect 0x7fa07812c8b0 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa06effd640 10 client.144566.objecter resend_mon_ops 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.847+0000 7fa07659c640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 0x7fa07812cc90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0780a7500 tx=0x7fa060019200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0600b9070 con 0x7fa07812c8b0 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 10 monclient: dump: 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.869 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa0600036e0 con 0x7fa07812c8b0 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:39.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:39.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 msgr2=0x7fa07812cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 0x7fa07812cc90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0780a7500 tx=0x7fa060019200 comp rx=0 tx=0).stop 2024-04-19T01:03:39.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 msgr2=0x7fa07812cc90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:39.871 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.851+0000 7fa06effd640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812c8b0 0x7fa07812cc90 secure :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa0780a7500 tx=0x7fa060019200 comp rx=0 tx=0).stop 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07cf63640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07cf63640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 0x7fa07812dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07cf63640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa07806d140 con 0x7fa07812d1d0 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 0x7fa07812dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient: get_auth_request con 0x7fa07812d1d0 auth_method 0 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:39.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.855+0000 7fa07659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07659c640 10 monclient(hunting): handle_auth_done global_id 144550 payload 824 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07659c640 1 -- 172.21.15.43:0/1771101868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa0780a5440 con 0x7fa07812d1d0 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07659c640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 0x7fa07812dda0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa06001cca0 tx=0x7fa0600a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 client.144566.objecter ms_handle_connect 0x7fa07812d1d0 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 client.144566.objecter resend_mon_ops 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa0600b0040 con 0x7fa07812d1d0 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 monclient: dump: 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:39.874 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9230 (secure 0 0 0) 0x7fa060000e50 con 0x7fa07812d1d0 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:39.875 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 msgr2=0x7fa07812dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 0x7fa07812dda0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa06001cca0 tx=0x7fa0600a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 msgr2=0x7fa07812dda0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:39.876 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa06effd640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa07812d1d0 0x7fa07812dda0 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa06001cca0 tx=0x7fa0600a8000 comp rx=0 tx=0).stop 2024-04-19T01:03:39.876 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:39.876 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:39.876 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 126, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 184491, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 7, 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:39.877 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:39.878 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.879 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:39.880 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:39.881 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:39.882 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.886 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:39.888 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.893 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.894 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:39.895 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.896 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.897 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:39.898 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 10 librados: watch_flush enter 2024-04-19T01:03:39.900 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 10 librados: watch_flush exit 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 20 client.144566.objecter shutdown clearing up homeless session... 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 10 client.144566.objecter successfully canceled tick 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa058038e10 msgr2=0x7fa05803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa058038e10 0x7fa05803b2d0 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7fa07809e3c0 tx=0x7fa068006d10 comp rx=0 tx=0).stop 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 10 monclient: shutdown 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa078138a50 msgr2=0x7fa0780a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 --2- 172.21.15.43:0/1771101868 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa078138a50 0x7fa0780a3030 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa07000be90 tx=0x7fa07000bf80 comp rx=0 tx=0).stop 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa076d9d640 1 -- 172.21.15.43:0/1771101868 reap_dead start 2024-04-19T01:03:39.901 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa076d9d640 1 -- 172.21.15.43:0/1771101868 reap_dead start 2024-04-19T01:03:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 shutdown_connections 2024-04-19T01:03:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 >> 172.21.15.43:0/1771101868 conn(0x7fa0780a7cb0 msgr2=0x7fa0780a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 shutdown_connections 2024-04-19T01:03:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 -- 172.21.15.43:0/1771101868 wait complete. 2024-04-19T01:03:39.902 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:39.859+0000 7fa07cf63640 1 librados: shutdown 2024-04-19T01:03:39.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T01:03:39.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon g 2024-04-19T01:03:39.902 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T01:03:40.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient: get_monmap_and_config 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient: build_initial_monmap 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient: monmap: 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:40.022994+0000 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:40.022994+0000 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:40.020 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:40.021 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 Processor -- start 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 -- start start 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient: init 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient: _add_conns ranks=[6,2,9,7,0,5,10,1,3,8,4] 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2200138cd0 0x7f22001390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): picked mon.noname-d con 0x7f2200138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 0x7f22001395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): picked mon.noname-b con 0x7f22000a5b80 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f22001358b0 0x7f2200135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): picked mon.noname-a con 0x7f22001358b0 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f22000a4410 con 0x7f22001358b0 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f2200139b30 con 0x7f22000a5b80 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f22000a3dc0 con 0x7f2200138cd0 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f220495e640 10 monclient(hunting): authenticate will time out at 2090.150318s 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fed9d640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7f22001358b0 msgr2=0x7f2200135d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fe59c640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2200138cd0 msgr2=0x7f22001390b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:40.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fed9d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f22001358b0 0x7f2200135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fe59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2200138cd0 0x7f22001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 0x7f22001395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 0x7f22001395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:39734/0 (socket says 172.21.15.43:39734) 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 -- 172.21.15.43:0/1092342463 learned_addr learned my addr 172.21.15.43:0/1092342463 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): get_auth_request con 0x7f22000a5b80 auth_method 0 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient(hunting): handle_auth_done global_id 144533 payload 274 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 -- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2200138cd0 msgr2=0x7f22001390b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:40.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2200138cd0 0x7f22001390b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 -- 172.21.15.43:0/1092342463 >> v2:172.21.15.43:3300/0 conn(0x7f22001358b0 msgr2=0x7f2200135d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1092342463 >> v2:172.21.15.43:3300/0 conn(0x7f22001358b0 0x7f2200135d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient: _finish_hunting 0 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 monclient: found mon.noname-b 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.019+0000 7f21fdd9b640 1 -- 172.21.15.43:0/1092342463 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22000a40a0 con 0x7f22000a5b80 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 0x7f22001395f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009870 tx=0x7f21f401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b050fcfee8e260c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 1 -- 172.21.15.43:0/1092342463 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21f40b1070 con 0x7f22000a5b80 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: dump: 2024-04-19T01:03:40.028 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.029 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: _finish_auth 0 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: _check_auth_tickets 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 1 -- 172.21.15.43:0/1092342463 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f40ab810 con 0x7f22000a5b80 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 5 monclient: authenticate success, global_id 144533 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:40.030 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 1 -- 172.21.15.43:0/1092342463 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21f40aa300 con 0x7f22000a5b80 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 4 set_mon_vals no callback set 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fd59a640 10 monclient: dump: 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.031 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: shutdown 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 msgr2=0x7f22001395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 0x7f22001395f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009870 tx=0x7f21f401c400 comp rx=0 tx=0).stop 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- 172.21.15.43:0/1092342463 shutdown_connections 2024-04-19T01:03:40.032 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- 172.21.15.43:0/1092342463 >> v2:172.21.15.43:3300/0 conn(0x7f22001358b0 0x7f2200135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f22000a5b80 0x7f22001395f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- 172.21.15.43:0/1092342463 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f2200138cd0 0x7f22001390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- 172.21.15.43:0/1092342463 >> 172.21.15.43:0/1092342463 conn(0x7f22000ab2f0 msgr2=0x7f22000ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:40.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- 172.21.15.43:0/1092342463 shutdown_connections 2024-04-19T01:03:40.033 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- 172.21.15.43:0/1092342463 wait complete. 2024-04-19T01:03:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: build_initial_monmap 2024-04-19T01:03:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: monmap: 2024-04-19T01:03:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:40.041 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:40.042 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 librados: starting msgr at 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 librados: starting objecter 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 Processor -- start 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- start start 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 librados: setting wanted keys 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 librados: calling monclient init 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: init 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient: _add_conns ranks=[6,1,8,7,5,3,4,2,0] 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f22000a5b80 0x7f22001d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): picked mon.noname-d con 0x7f22000a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22001358b0 0x7f22001d1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): picked mon.noname-f con 0x7f22001358b0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:03:40.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 0x7f22001d16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): picked mon.noname-e con 0x7f2200138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22000a2010 con 0x7f22001358b0 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f22000a1e90 con 0x7f22000a5b80 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f2200142aa0 con 0x7f2200138cd0 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f220495e640 10 monclient(hunting): authenticate will time out at 2090.155521s 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fe59c640 1 -- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f22000a5b80 msgr2=0x7f22001d0c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3303/0 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fe59c640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f22000a5b80 0x7f22001d0c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 0x7f22001d16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 0x7f22001d16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:43308/0 (socket says 172.21.15.43:43308) 2024-04-19T01:03:40.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 -- 172.21.15.43:0/1936994310 learned_addr learned my addr 172.21.15.43:0/1936994310 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): get_auth_request con 0x7f2200138cd0 auth_method 0 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22001358b0 0x7f22001d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient(hunting): handle_auth_done global_id 144575 payload 824 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f22000a5b80 msgr2=0x7f22001d0c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f22000a5b80 0x7f22001d0c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22001358b0 msgr2=0x7f22001d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22001358b0 0x7f22001d1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.023+0000 7f21fed9d640 10 monclient: _finish_hunting 0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fed9d640 1 monclient: found mon.noname-e 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fed9d640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f220013eaa0 con 0x7f2200138cd0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fdd9b640 10 monclient: get_auth_request con 0x7f22001358b0 auth_method 0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22001358b0 0x7f22001d1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fed9d640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 0x7f22001d16c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f21f800be90 tx=0x7f21f800bf80 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 client.?.objecter ms_handle_connect 0x7f2200138cd0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21f800cd20 con 0x7f2200138cd0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: dump: 2024-04-19T01:03:40.046 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.047 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: _finish_auth 0 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: _check_auth_tickets 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f8010030 con 0x7f2200138cd0 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21f8015cc0 con 0x7f2200138cd0 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 5 monclient: authenticate success, global_id 144575 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 monclient: _renew_subs 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22000a3590 con 0x7f2200138cd0 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fcd99640 4 set_mon_vals no callback set 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: dump: 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.049 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 client.144575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 monclient: _renew_subs 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f220013c7b0 con 0x7f2200138cd0 2024-04-19T01:03:40.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 1 librados: init done 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 0x7f22001348a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f220006d140 con 0x7f220013e680 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f21f80153e0 con 0x7f2200138cd0 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f21e0038e10 0x7f21e003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(706..706 src has 501..706) ==== 5607+0+0 (secure 0 0 0) 0x7f21f8049690 con 0x7f2200138cd0 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 client.144575.objecter ms_dispatch 0x7f2200000ba0 osd_map(706..706 src has 501..706) 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 3 client.144575.objecter handle_osd_map got epochs [706,706] > 0 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 3 client.144575.objecter handle_osd_map decoding full epoch 706 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 0x7f22001348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient: get_auth_request con 0x7f220013e680 auth_method 0 2024-04-19T01:03:40.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 20 client.144575.objecter dump_active .. 0 homeless 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f21e0038e10 0x7f21e003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fdd9b640 10 monclient: get_auth_request con 0x7f21e0038e10 auth_method 0 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 10 monclient(hunting): handle_auth_done global_id 164199 payload 824 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22000693b0 con 0x7f220013e680 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 client.144575.objecter ms_handle_connect 0x7f220013e680 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 client.144575.objecter resend_mon_ops 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fe59c640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 0x7f22001348a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038c0 tx=0x7f21f000b810 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 client.144575.objecter ms_handle_connect 0x7f21e0038e10 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21fdd9b640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f21e0038e10 0x7f21e003b2d0 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f21f40ae000 tx=0x7f21f40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21f0016070 con 0x7f220013e680 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.027+0000 7f21eeffd640 10 monclient: dump: 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.053 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f21f0003b00 con 0x7f220013e680 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 msgr2=0x7f22001348a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 0x7f22001348a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038c0 tx=0x7f21f000b810 comp rx=0 tx=0).stop 2024-04-19T01:03:40.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 msgr2=0x7f22001348a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.031+0000 7f21eeffd640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f220013e680 0x7f22001348a0 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038c0 tx=0x7f21f000b810 comp rx=0 tx=0).stop 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f220495e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f220495e640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 0x7f22001351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f220495e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f2200135730 con 0x7f2200134de0 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 0x7f22001351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient: get_auth_request con 0x7f2200134de0 auth_method 0 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 10 monclient(hunting): handle_auth_done global_id 164208 payload 824 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 1 -- 172.21.15.43:0/1936994310 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f220013d060 con 0x7f2200134de0 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21fe59c640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 0x7f22001351f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038c0 tx=0x7f21f0005950 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21eeffd640 10 client.144575.objecter ms_handle_connect 0x7f2200134de0 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.035+0000 7f21eeffd640 10 client.144575.objecter resend_mon_ops 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f21f0016030 con 0x7f2200134de0 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 10 monclient: dump: 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.056 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9401 (secure 0 0 0) 0x7f21f0021410 con 0x7f2200134de0 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:40.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 msgr2=0x7f22001351f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 0x7f22001351f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f21f00038c0 tx=0x7f21f0005950 comp rx=0 tx=0).stop 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 msgr2=0x7f22001351f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21eeffd640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f2200134de0 0x7f22001351f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:03:40.058 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 126, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 51406, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 7, 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:40.059 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:40.060 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:40.061 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:40.062 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:40.063 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.064 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.065 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.066 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.067 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.068 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.068 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:40.068 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:40.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:40.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.072 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.074 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:40.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:40.078 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.079 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:40.080 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:40.081 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:40.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 10 librados: watch_flush enter 2024-04-19T01:03:40.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 10 librados: watch_flush exit 2024-04-19T01:03:40.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 20 client.144575.objecter shutdown clearing up homeless session... 2024-04-19T01:03:40.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 10 client.144575.objecter successfully canceled tick 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f21e0038e10 msgr2=0x7f21e003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f21e0038e10 0x7f21e003b2d0 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f21f40ae000 tx=0x7f21f40a7000 comp rx=0 tx=0).stop 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 10 monclient: shutdown 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 msgr2=0x7f22001d16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f21fed9d640 1 -- 172.21.15.43:0/1936994310 reap_dead start 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 0x7f22001d16c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f21f800be90 tx=0x7f21f800bf80 comp rx=0 tx=0).stop 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 shutdown_connections 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 --2- 172.21.15.43:0/1936994310 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f2200138cd0 0x7f22001d16c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 >> 172.21.15.43:0/1936994310 conn(0x7f22000ab2f0 msgr2=0x7f2200137610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 shutdown_connections 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 -- 172.21.15.43:0/1936994310 wait complete. 2024-04-19T01:03:40.084 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.039+0000 7f220495e640 1 librados: shutdown 2024-04-19T01:03:40.085 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T01:03:40.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 5 for mon h 2024-04-19T01:03:40.085 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient: get_monmap_and_config 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient: build_initial_monmap 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient: monmap: 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:03:40.211094+0000 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:03:40.211094+0000 2024-04-19T01:03:40.208 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:03:40.209 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 Processor -- start 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 -- start start 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient: init 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient: _add_conns ranks=[10,9,7,3,5,6,1,4,0,2,8] 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f10f0138a50 0x7f10f0138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): picked mon.noname-f con 0x7f10f0138a50 addr v2:172.21.15.161:3300/0 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f10f00a5bb0 0x7f10f0139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): picked mon.noname-a con 0x7f10f00a5bb0 addr v2:172.21.15.43:3300/0 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f10f0137960 0x7f10f0137df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): picked mon.noname-i con 0x7f10f0137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f10f00a1020 con 0x7f10f00a5bb0 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f10f01398b0 con 0x7f10f0138a50 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f10f00a0780 con 0x7f10f0137960 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10f5477640 10 monclient(hunting): authenticate will time out at 2090.338418s 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10ee7fc640 1 -- >> v2:172.21.15.43:3300/0 conn(0x7f10f00a5bb0 msgr2=0x7f10f0139370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10ee7fc640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f10f00a5bb0 0x7f10f0139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10ef7fe640 1 -- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f10f0137960 msgr2=0x7f10f0137df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.161:3303/0 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10ef7fe640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f10f0137960 0x7f10f0137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f10f0138a50 0x7f10f0138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f10f0138a50 0x7f10f0138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.43:36290/0 (socket says 172.21.15.43:36290) 2024-04-19T01:03:40.213 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 -- 172.21.15.43:0/2389105064 learned_addr learned my addr 172.21.15.43:0/2389105064 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): get_auth_request con 0x7f10f0138a50 auth_method 0 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient(hunting): handle_auth_done global_id 144559 payload 274 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 -- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f10f0137960 msgr2=0x7f10f0137df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f10f0137960 0x7f10f0137df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 -- 172.21.15.43:0/2389105064 >> v2:172.21.15.43:3300/0 conn(0x7f10f00a5bb0 msgr2=0x7f10f0139370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2389105064 >> v2:172.21.15.43:3300/0 conn(0x7f10f00a5bb0 0x7f10f0139370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient: _finish_hunting 0 2024-04-19T01:03:40.214 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 monclient: found mon.noname-f 2024-04-19T01:03:40.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:40.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 10 monclient: _send_mon_message to mon.noname-f at v2:172.21.15.161:3300/0 2024-04-19T01:03:40.215 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.207+0000 7f10eeffd640 1 -- 172.21.15.43:0/2389105064 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f00a0d40 con 0x7f10f0138a50 2024-04-19T01:03:40.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10f0138a50 0x7f10f0138e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f10f00a72d0 tx=0x7f10e001c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca7ae2a46983113c server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 1 -- 172.21.15.43:0/2389105064 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e00b0070 con 0x7f10f0138a50 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: got monmap 1 from mon.noname-f (according to old e1) 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: dump: 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.223 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: _finish_auth 0 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: _check_auth_tickets 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 1 -- 172.21.15.43:0/2389105064 <== mon.1 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e00aa810 con 0x7f10f0138a50 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 5 monclient: authenticate success, global_id 144559 2024-04-19T01:03:40.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 10 monclient: get_monmap_and_config success 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 4 set_mon_vals no callback set 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 1 -- 172.21.15.43:0/2389105064 <== mon.1 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e00a9300 con 0x7f10f0138a50 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10edffb640 10 monclient: dump: 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.225 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 10 monclient: shutdown 2024-04-19T01:03:40.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 -- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10f0138a50 msgr2=0x7f10f0138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 --2- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10f0138a50 0x7f10f0138e30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f10f00a72d0 tx=0x7f10e001c420 comp rx=0 tx=0).stop 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 -- 172.21.15.43:0/2389105064 shutdown_connections 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 --2- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f10f0137960 0x7f10f0137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 --2- 172.21.15.43:0/2389105064 >> v2:172.21.15.43:3300/0 conn(0x7f10f00a5bb0 0x7f10f0139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 --2- 172.21.15.43:0/2389105064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10f0138a50 0x7f10f0138e30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 -- 172.21.15.43:0/2389105064 >> 172.21.15.43:0/2389105064 conn(0x7f10f00a7cb0 msgr2=0x7f10f00a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 -- 172.21.15.43:0/2389105064 shutdown_connections 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.211+0000 7f10f5477640 1 -- 172.21.15.43:0/2389105064 wait complete. 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: build_initial_monmap 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: monmap: 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:03:40.227 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:03:40.228 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 librados: starting msgr at 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 librados: starting objecter 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 Processor -- start 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- start start 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 librados: setting wanted keys 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 librados: calling monclient init 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: init 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: _reopen_session rank -1 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: _add_conns ranks=[0,5,4,3,1,7,6,2,8] 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f00a5bb0 0x7f10f01d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): picked mon.noname-a con 0x7f10f00a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0137960 0x7f10f01d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): picked mon.noname-h con 0x7f10f0137960 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:03:40.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f10f0138a50 0x7f10f01d1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): picked mon.noname-c con 0x7f10f0138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f10f00a2100 con 0x7f10f00a5bb0 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f10f00a1f80 con 0x7f10f0138a50 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f10f0142810 con 0x7f10f0137960 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): _renew_subs 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): authenticate will time out at 2090.345077s 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10eeffd640 1 -- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f00a5bb0 msgr2=0x7f10f01d0b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3300/0 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10eeffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f00a5bb0 0x7f10f01d0b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ef7fe640 1 -- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f10f0138a50 msgr2=0x7f10f01d1610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.43:3302/0 2024-04-19T01:03:40.230 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ef7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f10f0138a50 0x7f10f01d1610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0137960 0x7f10f01d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0137960 0x7f10f01d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:39384/0 (socket says 172.21.15.43:39384) 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 -- 172.21.15.43:0/2807675208 learned_addr learned my addr 172.21.15.43:0/2807675208 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): get_auth_request con 0x7f10f0137960 auth_method 0 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient(hunting): handle_auth_done global_id 75065 payload 824 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f10f0138a50 msgr2=0x7f10f01d1610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:40.231 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f10f0138a50 0x7f10f01d1610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f00a5bb0 msgr2=0x7f10f01d0b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f10f00a5bb0 0x7f10f01d0b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 monclient: found mon.noname-h 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f009ee60 con 0x7f10f0137960 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ee7fc640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0137960 0x7f10f01d10d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f10e400b550 tx=0x7f10e400ba10 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 client.?.objecter ms_handle_connect 0x7f10f0137960 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e40092d0 con 0x7f10f0137960 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: dump: 2024-04-19T01:03:40.232 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.233 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: _finish_auth 0 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: _check_auth_tickets 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 5 monclient: authenticate success, global_id 75065 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: _renew_subs 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f009fc90 con 0x7f10f0137960 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e4015070 con 0x7f10f0137960 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e4003c10 con 0x7f10f0137960 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 client.75065.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10d77fe640 10 monclient: dump: 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.235 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10ecff9640 4 set_mon_vals no callback set 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: _renew_subs 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f013c0a0 con 0x7f10f0137960 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 librados: init done 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 0x7f10f0132910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f10f006d400 con 0x7f10f013f7f0 2024-04-19T01:03:40.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.215+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 0x7f10f0132910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient: get_auth_request con 0x7f10f013f7f0 auth_method 0 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 10 monclient(hunting): handle_auth_done global_id 75074 payload 824 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10f0136b20 con 0x7f10f013f7f0 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f10e40050f0 con 0x7f10f0137960 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 0x7f10f0132910 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10f006e8c0 tx=0x7f10e001ac60 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f10dc038e10 0x7f10dc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 client.75065.objecter ms_handle_connect 0x7f10f013f7f0 2024-04-19T01:03:40.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 client.75065.objecter resend_mon_ops 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(706..706 src has 501..706) ==== 5607+0+0 (secure 0 0 0) 0x7f10e4049080 con 0x7f10f0137960 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 client.75065.objecter ms_dispatch 0x7f10f0000ba0 osd_map(706..706 src has 501..706) 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 3 client.75065.objecter handle_osd_map got epochs [706,706] > 0 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 3 client.75065.objecter handle_osd_map decoding full epoch 706 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 20 client.75065.objecter dump_active .. 0 homeless 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e00ba070 con 0x7f10f013f7f0 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 monclient: dump: 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.238 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10ef7fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f10dc038e10 0x7f10dc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10ef7fe640 10 monclient: get_auth_request con 0x7f10dc038e10 auth_method 0 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f10e00093d0 con 0x7f10f013f7f0 2024-04-19T01:03:40.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 msgr2=0x7f10f0132910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 0x7f10f0132910 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10f006e8c0 tx=0x7f10e001ac60 comp rx=0 tx=0).stop 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 msgr2=0x7f10f0132910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f013f7f0 0x7f10f0132910 secure :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f10f006e8c0 tx=0x7f10e001ac60 comp rx=0 tx=0).stop 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10ef7fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f10dc038e10 0x7f10dc03b2d0 secure :-1 s=READY pgs=3654 cs=0 l=1 rev1=1 crypto rx=0x7f10e8003c70 tx=0x7f10e80091f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.219+0000 7f10d77fe640 10 client.75065.objecter ms_handle_connect 0x7f10dc038e10 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.223+0000 7f10f5477640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.223+0000 7f10f5477640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 0x7f10f0133230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 10 monclient(hunting): start opening mon connection 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f10f006d140 con 0x7f10f0132e50 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 0x7f10f0133230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient: get_auth_request con 0x7f10f0132e50 auth_method 0 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:03:40.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 10 monclient(hunting): handle_auth_done global_id 75083 payload 824 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 1 -- 172.21.15.43:0/2807675208 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10f0133770 con 0x7f10f0132e50 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10eeffd640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 0x7f10f0133230 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f10e001aad0 tx=0x7f10e00191c0 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 client.75065.objecter ms_handle_connect 0x7f10f0132e50 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 client.75065.objecter resend_mon_ops 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f10e00ba040 con 0x7f10f0132e50 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 monclient: dump: 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:03:40.243 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:03:40.244 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9230 (secure 0 0 0) 0x7f10e0003620 con 0x7f10f0132e50 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 msgr2=0x7f10f0133230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 0x7f10f0133230 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f10e001aad0 tx=0x7f10e00191c0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 msgr2=0x7f10f0133230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10d77fe640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0132e50 0x7f10f0133230 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:40.245 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 126, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 599844, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 7, 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:03:40.246 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:03:40.247 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.248 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:03:40.249 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:03:40.250 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.251 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.252 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.253 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.254 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.255 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.256 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:03:40.257 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.258 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.259 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.260 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.261 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:03:40.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:03:40.265 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:03:40.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:03:40.267 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:03:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 10 librados: watch_flush enter 2024-04-19T01:03:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 10 librados: watch_flush exit 2024-04-19T01:03:40.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 20 client.75065.objecter shutdown clearing up homeless session... 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 10 client.75065.objecter successfully canceled tick 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f10dc038e10 msgr2=0x7f10dc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f10dc038e10 0x7f10dc03b2d0 secure :-1 s=READY pgs=3654 cs=0 l=1 rev1=1 crypto rx=0x7f10e8003c70 tx=0x7f10e80091f0 comp rx=0 tx=0).stop 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 10 monclient: shutdown 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0137960 msgr2=0x7f10f01d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 --2- 172.21.15.43:0/2807675208 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f10f0137960 0x7f10f01d10d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f10e400b550 tx=0x7f10e400ba10 comp rx=0 tx=0).stop 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10ef7fe640 1 -- 172.21.15.43:0/2807675208 reap_dead start 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10ef7fe640 1 -- 172.21.15.43:0/2807675208 reap_dead start 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 shutdown_connections 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 >> 172.21.15.43:0/2807675208 conn(0x7f10f00a7cb0 msgr2=0x7f10f00a25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 shutdown_connections 2024-04-19T01:03:40.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 -- 172.21.15.43:0/2807675208 wait complete. 2024-04-19T01:03:40.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:03:40.227+0000 7f10f5477640 1 librados: shutdown 2024-04-19T01:03:40.271 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 5 2024-04-19T01:03:40.271 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7935 ms) 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13075 ms) 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7943 ms) 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (47684 ms) 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-19T01:03:40.695 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7818 ms) 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8089 ms) 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: ok, hit_set contains 333:602f83fe:::foo:head 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9100 ms) 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,0 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: first is 1713488509 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,0 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,0 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,0 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,0 2024-04-19T01:03:40.696 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,1713488514,1713488515,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,1713488514,1713488515,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,1713488514,1713488515,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,1713488514,1713488515,1713488517,1713488518,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,1713488514,1713488515,1713488517,1713488518,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488509,1713488511,1713488512,1713488514,1713488515,1713488517,1713488518,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: got ls 1713488512,1713488514,1713488515,1713488517,1713488518,1713488520,1713488521,0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: first now 1713488512, trimmed 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20937 ms) 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: foo0 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13972 ms) 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-19T01:03:40.697 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (19021 ms) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (24561 ms) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (4401 ms) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3256 ms) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (27108 ms) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3035 ms) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (330951 ms total) 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-19T01:03:40.698 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1094318 ms total) 2024-04-19T01:03:40.699 INFO:tasks.workunit.client.0.smithi161.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28744 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28744 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28886 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28886 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29282 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29282 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.700 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29088 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29088 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29010 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29010 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28793 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28793 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29046 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29046 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29476 2024-04-19T01:03:40.701 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29476 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29543 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29543 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28822 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28822 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29357 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29357 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.708 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29247 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29247 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28728 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28728 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28760 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28760 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29255 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29255 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28720 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28720 2024-04-19T01:03:40.709 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28736 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28736 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29634 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29634 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28932 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28932 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29661 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29661 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29035 2024-04-19T01:03:40.710 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29035 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29450 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29450 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29686 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29686 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28960 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28960 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29263 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29263 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.711 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29620 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29620 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29433 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29433 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29647 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29647 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29238 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29238 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28865 2024-04-19T01:03:40.712 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28865 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29670 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29670 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29108 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29108 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28712 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28712 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28752 2024-04-19T01:03:40.713 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28752 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29080 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29080 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=28768 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 28768 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29175 2024-04-19T01:03:40.714 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29175 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29213 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29213 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29603 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29603 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ for t in "${!pids[@]}" 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ pid=29703 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ wait 29703 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ exit 0 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ cleanup 2024-04-19T01:03:40.715 INFO:tasks.workunit.client.0.smithi161.stderr:+ pkill -P 28706 2024-04-19T01:03:40.716 INFO:tasks.workunit.client.0.smithi161.stderr:+ true 2024-04-19T01:03:40.717 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-19T01:03:40.717 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-19T01:03:40.731 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-04-19T01:03:40.731 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-19T01:03:41.079 DEBUG:teuthology.parallel:result is None 2024-04-19T01:03:41.080 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-19T01:03:41.093 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-19T01:03:41.093 DEBUG:teuthology.orchestra.run.smithi161:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-19T01:03:41.201 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-19T01:03:41.201 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-19T01:03:41.212 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-19T01:05:10.272 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-19T01:05:10.272 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-19T01:05:10.272 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-19T01:05:10.273 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-19T01:05:10.277 INFO:tasks.ceph.mon.a:Started 2024-04-19T01:05:10.278 INFO:tasks.mon_thrash.mon_thrasher:killing mon.i 2024-04-19T01:05:10.278 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.i 2024-04-19T01:05:10.278 INFO:tasks.ceph.mon.i:Restarting daemon 2024-04-19T01:05:10.278 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i i 2024-04-19T01:05:10.281 INFO:tasks.ceph.mon.i:Started 2024-04-19T01:05:10.281 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-19T01:05:10.281 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-19T01:05:10.281 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-19T01:05:10.281 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-19T01:05:10.283 INFO:tasks.ceph.mon.c:Started 2024-04-19T01:05:10.283 INFO:tasks.mon_thrash.mon_thrasher:killing mon.d 2024-04-19T01:05:10.284 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.d 2024-04-19T01:05:10.284 INFO:tasks.ceph.mon.d:Restarting daemon 2024-04-19T01:05:10.284 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i d 2024-04-19T01:05:10.286 INFO:tasks.ceph.mon.d:Started 2024-04-19T01:05:10.286 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-04-19T01:05:10.287 INFO:tasks.ceph.mon.b:Sent signal 19 2024-04-19T01:05:10.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-04-19T01:05:10.413 INFO:tasks.ceph.mon.i.smithi161.stderr:2024-04-19T01:05:10.415+0000 7efcedafdc80 -1 mon.i@-1(???).log v0 log_external_backlog rewinding external_log_to from 739 -> 0 (sync_force? mon rebuild?) 2024-04-19T01:05:25.287 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-04-19T01:05:25.288 INFO:tasks.ceph.mon.b:Sent signal 18 2024-04-19T01:05:25.288 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T01:05:25.289 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:05:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient: get_monmap_and_config 2024-04-19T01:05:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient: build_initial_monmap 2024-04-19T01:05:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient: monmap: 2024-04-19T01:05:25.432 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:25.435349+0000 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:25.435349+0000 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:25.433 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:25.434 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:25.434 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 Processor -- start 2024-04-19T01:05:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 -- start start 2024-04-19T01:05:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient: init 2024-04-19T01:05:25.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient: _add_conns ranks=[9,5,8,6,10,0,4,7,2,1,3] 2024-04-19T01:05:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f639c0a3050 0x7f639c136e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): picked mon.noname-a con 0x7f639c0a3050 addr v2:172.21.15.43:3300/0 2024-04-19T01:05:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f639c0a3430 0x7f639c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): picked mon.noname-h con 0x7f639c0a3430 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:25.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 0x7f639c09a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): picked mon.noname-e con 0x7f639c09a260 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f639c09ec70 con 0x7f639c0a3050 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f639c09ef50 con 0x7f639c0a3430 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f639c09f230 con 0x7f639c09a260 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a37c1640 10 monclient(hunting): authenticate will time out at 2195.563286s 2024-04-19T01:05:25.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 0x7f639c09a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f639c0a3050 0x7f639c136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 0x7f639c09a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57892/0 (socket says 172.21.15.43:57892) 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 -- 172.21.15.43:0/1241305230 learned_addr learned my addr 172.21.15.43:0/1241305230 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): get_auth_request con 0x7f639c09a260 auth_method 0 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): get_auth_request con 0x7f639c0a3050 auth_method 0 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a155d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a0d5c640 1 --2- 172.21.15.43:0/1241305230 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f639c0a3430 0x7f639c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a0d5c640 10 monclient(hunting): get_auth_request con 0x7f639c0a3430 auth_method 0 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a0d5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a0d5c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a0d5c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_done global_id 144584 payload 274 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 -- 172.21.15.43:0/1241305230 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f639c0a3430 msgr2=0x7f639c0a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 --2- 172.21.15.43:0/1241305230 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f639c0a3430 0x7f639c0a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 -- 172.21.15.43:0/1241305230 >> v2:172.21.15.43:3300/0 conn(0x7f639c0a3050 msgr2=0x7f639c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 --2- 172.21.15.43:0/1241305230 >> v2:172.21.15.43:3300/0 conn(0x7f639c0a3050 0x7f639c136e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient: _finish_hunting 0 2024-04-19T01:05:25.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 monclient: found mon.noname-e 2024-04-19T01:05:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:25.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.434+0000 7f63a1d5e640 1 -- 172.21.15.43:0/1241305230 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c0a4f50 con 0x7f639c09a260 2024-04-19T01:05:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a1d5e640 1 --2- 172.21.15.43:0/1241305230 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 0x7f639c09a6f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6394009e50 tx=0x7f639401c8f0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=79ac9f6d817ce334 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 1 -- 172.21.15.43:0/1241305230 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63940090d0 con 0x7f639c09a260 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: dump: 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.447 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: _finish_auth 0 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: _check_auth_tickets 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 1 -- 172.21.15.43:0/1241305230 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63940a5a20 con 0x7f639c09a260 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 5 monclient: authenticate success, global_id 144584 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 4 set_mon_vals no callback set 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 1 -- 172.21.15.43:0/1241305230 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f63940a5c10 con 0x7f639c09a260 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f6393fff640 10 monclient: dump: 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.449 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 10 monclient: shutdown 2024-04-19T01:05:25.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 -- 172.21.15.43:0/1241305230 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 msgr2=0x7f639c09a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 --2- 172.21.15.43:0/1241305230 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 0x7f639c09a6f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6394009e50 tx=0x7f639401c8f0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 -- 172.21.15.43:0/1241305230 shutdown_connections 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 --2- 172.21.15.43:0/1241305230 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f639c09a260 0x7f639c09a6f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 --2- 172.21.15.43:0/1241305230 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f639c0a3430 0x7f639c0a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 --2- 172.21.15.43:0/1241305230 >> v2:172.21.15.43:3300/0 conn(0x7f639c0a3050 0x7f639c136e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 -- 172.21.15.43:0/1241305230 >> 172.21.15.43:0/1241305230 conn(0x7f639c0a4140 msgr2=0x7f639c0a4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 -- 172.21.15.43:0/1241305230 shutdown_connections 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 1 -- 172.21.15.43:0/1241305230 wait complete. 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 10 monclient: build_initial_monmap 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.438+0000 7f63a37c1640 10 monclient: monmap: 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:25.451 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:25.452 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 librados: starting msgr at 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 librados: starting objecter 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 Processor -- start 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- start start 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 librados: setting wanted keys 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 librados: calling monclient init 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: init 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _add_conns ranks=[1,4,2,5,0,8,7,3,6] 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f639c09a260 0x7f639c1cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): picked mon.noname-f con 0x7f639c09a260 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:25.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 0x7f639c1cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): picked mon.noname-c con 0x7f639c0a3050 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f639c0a3430 0x7f639c1cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): picked mon.noname-b con 0x7f639c0a3430 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f639c140750 con 0x7f639c09a260 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f639c137590 con 0x7f639c0a3430 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f639c1408d0 con 0x7f639c0a3050 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient(hunting): authenticate will time out at 2195.568874s 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 0x7f639c1cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f639c0a3430 0x7f639c1cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 0x7f639c1cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34958/0 (socket says 172.21.15.43:34958) 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 -- 172.21.15.43:0/229443266 learned_addr learned my addr 172.21.15.43:0/229443266 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): get_auth_request con 0x7f639c0a3050 auth_method 0 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): get_auth_request con 0x7f639c0a3430 auth_method 0 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a1d5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a155d640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f639c09a260 0x7f639c1cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient(hunting): handle_auth_done global_id 184108 payload 824 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 -- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f639c0a3430 msgr2=0x7f639c1cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f639c0a3430 0x7f639c1cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 -- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f639c09a260 msgr2=0x7f639c1cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f639c09a260 0x7f639c1cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient: _finish_hunting 0 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 monclient: found mon.noname-c 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 -- 172.21.15.43:0/229443266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c08ec40 con 0x7f639c0a3050 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a155d640 10 monclient: get_auth_request con 0x7f639c09a260 auth_method 0 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a155d640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f639c09a260 0x7f639c1cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:25.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a0d5c640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 0x7f639c1cf200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f638c00d630 tx=0x7f638c00daf0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 client.?.objecter ms_handle_connect 0x7f639c0a3050 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f638c004240 con 0x7f639c0a3050 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: dump: 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.457 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: _finish_auth 0 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:05:25.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638c0043e0 con 0x7f639c0a3050 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f638c012420 con 0x7f639c0a3050 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 monclient: dump: 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.459 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63937fe640 4 set_mon_vals no callback set 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 5 monclient: authenticate success, global_id 184108 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _renew_subs 2024-04-19T01:05:25.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639c08fcb0 con 0x7f639c0a3050 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 client.184108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _renew_subs 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f639c09df40 con 0x7f639c0a3050 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 librados: init done 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f638c003620 con 0x7f639c0a3050 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639c0a1ab0 con 0x7f639c0a3050 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6380038e10 0x7f638003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f638c00f040 con 0x7f639c0a3050 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 10 client.184108.objecter ms_dispatch 0x7f639c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:25.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 3 client.184108.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 3 client.184108.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f6391ffb640 20 client.184108.objecter dump_active .. 0 homeless 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a155d640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6380038e10 0x7f638003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.442+0000 7f63a155d640 10 monclient: get_auth_request con 0x7f6380038e10 auth_method 0 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.446+0000 7f6391ffb640 10 client.184108.objecter ms_handle_connect 0x7f6380038e10 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.446+0000 7f63a155d640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6380038e10 0x7f638003b2d0 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f6398009ae0 tx=0x7f6398008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.450+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f639c08ec40 con 0x7f639c0a3050 2024-04-19T01:05:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.450+0000 7f6391ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:25.463 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.450+0000 7f6391ffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.674+0000 7f63a37c1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:05:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.674+0000 7f63a37c1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:05:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.674+0000 7f63a37c1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:25.675 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.674+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f639c099eb0 con 0x7f639c0a3050 2024-04-19T01:05:25.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f6391ffb640 1 -- 172.21.15.43:0/229443266 <== mon.4 v2:172.21.15.43:3302/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f638c00bd10 con 0x7f639c0a3050 2024-04-19T01:05:25.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f6391ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:05:25.676 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f6391ffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:25.676 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":132,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 10 librados: watch_flush enter 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 10 librados: watch_flush exit 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 20 client.184108.objecter shutdown clearing up homeless session... 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 10 client.184108.objecter successfully canceled tick 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6380038e10 msgr2=0x7f638003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6380038e10 0x7f638003b2d0 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f6398009ae0 tx=0x7f6398008040 comp rx=0 tx=0).stop 2024-04-19T01:05:25.679 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 10 monclient: shutdown 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 msgr2=0x7f639c1cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 0x7f639c1cf200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f638c00d630 tx=0x7f638c00daf0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 shutdown_connections 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6380038e10 0x7f638003b2d0 unknown :-1 s=CLOSED pgs=3660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f639c0a3430 0x7f639c1cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f639c0a3050 0x7f639c1cf200 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 --2- 172.21.15.43:0/229443266 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f639c09a260 0x7f639c1cecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 >> 172.21.15.43:0/229443266 conn(0x7f639c0a4140 msgr2=0x7f639c08d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 shutdown_connections 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 -- 172.21.15.43:0/229443266 wait complete. 2024-04-19T01:05:25.680 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.678+0000 7f63a37c1640 1 librados: shutdown 2024-04-19T01:05:25.696 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T01:05:25.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon a 2024-04-19T01:05:25.696 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient: get_monmap_and_config 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient: build_initial_monmap 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient: monmap: 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:25.819116+0000 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:25.819116+0000 2024-04-19T01:05:25.816 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:25.817 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 Processor -- start 2024-04-19T01:05:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 -- start start 2024-04-19T01:05:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient: init 2024-04-19T01:05:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:25.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient: _add_conns ranks=[3,4,1,10,0,9,2,5,8,7,6] 2024-04-19T01:05:25.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32f80a1bf0 0x7f32f80a1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): picked mon.noname-g con 0x7f32f80a1bf0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:25.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 0x7f32f80a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): picked mon.noname-c con 0x7f32f80a1270 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:25.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f32f80a3ef0 addr v1:172.21.15.161:6789/0 2024-04-19T01:05:25.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32f806c330 con 0x7f32f80a3ef0 2024-04-19T01:05:25.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f32f80a2510 con 0x7f32f80a1bf0 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f32f809fff0 con 0x7f32f80a1270 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f330042a640 10 monclient(hunting): authenticate will time out at 2195.946245s 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 0x7f32f80a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 0x7f32f80a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34960/0 (socket says 172.21.15.43:34960) 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 -- 172.21.15.43:0/944202343 learned_addr learned my addr 172.21.15.43:0/944202343 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): get_auth_request con 0x7f32f80a1270 auth_method 0 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fe1c6640 1 --2- 172.21.15.43:0/944202343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32f80a1bf0 0x7f32f80a1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fe1c6640 10 monclient(hunting): get_auth_request con 0x7f32f80a1bf0 auth_method 0 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fe1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fe1c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fe1c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_done global_id 184117 payload 274 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 -- 172.21.15.43:0/944202343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32f80a1bf0 msgr2=0x7f32f80a1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 --2- 172.21.15.43:0/944202343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32f80a1bf0 0x7f32f80a1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 -- 172.21.15.43:0/944202343 >> v1:172.21.15.161:6789/0 conn(0x7f32f80a3ef0 legacy=0x7f32f813fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient: _finish_hunting 0 2024-04-19T01:05:25.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 monclient: found mon.noname-c 2024-04-19T01:05:25.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:25.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:25.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.818+0000 7f32fd9c5640 1 -- 172.21.15.43:0/944202343 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f80a02d0 con 0x7f32f80a1270 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd9c5640 1 --2- 172.21.15.43:0/944202343 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 0x7f32f80a39b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098a0 tx=0x7f32e801c470 comp rx=0 tx=0).ready entity=mon.4 client_cookie=84df3cf124b94480 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 1 -- 172.21.15.43:0/944202343 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32e80b0070 con 0x7f32f80a1270 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: dump: 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.843 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: _finish_auth 0 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: _check_auth_tickets 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 1 -- 172.21.15.43:0/944202343 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e80aa810 con 0x7f32f80a1270 2024-04-19T01:05:25.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 1 -- 172.21.15.43:0/944202343 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32e80a9300 con 0x7f32f80a1270 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 5 monclient: authenticate success, global_id 184117 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 4 set_mon_vals no callback set 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f32fd1c4640 10 monclient: dump: 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.845 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.848 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.849 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.850 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 10 monclient: shutdown 2024-04-19T01:05:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 -- 172.21.15.43:0/944202343 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 msgr2=0x7f32f80a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.850 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 --2- 172.21.15.43:0/944202343 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 0x7f32f80a39b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f32e80098a0 tx=0x7f32e801c470 comp rx=0 tx=0).stop 2024-04-19T01:05:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 -- 172.21.15.43:0/944202343 shutdown_connections 2024-04-19T01:05:25.851 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 --2- 172.21.15.43:0/944202343 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f32f80a1270 0x7f32f80a39b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 --2- 172.21.15.43:0/944202343 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f32f80a1bf0 0x7f32f80a1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 -- 172.21.15.43:0/944202343 >> 172.21.15.43:0/944202343 conn(0x7f32f80a85a0 msgr2=0x7f32f80a4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 -- 172.21.15.43:0/944202343 shutdown_connections 2024-04-19T01:05:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 -- 172.21.15.43:0/944202343 wait complete. 2024-04-19T01:05:25.852 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 10 monclient: build_initial_monmap 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 10 monclient: monmap: 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:25.854 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 librados: starting msgr at 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 librados: starting objecter 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 Processor -- start 2024-04-19T01:05:25.855 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 -- start start 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 librados: setting wanted keys 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 1 librados: calling monclient init 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.822+0000 7f330042a640 10 monclient: init 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient: _add_conns ranks=[1,2,6,7,5,3,0,4,8] 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32f80a1bf0 0x7f32f81d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): picked mon.noname-f con 0x7f32f80a1bf0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32f80a3ef0 0x7f32f81380f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): picked mon.noname-b con 0x7f32f80a3ef0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32f8138630 0x7f32f813d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32f80a3ef0 0x7f32f81380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32f80a3ef0 0x7f32f81380f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47128/0 (socket says 172.21.15.43:47128) 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): picked mon.noname-d con 0x7f32f8138630 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:25.856 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32f806c7f0 con 0x7f32f80a1bf0 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 -- 172.21.15.43:0/4247224073 learned_addr learned my addr 172.21.15.43:0/4247224073 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f32f80a5470 con 0x7f32f80a3ef0 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f32f806c970 con 0x7f32f8138630 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): authenticate will time out at 2195.952638s 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): get_auth_request con 0x7f32f80a3ef0 auth_method 0 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32f80a1bf0 0x7f32f81d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.857 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): get_auth_request con 0x7f32f80a1bf0 auth_method 0 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe9c7640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32f8138630 0x7f32f813d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe9c7640 10 monclient(hunting): get_auth_request con 0x7f32f8138630 auth_method 0 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe9c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe9c7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe9c7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.858 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient(hunting): handle_auth_done global_id 144551 payload 824 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32f8138630 msgr2=0x7f32f813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f32f8138630 0x7f32f813d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32f80a1bf0 msgr2=0x7f32f81d0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32f80a1bf0 0x7f32f81d0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient: _finish_hunting 0 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 monclient: found mon.noname-b 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32f8141070 con 0x7f32f80a3ef0 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fd9c5640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32f80a3ef0 0x7f32f81380f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f32e80ae040 tx=0x7f32e8019270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 client.?.objecter ms_handle_connect 0x7f32f80a3ef0 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:25.859 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32e80b9070 con 0x7f32f80a3ef0 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: dump: 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.860 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: _finish_auth 0 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32e80044b0 con 0x7f32f80a3ef0 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32e80b0040 con 0x7f32f80a3ef0 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 monclient: dump: 2024-04-19T01:05:25.861 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.862 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fc9c3640 4 set_mon_vals no callback set 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 5 monclient: authenticate success, global_id 144551 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient: _renew_subs 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32f813f060 con 0x7f32f80a3ef0 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 client.144551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient: _renew_subs 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f32f81d1b80 con 0x7f32f80a3ef0 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 librados: init done 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 0x7f32f813d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.863 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32f8073100 con 0x7f32f80a1270 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f32e8004040 con 0x7f32f80a3ef0 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 0x7f32f813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient: get_auth_request con 0x7f32f80a1270 auth_method 0 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f32dc038e40 0x7f32dc03b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f32e80e2250 con 0x7f32f80a3ef0 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 10 client.144551.objecter ms_dispatch 0x7f32f8000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 3 client.144551.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:25.864 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 3 client.144551.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32eeffd640 20 client.144551.objecter dump_active .. 0 homeless 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_done global_id 184113 payload 824 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.826+0000 7f32fe1c6640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32f809f650 con 0x7f32f80a1270 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32fe1c6640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 0x7f32f813d970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32f4004680 tx=0x7f32f4007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 client.144551.objecter ms_handle_connect 0x7f32f80a1270 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32fe9c7640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f32dc038e40 0x7f32dc03b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 client.144551.objecter resend_mon_ops 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32fe9c7640 10 monclient: get_auth_request con 0x7f32dc038e40 auth_method 0 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32f4015890 con 0x7f32f80a1270 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 monclient: dump: 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.865 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.866 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f32f401a070 con 0x7f32f80a1270 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 msgr2=0x7f32f813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 0x7f32f813d970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f32f4004680 tx=0x7f32f4007700 comp rx=0 tx=0).stop 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 msgr2=0x7f32f813d970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f80a1270 0x7f32f813d970 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32eeffd640 10 client.144551.objecter ms_handle_connect 0x7f32dc038e40 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.830+0000 7f32fe9c7640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f32dc038e40 0x7f32dc03b300 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7f32f8138a50 tx=0x7f32f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 0x7f32f813deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f32f806d140 con 0x7f32f813fde0 2024-04-19T01:05:25.867 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 0x7f32f813deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient: get_auth_request con 0x7f32f813fde0 auth_method 0 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 10 monclient(hunting): handle_auth_done global_id 184122 payload 824 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 1 -- 172.21.15.43:0/4247224073 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f32f80766e0 con 0x7f32f813fde0 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe1c6640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 0x7f32f813deb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32f4017040 tx=0x7f32f4004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 client.144551.objecter ms_handle_connect 0x7f32f813fde0 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 client.144551.objecter resend_mon_ops 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f32f4015de0 con 0x7f32f813fde0 2024-04-19T01:05:25.868 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 monclient: dump: 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:25.869 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 <== mon.0 v2:172.21.15.43:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f32f4015970 con 0x7f32f813fde0 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 msgr2=0x7f32f813deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 0x7f32f813deb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32f4017040 tx=0x7f32f4004a50 comp rx=0 tx=0).stop 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 msgr2=0x7f32f813deb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:25.870 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32eeffd640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f32f813fde0 0x7f32f813deb0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f32f4017040 tx=0x7f32f4004a50 comp rx=0 tx=0).stop 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15455, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:25.871 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:25.872 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:25.873 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:25.874 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:25.875 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:25.876 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.877 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.878 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.879 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:25.880 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.881 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:25.882 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.883 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.884 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.885 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.886 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.887 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:25.888 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:25.889 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:25.890 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.891 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:25.892 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:25.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 10 librados: watch_flush enter 2024-04-19T01:05:25.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 10 librados: watch_flush exit 2024-04-19T01:05:25.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 20 client.144551.objecter shutdown clearing up homeless session... 2024-04-19T01:05:25.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 10 client.144551.objecter successfully canceled tick 2024-04-19T01:05:25.894 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f32dc038e40 msgr2=0x7f32dc03b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f32dc038e40 0x7f32dc03b300 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7f32f8138a50 tx=0x7f32f0009040 comp rx=0 tx=0).stop 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 10 monclient: shutdown 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32f80a3ef0 msgr2=0x7f32f81380f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 --2- 172.21.15.43:0/4247224073 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f32f80a3ef0 0x7f32f81380f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f32e80ae040 tx=0x7f32e8019270 comp rx=0 tx=0).stop 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe9c7640 1 -- 172.21.15.43:0/4247224073 reap_dead start 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f32fe9c7640 1 -- 172.21.15.43:0/4247224073 reap_dead start 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 shutdown_connections 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 >> 172.21.15.43:0/4247224073 conn(0x7f32f80a85a0 msgr2=0x7f32f80a7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 shutdown_connections 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 -- 172.21.15.43:0/4247224073 wait complete. 2024-04-19T01:05:25.895 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:25.838+0000 7f330042a640 1 librados: shutdown 2024-04-19T01:05:25.896 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:25.896 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon b 2024-04-19T01:05:25.896 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-19T01:05:26.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.018+0000 7f5d2aa16640 10 monclient: get_monmap_and_config 2024-04-19T01:05:26.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.018+0000 7f5d2aa16640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.019 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.018+0000 7f5d2aa16640 10 monclient: monmap: 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:26.021436+0000 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:26.021436+0000 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.020 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.021 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:26.021 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:26.021 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 Processor -- start 2024-04-19T01:05:26.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 -- start start 2024-04-19T01:05:26.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient: init 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient: _add_conns ranks=[1,2,3,0,9,6,8,7,4,10,5] 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f5d24138a50 addr v1:172.21.15.161:6789/0 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 0x7f5d2413dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): picked mon.noname-b con 0x7f5d24139540 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d240a5bb0 0x7f5d24140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): picked mon.noname-g con 0x7f5d240a5bb0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d240a0d40 con 0x7f5d24138a50 2024-04-19T01:05:26.024 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5d240a1020 con 0x7f5d24139540 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f5d240a0780 con 0x7f5d240a5bb0 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient(hunting): authenticate will time out at 2196.149062s 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 0x7f5d2413dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 0x7f5d2413dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47132/0 (socket says 172.21.15.43:47132) 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 -- 172.21.15.43:0/3653782239 learned_addr learned my addr 172.21.15.43:0/3653782239 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): get_auth_request con 0x7f5d24139540 auth_method 0 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.025 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d28fb3640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d240a5bb0 0x7f5d24140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient(hunting): handle_auth_done global_id 144560 payload 274 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 -- 172.21.15.43:0/3653782239 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d240a5bb0 msgr2=0x7f5d24140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d240a5bb0 0x7f5d24140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 -- 172.21.15.43:0/3653782239 >> v1:172.21.15.161:6789/0 conn(0x7f5d24138a50 legacy=0x7f5d24138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 monclient: found mon.noname-b 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 -- 172.21.15.43:0/3653782239 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d240a0a60 con 0x7f5d24139540 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d28fb3640 10 monclient: get_auth_request con 0x7f5d240a5bb0 auth_method 0 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d28fb3640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d240a5bb0 0x7f5d24140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:26.026 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 0x7f5d2413dfe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5d180098a0 tx=0x7f5d1801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e920810305737b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 1 -- 172.21.15.43:0/3653782239 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d180b0070 con 0x7f5d24139540 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: dump: 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.027 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 5 monclient: authenticate success, global_id 144560 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 1 -- 172.21.15.43:0/3653782239 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d180aa810 con 0x7f5d24139540 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.028 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 1 -- 172.21.15.43:0/3653782239 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d180a9300 con 0x7f5d24139540 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d2aa16640 4 set_mon_vals no callback set 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.022+0000 7f5d22ffd640 10 monclient: dump: 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.029 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.030 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: shutdown 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3653782239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 msgr2=0x7f5d2413dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 0x7f5d2413dfe0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5d180098a0 tx=0x7f5d1801c440 comp rx=0 tx=0).stop 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3653782239 shutdown_connections 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5d240a5bb0 0x7f5d24140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3653782239 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d24139540 0x7f5d2413dfe0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3653782239 >> 172.21.15.43:0/3653782239 conn(0x7f5d240a7cb0 msgr2=0x7f5d240a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3653782239 shutdown_connections 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3653782239 wait complete. 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.031 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: monmap: 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.032 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 librados: starting msgr at 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 librados: starting objecter 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 Processor -- start 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- start start 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 librados: setting wanted keys 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 librados: calling monclient init 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: init 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: _add_conns ranks=[0,5,7,2,8,3,1,4,6] 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 0x7f5d241d0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.042 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): picked mon.noname-a con 0x7f5d240a5bb0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5d24138a50 0x7f5d241378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): picked mon.noname-h con 0x7f5d24138a50 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5d24139540 0x7f5d2413c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): picked mon.noname-i con 0x7f5d24139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24143620 con 0x7f5d240a5bb0 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5d241434a0 con 0x7f5d24138a50 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f5d241437a0 con 0x7f5d24139540 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): authenticate will time out at 2196.153971s 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 0x7f5d241d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.043 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 0x7f5d241d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57366/0 (socket says 172.21.15.43:57366) 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 -- 172.21.15.43:0/3687746133 learned_addr learned my addr 172.21.15.43:0/3687746133 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): get_auth_request con 0x7f5d240a5bb0 auth_method 0 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d28fb3640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5d24139540 0x7f5d2413c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5d24138a50 0x7f5d241378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d28fb3640 10 monclient(hunting): get_auth_request con 0x7f5d24139540 auth_method 0 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d28fb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d28fb3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.044 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d28fb3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): get_auth_request con 0x7f5d24138a50 auth_method 0 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient(hunting): handle_auth_done global_id 184131 payload 824 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5d24139540 msgr2=0x7f5d2413c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5d24139540 0x7f5d2413c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.045 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5d24138a50 msgr2=0x7f5d241378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5d24138a50 0x7f5d241378f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.046 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 monclient: found mon.noname-a 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2409fc90 con 0x7f5d240a5bb0 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d23fff640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 0x7f5d241d0720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400c480 tx=0x7f5d1400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 client.?.objecter ms_handle_connect 0x7f5d240a5bb0 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d14012aa0 con 0x7f5d240a5bb0 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: dump: 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.047 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 5 monclient: authenticate success, global_id 184131 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d14017070 con 0x7f5d240a5bb0 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: _renew_subs 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d24121fa0 con 0x7f5d240a5bb0 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d14012c80 con 0x7f5d240a5bb0 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 client.184131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d227fc640 4 set_mon_vals no callback set 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 10 monclient: dump: 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.049 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: _renew_subs 2024-04-19T01:05:26.050 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2413f110 con 0x7f5d240a5bb0 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 librados: init done 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 0x7f5d2402c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2413ed20 con 0x7f5d2402c450 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5d14012630 con 0x7f5d240a5bb0 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 0x7f5d2402c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient: get_auth_request con 0x7f5d2402c450 auth_method 0 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.026+0000 7f5d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.051 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5d10038e10 0x7f5d1003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f5d14048f40 con 0x7f5d240a5bb0 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 client.184131.objecter ms_dispatch 0x7f5d24000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 3 client.184131.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 3 client.184131.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d237fe640 10 monclient(hunting): handle_auth_done global_id 144569 payload 824 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d237fe640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d2409ee60 con 0x7f5d2402c450 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 20 client.184131.objecter dump_active .. 0 homeless 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 client.184131.objecter ms_handle_connect 0x7f5d2402c450 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 0x7f5d2402c830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d240a72d0 tx=0x7f5d180191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 client.184131.objecter resend_mon_ops 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d28fb3640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5d10038e10 0x7f5d1003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d28fb3640 10 monclient: get_auth_request con 0x7f5d10038e10 auth_method 0 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d180ba030 con 0x7f5d2402c450 2024-04-19T01:05:26.052 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 monclient: dump: 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.053 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d180036e0 con 0x7f5d2402c450 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 msgr2=0x7f5d2402c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 0x7f5d2402c830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d240a72d0 tx=0x7f5d180191e0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 msgr2=0x7f5d2402c830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402c450 0x7f5d2402c830 secure :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5d240a72d0 tx=0x7f5d180191e0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d20ff9640 10 client.184131.objecter ms_handle_connect 0x7f5d10038e10 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.030+0000 7f5d28fb3640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5d10038e10 0x7f5d1003b2d0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7f5d2413dec0 tx=0x7f5d1c007540 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d2aa16640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:26.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 0x7f5d2402d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d2aa16640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2406d140 con 0x7f5d2402cd70 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 0x7f5d2402d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient: get_auth_request con 0x7f5d2402cd70 auth_method 0 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.034+0000 7f5d237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d237fe640 10 monclient(hunting): handle_auth_done global_id 144578 payload 824 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d237fe640 1 -- 172.21.15.43:0/3687746133 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d2409e190 con 0x7f5d2402cd70 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 client.184131.objecter ms_handle_connect 0x7f5d2402cd70 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d237fe640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 0x7f5d2402d180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5d18000c10 tx=0x7f5d180a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 client.184131.objecter resend_mon_ops 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5d180ba040 con 0x7f5d2402cd70 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 monclient: dump: 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.056 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.058 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.058 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.058 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 <== mon.2 v2:172.21.15.43:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9614 (secure 0 0 0) 0x7f5d180b6470 con 0x7f5d2402cd70 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 msgr2=0x7f5d2402d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 0x7f5d2402d180 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5d18000c10 tx=0x7f5d180a8000 comp rx=0 tx=0).stop 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 msgr2=0x7f5d2402d180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d20ff9640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5d2402cd70 0x7f5d2402d180 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5d18000c10 tx=0x7f5d180a8000 comp rx=0 tx=0).stop 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 290668, 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:26.060 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.061 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.062 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.063 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.064 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.065 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.066 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.067 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.068 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.069 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.070 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.071 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.072 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.073 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.074 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.075 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.076 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.077 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.078 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.080 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.081 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.082 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.083 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.084 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.085 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 5 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "mgr": [ 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.086 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:26.087 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 10 librados: watch_flush enter 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 10 librados: watch_flush exit 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 20 client.184131.objecter shutdown clearing up homeless session... 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 10 client.184131.objecter successfully canceled tick 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5d10038e10 msgr2=0x7f5d1003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5d10038e10 0x7f5d1003b2d0 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7f5d2413dec0 tx=0x7f5d1c007540 comp rx=0 tx=0).stop 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 10 monclient: shutdown 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 msgr2=0x7f5d241d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.089 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 0x7f5d241d0720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400c480 tx=0x7f5d1400c940 comp rx=0 tx=0).stop 2024-04-19T01:05:26.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d28fb3640 1 -- 172.21.15.43:0/3687746133 reap_dead start 2024-04-19T01:05:26.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 shutdown_connections 2024-04-19T01:05:26.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 --2- 172.21.15.43:0/3687746133 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5d240a5bb0 0x7f5d241d0720 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.090 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 >> 172.21.15.43:0/3687746133 conn(0x7f5d240a7cb0 msgr2=0x7f5d240a45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 shutdown_connections 2024-04-19T01:05:26.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 -- 172.21.15.43:0/3687746133 wait complete. 2024-04-19T01:05:26.091 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.038+0000 7f5d2aa16640 1 librados: shutdown 2024-04-19T01:05:26.091 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:26.091 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon c 2024-04-19T01:05:26.091 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient: get_monmap_and_config 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient: monmap: 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:26.214998+0000 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:26.214998+0000 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.212 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:26.213 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 Processor -- start 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 -- start start 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient: init 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient: _add_conns ranks=[3,6,5,8,7,1,2,9,0,4,10] 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5880138cd0 0x7f58801390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.218 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): picked mon.noname-g con 0x7f5880138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 0x7f58801395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): picked mon.noname-d con 0x7f58800a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f5880135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): picked mon.noname-h con 0x7f58801358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f58800a4410 con 0x7f5880138cd0 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5880139b30 con 0x7f58801358b0 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f58800a3dc0 con 0x7f58800a5b80 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5887d3b640 10 monclient(hunting): authenticate will time out at 2196.342160s 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 0x7f58801395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.219 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 0x7f58801395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:36630/0 (socket says 172.21.15.43:36630) 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 -- 172.21.15.43:0/3335731400 learned_addr learned my addr 172.21.15.43:0/3335731400 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): get_auth_request con 0x7f58800a5b80 auth_method 0 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58862d8640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f5880135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5885ad7640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5880138cd0 0x7f58801390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58862d8640 10 monclient(hunting): get_auth_request con 0x7f58801358b0 auth_method 0 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58862d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58862d8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58862d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5885ad7640 10 monclient(hunting): get_auth_request con 0x7f5880138cd0 auth_method 0 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5885ad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5885ad7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f5885ad7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient(hunting): handle_auth_done global_id 184119 payload 274 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 -- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 msgr2=0x7f5880135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f5880135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 -- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5880138cd0 msgr2=0x7f58801390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5880138cd0 0x7f58801390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 monclient: found mon.noname-d 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.214+0000 7f58852d6640 1 -- 172.21.15.43:0/3335731400 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58800a40a0 con 0x7f58800a5b80 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58852d6640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 0x7f58801395f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58700098a0 tx=0x7f587001c3e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=7d525d10df7062b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 1 -- 172.21.15.43:0/3335731400 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58700b0070 con 0x7f58800a5b80 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: dump: 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.222 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 1 -- 172.21.15.43:0/3335731400 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58700aa810 con 0x7f58800a5b80 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 5 monclient: authenticate success, global_id 184119 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 1 -- 172.21.15.43:0/3335731400 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f58700aaa40 con 0x7f58800a5b80 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:26.223 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 4 set_mon_vals no callback set 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5884ad5640 10 monclient: dump: 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.224 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: shutdown 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- 172.21.15.43:0/3335731400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 msgr2=0x7f58801395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 0x7f58801395f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58700098a0 tx=0x7f587001c3e0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- 172.21.15.43:0/3335731400 shutdown_connections 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f5880135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f58800a5b80 0x7f58801395f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- 172.21.15.43:0/3335731400 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f5880138cd0 0x7f58801390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- 172.21.15.43:0/3335731400 >> 172.21.15.43:0/3335731400 conn(0x7f58800ab2f0 msgr2=0x7f58800ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- 172.21.15.43:0/3335731400 shutdown_connections 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- 172.21.15.43:0/3335731400 wait complete. 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: monmap: 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.226 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 librados: starting msgr at 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 librados: starting objecter 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 Processor -- start 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- start start 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 librados: setting wanted keys 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 librados: calling monclient init 2024-04-19T01:05:26.227 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: init 2024-04-19T01:05:26.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient: _add_conns ranks=[1,5,4,2,3,7,0,8,6] 2024-04-19T01:05:26.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58800a5b80 0x7f588013ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): picked mon.noname-f con 0x7f58800a5b80 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f588013f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): picked mon.noname-h con 0x7f58801358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5880138cd0 0x7f58801d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f5880138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f58800a2010 con 0x7f58800a5b80 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f58800a1e90 con 0x7f5880138cd0 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5880142aa0 con 0x7f58801358b0 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5887d3b640 10 monclient(hunting): authenticate will time out at 2196.347595s 2024-04-19T01:05:26.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5880138cd0 0x7f58801d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5880138cd0 0x7f58801d0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34970/0 (socket says 172.21.15.43:34970) 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 1 -- 172.21.15.43:0/2642839345 learned_addr learned my addr 172.21.15.43:0/2642839345 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): get_auth_request con 0x7f5880138cd0 auth_method 0 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58862d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f5885ad7640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58800a5b80 0x7f588013ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.218+0000 7f58852d6640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f588013f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 10 monclient(hunting): get_auth_request con 0x7f58801358b0 auth_method 0 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): get_auth_request con 0x7f58800a5b80 auth_method 0 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 10 monclient(hunting): handle_auth_done global_id 184126 payload 824 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 msgr2=0x7f588013f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f588013f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58800a5b80 msgr2=0x7f588013ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58800a5b80 0x7f588013ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 monclient: found mon.noname-c 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58800a3590 con 0x7f5880138cd0 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58800a5b80 0x7f588013ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f58801358b0 0x7f588013f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58862d8640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5880138cd0 0x7f58801d0ba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5878007e40 tx=0x7f587800e4b0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 client.?.objecter ms_handle_connect 0x7f5880138cd0 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5878014890 con 0x7f5880138cd0 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: dump: 2024-04-19T01:05:26.238 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.239 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 5 monclient: authenticate success, global_id 184126 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 monclient: _renew_subs 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5878019070 con 0x7f5880138cd0 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f588013c540 con 0x7f5880138cd0 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5878013380 con 0x7f5880138cd0 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5877fff640 4 set_mon_vals no callback set 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 client.184126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: dump: 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.241 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 monclient: _renew_subs 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f58800a3040 con 0x7f5880138cd0 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 1 librados: init done 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 0x7f58801d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f588013acd0 con 0x7f588013a170 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5878014420 con 0x7f5880138cd0 2024-04-19T01:05:26.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 0x7f58801d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient: get_auth_request con 0x7f588013a170 auth_method 0 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5864038e40 0x7f586403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f5878049b20 con 0x7f5880138cd0 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 client.184126.objecter ms_dispatch 0x7f5880000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 3 client.184126.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 3 client.184126.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 10 monclient(hunting): handle_auth_done global_id 184135 payload 824 2024-04-19T01:05:26.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58800693b0 con 0x7f588013a170 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 20 client.184126.objecter dump_active .. 0 homeless 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 client.184126.objecter ms_handle_connect 0x7f588013a170 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 client.184126.objecter resend_mon_ops 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f5885ad7640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 0x7f58801d10e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f587c00bfd0 tx=0x7f587c00f910 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5864038e40 0x7f586403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 10 monclient: get_auth_request con 0x7f5864038e40 auth_method 0 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f587c019020 con 0x7f588013a170 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:26.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: dump: 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.248 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f587c014670 con 0x7f588013a170 2024-04-19T01:05:26.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 msgr2=0x7f58801d10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 0x7f58801d10e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f587c00bfd0 tx=0x7f587c00f910 comp rx=0 tx=0).stop 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 msgr2=0x7f58801d10e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f588013a170 0x7f58801d10e0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f587c00bfd0 tx=0x7f587c00f910 comp rx=0 tx=0).stop 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58767fc640 10 client.184126.objecter ms_handle_connect 0x7f5864038e40 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.222+0000 7f58852d6640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5864038e40 0x7f586403b300 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7f58700ad000 tx=0x7f58700a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:26.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 0x7f5880133aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f58801d1a30 con 0x7f58801d1620 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 0x7f5880133aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient: get_auth_request con 0x7f58801d1620 auth_method 0 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 10 monclient(hunting): handle_auth_done global_id 184144 payload 824 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 1 -- 172.21.15.43:0/2642839345 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f58800a1330 con 0x7f58801d1620 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5885ad7640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 0x7f5880133aa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f587c01a000 tx=0x7f587c0046d0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 client.184126.objecter ms_handle_connect 0x7f58801d1620 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 client.184126.objecter resend_mon_ops 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f587c019040 con 0x7f58801d1620 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 monclient: dump: 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.252 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.253 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 <== mon.4 v2:172.21.15.43:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f587c01e420 con 0x7f58801d1620 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 msgr2=0x7f5880133aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 0x7f5880133aa0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f587c01a000 tx=0x7f587c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 msgr2=0x7f5880133aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58767fc640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f58801d1620 0x7f5880133aa0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f587c01a000 tx=0x7f587c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 15847, 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:26.255 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 0, 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.256 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:26.257 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.258 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.259 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:26.260 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.261 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:26.262 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.263 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.264 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.264 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.264 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.264 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.268 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.270 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.271 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.272 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:26.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:26.273 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.274 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.276 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.277 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.279 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.280 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:26.281 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:26.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 10 librados: watch_flush enter 2024-04-19T01:05:26.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 10 librados: watch_flush exit 2024-04-19T01:05:26.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 20 client.184126.objecter shutdown clearing up homeless session... 2024-04-19T01:05:26.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 10 client.184126.objecter successfully canceled tick 2024-04-19T01:05:26.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5864038e40 msgr2=0x7f586403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5864038e40 0x7f586403b300 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7f58700ad000 tx=0x7f58700a4f90 comp rx=0 tx=0).stop 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 10 monclient: shutdown 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5880138cd0 msgr2=0x7f58801d0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 --2- 172.21.15.43:0/2642839345 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5880138cd0 0x7f58801d0ba0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5878007e40 tx=0x7f587800e4b0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58862d8640 1 -- 172.21.15.43:0/2642839345 reap_dead start 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f58862d8640 1 -- 172.21.15.43:0/2642839345 reap_dead start 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 shutdown_connections 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 >> 172.21.15.43:0/2642839345 conn(0x7f58800ab2f0 msgr2=0x7f58801368c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 shutdown_connections 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 -- 172.21.15.43:0/2642839345 wait complete. 2024-04-19T01:05:26.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.230+0000 7f5887d3b640 1 librados: shutdown 2024-04-19T01:05:26.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:26.285 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon d 2024-04-19T01:05:26.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.d mon_status 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient: get_monmap_and_config 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient: monmap: 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:26.414911+0000 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:26.414911+0000 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.412 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:26.413 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 Processor -- start 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 -- start start 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient: init 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient: _add_conns ranks=[2,6,10,1,7,0,9,4,8,3,5] 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 0x7f03481390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): picked mon.noname-b con 0x7f0348138cd0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f03481395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): picked mon.noname-d con 0x7f03480a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f03481358b0 0x7f0348135d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): picked mon.noname-f con 0x7f03481358b0 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f03480a4410 con 0x7f03481358b0 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f0348139b30 con 0x7f0348138cd0 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f03480a3dc0 con 0x7f03480a5b80 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034f061640 10 monclient(hunting): authenticate will time out at 2196.542018s 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 0x7f03481390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 0x7f03481390b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47164/0 (socket says 172.21.15.43:47164) 2024-04-19T01:05:26.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 -- 172.21.15.43:0/3360202387 learned_addr learned my addr 172.21.15.43:0/3360202387 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): get_auth_request con 0x7f0348138cd0 auth_method 0 2024-04-19T01:05:26.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 1 --2- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f03481395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): get_auth_request con 0x7f03480a5b80 auth_method 0 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034d5fe640 1 --2- 172.21.15.43:0/3360202387 >> v2:172.21.15.161:3300/0 conn(0x7f03481358b0 0x7f0348135d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient(hunting): handle_auth_done global_id 144587 payload 274 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 -- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 msgr2=0x7f03481395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 --2- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f03481395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 -- 172.21.15.43:0/3360202387 >> v2:172.21.15.161:3300/0 conn(0x7f03481358b0 msgr2=0x7f0348135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 --2- 172.21.15.43:0/3360202387 >> v2:172.21.15.161:3300/0 conn(0x7f03481358b0 0x7f0348135d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 monclient: found mon.noname-b 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034cdfd640 1 -- 172.21.15.43:0/3360202387 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03480a40a0 con 0x7f0348138cd0 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034d5fe640 10 monclient: get_auth_request con 0x7f03481358b0 auth_method 0 2024-04-19T01:05:26.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.414+0000 7f034d5fe640 1 --2- 172.21.15.43:0/3360202387 >> v2:172.21.15.161:3300/0 conn(0x7f03481358b0 0x7f0348135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 --2- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 0x7f03481390b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0338009870 tx=0x7f033801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0489c3efcf3d72a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 1 -- 172.21.15.43:0/3360202387 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03380b0070 con 0x7f0348138cd0 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: dump: 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.426 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.427 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.427 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.427 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.427 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 1 -- 172.21.15.43:0/3360202387 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03380aa810 con 0x7f0348138cd0 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 1 -- 172.21.15.43:0/3360202387 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03380a9300 con 0x7f0348138cd0 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 5 monclient: authenticate success, global_id 144587 2024-04-19T01:05:26.428 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 4 set_mon_vals no callback set 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03477fe640 10 monclient: dump: 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.429 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: shutdown 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 msgr2=0x7f03481390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 0x7f03481390b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0338009870 tx=0x7f033801c3c0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- 172.21.15.43:0/3360202387 shutdown_connections 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- 172.21.15.43:0/3360202387 >> v2:172.21.15.161:3300/0 conn(0x7f03481358b0 0x7f0348135d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f03481395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- 172.21.15.43:0/3360202387 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f0348138cd0 0x7f03481390b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- 172.21.15.43:0/3360202387 >> 172.21.15.43:0/3360202387 conn(0x7f03480ab2f0 msgr2=0x7f03480ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- 172.21.15.43:0/3360202387 shutdown_connections 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- 172.21.15.43:0/3360202387 wait complete. 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: monmap: 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:26.431 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 librados: starting msgr at 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 librados: starting objecter 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 Processor -- start 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- start start 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 librados: setting wanted keys 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 librados: calling monclient init 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: init 2024-04-19T01:05:26.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient: _add_conns ranks=[6,8,0,4,7,5,2,1,3] 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f0348134c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): picked mon.noname-d con 0x7f03480a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f03481358b0 0x7f03481351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): picked mon.noname-e con 0x7f03481358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0348138cd0 0x7f03481d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): picked mon.noname-a con 0x7f0348138cd0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f03480a2010 con 0x7f0348138cd0 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f03480a1e90 con 0x7f03480a5b80 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0348142aa0 con 0x7f03481358b0 2024-04-19T01:05:26.433 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034f061640 10 monclient(hunting): authenticate will time out at 2196.547188s 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f03481358b0 0x7f03481351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f0348134c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0348138cd0 0x7f03481d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f03481358b0 0x7f03481351a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57908/0 (socket says 172.21.15.43:57908) 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 1 -- 172.21.15.43:0/386481733 learned_addr learned my addr 172.21.15.43:0/386481733 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): get_auth_request con 0x7f03481358b0 auth_method 0 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f0348134c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:36648/0 (socket says 172.21.15.43:36648) 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): get_auth_request con 0x7f03480a5b80 auth_method 0 2024-04-19T01:05:26.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): get_auth_request con 0x7f0348138cd0 auth_method 0 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034d5fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient(hunting): handle_auth_done global_id 184137 payload 824 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f03481358b0 msgr2=0x7f03481351a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f03481358b0 0x7f03481351a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0348138cd0 msgr2=0x7f03481d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f0348138cd0 0x7f03481d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 monclient: found mon.noname-d 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034809f180 con 0x7f03480a5b80 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f034cdfd640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f0348134c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03380ab000 tx=0x7f03380191c0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.418+0000 7f03457fa640 10 client.?.objecter ms_handle_connect 0x7f03480a5b80 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03380b0040 con 0x7f03480a5b80 2024-04-19T01:05:26.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: dump: 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.437 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03380044b0 con 0x7f03480a5b80 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 5 monclient: authenticate success, global_id 184137 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 monclient: _renew_subs 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03481356e0 con 0x7f03480a5b80 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f03380aad70 con 0x7f03480a5b80 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 client.184137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0346ffd640 4 set_mon_vals no callback set 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: dump: 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.439 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 monclient: _renew_subs 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f03480a3590 con 0x7f03480a5b80 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 1 librados: init done 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 0x7f03481d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f034809ca10 con 0x7f03481d11f0 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 0x7f03481d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient: get_auth_request con 0x7f03481d11f0 auth_method 0 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.441 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0338004040 con 0x7f03480a5b80 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0334038e10 0x7f033403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 10 monclient(hunting): handle_auth_done global_id 184146 payload 824 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f03380e2af0 con 0x7f03480a5b80 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03480a3040 con 0x7f03481d11f0 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 client.184137.objecter ms_dispatch 0x7f0348000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 3 client.184137.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 3 client.184137.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 20 client.184137.objecter dump_active .. 0 homeless 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 client.184137.objecter ms_handle_connect 0x7f03481d11f0 2024-04-19T01:05:26.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 client.184137.objecter resend_mon_ops 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f0347fff640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 0x7f03481d15d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03480a1b80 tx=0x7f033c011e30 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034d5fe640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0334038e10 0x7f033403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f033c0044a0 con 0x7f03481d11f0 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: dump: 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.443 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034d5fe640 10 monclient: get_auth_request con 0x7f0334038e10 auth_method 0 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f033c00fbf0 con 0x7f03481d11f0 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 msgr2=0x7f03481d15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 0x7f03481d15d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03480a1b80 tx=0x7f033c011e30 comp rx=0 tx=0).stop 2024-04-19T01:05:26.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 msgr2=0x7f03481d15d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03481d11f0 0x7f03481d15d0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03480a1b80 tx=0x7f033c011e30 comp rx=0 tx=0).stop 2024-04-19T01:05:26.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f03457fa640 10 client.184137.objecter ms_handle_connect 0x7f0334038e10 2024-04-19T01:05:26.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.422+0000 7f034d5fe640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0334038e10 0x7f033403b2d0 secure :-1 s=READY pgs=3664 cs=0 l=1 rev1=1 crypto rx=0x7f034000d480 tx=0x7f0340007780 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 0x7f034813c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f034806d140 con 0x7f034813bcf0 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 0x7f034813c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient: get_auth_request con 0x7f034813bcf0 auth_method 0 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 10 monclient(hunting): handle_auth_done global_id 184155 payload 824 2024-04-19T01:05:26.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 1 -- 172.21.15.43:0/386481733 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f03481d1b10 con 0x7f034813bcf0 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 client.184137.objecter ms_handle_connect 0x7f034813bcf0 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f0347fff640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 0x7f034813c0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f034809f810 tx=0x7f033c0046d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 client.184137.objecter resend_mon_ops 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f033c0148b0 con 0x7f034813bcf0 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 monclient: dump: 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.448 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 <== mon.6 v2:172.21.15.43:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f033c01c3a0 con 0x7f034813bcf0 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:26.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 msgr2=0x7f034813c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 0x7f034813c0d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f034809f810 tx=0x7f033c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 msgr2=0x7f034813c0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f03457fa640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f034813bcf0 0x7f034813c0d0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f034809f810 tx=0x7f033c0046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:26.450 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 16047, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.451 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.452 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.453 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.454 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:26.454 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:26.454 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:26.454 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:26.454 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:26.455 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.456 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.457 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:26.458 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.459 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.460 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.461 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.463 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:26.464 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:26.466 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.468 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.472 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:26.473 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:26.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 10 librados: watch_flush enter 2024-04-19T01:05:26.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 10 librados: watch_flush exit 2024-04-19T01:05:26.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 20 client.184137.objecter shutdown clearing up homeless session... 2024-04-19T01:05:26.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 10 client.184137.objecter successfully canceled tick 2024-04-19T01:05:26.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0334038e10 msgr2=0x7f033403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0334038e10 0x7f033403b2d0 secure :-1 s=READY pgs=3664 cs=0 l=1 rev1=1 crypto rx=0x7f034000d480 tx=0x7f0340007780 comp rx=0 tx=0).stop 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 10 monclient: shutdown 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 msgr2=0x7f0348134c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 --2- 172.21.15.43:0/386481733 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f03480a5b80 0x7f0348134c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f03380ab000 tx=0x7f03380191c0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034d5fe640 1 -- 172.21.15.43:0/386481733 reap_dead start 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034d5fe640 1 -- 172.21.15.43:0/386481733 reap_dead start 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 shutdown_connections 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 >> 172.21.15.43:0/386481733 conn(0x7f03480ab2f0 msgr2=0x7f03481374d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 shutdown_connections 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 -- 172.21.15.43:0/386481733 wait complete. 2024-04-19T01:05:26.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.430+0000 7f034f061640 1 librados: shutdown 2024-04-19T01:05:26.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:26.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon e 2024-04-19T01:05:26.477 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.e mon_status 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient: get_monmap_and_config 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient: monmap: 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:26.599246+0000 2024-04-19T01:05:26.596 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:26.599246+0000 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:26.597 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 Processor -- start 2024-04-19T01:05:26.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 -- start start 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient: init 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient: _add_conns ranks=[2,6,8,9,3,1,5,10,0,7,4] 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 0x7fb5ac136f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): picked mon.noname-b con 0x7fb5ac09e450 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb5ac09e830 0x7fb5ac0a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): picked mon.noname-d con 0x7fb5ac09e830 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 0x7fb5ac0a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): picked mon.noname-e con 0x7fb5ac09ec50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ac0735a0 con 0x7fb5ac09e450 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fb5ac0a1bc0 con 0x7fb5ac09e830 2024-04-19T01:05:26.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb5ac0a2000 con 0x7fb5ac09ec50 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b32a0640 10 monclient(hunting): authenticate will time out at 2196.726396s 2024-04-19T01:05:26.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 0x7fb5ac136f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 0x7fb5ac0a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 0x7fb5ac136f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47178/0 (socket says 172.21.15.43:47178) 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 1 -- 172.21.15.43:0/2743572834 learned_addr learned my addr 172.21.15.43:0/2743572834 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 0x7fb5ac0a3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57914/0 (socket says 172.21.15.43:57914) 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): get_auth_request con 0x7fb5ac09e450 auth_method 0 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): get_auth_request con 0x7fb5ac09ec50 auth_method 0 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb5ac09e830 0x7fb5ac0a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): get_auth_request con 0x7fb5ac09e830 auth_method 0 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.603 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b083b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient(hunting): handle_auth_done global_id 144602 payload 274 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 -- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb5ac09e830 msgr2=0x7fb5ac0a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb5ac09e830 0x7fb5ac0a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 -- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 msgr2=0x7fb5ac136f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 0x7fb5ac136f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 monclient: found mon.noname-e 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b183d640 1 -- 172.21.15.43:0/2743572834 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac06c6f0 con 0x7fb5ac09ec50 2024-04-19T01:05:26.604 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.598+0000 7fb5b103c640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 0x7fb5ac136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b183d640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 0x7fb5ac0a3e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009dc0 tx=0x7fb5a401c850 comp rx=0 tx=0).ready entity=mon.8 client_cookie=bc661723335ca934 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 1 -- 172.21.15.43:0/2743572834 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a40ac070 con 0x7fb5ac09ec50 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:26.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: dump: 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.612 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 1 -- 172.21.15.43:0/2743572834 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a40a68b0 con 0x7fb5ac09ec50 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 1 -- 172.21.15.43:0/2743572834 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a40a7420 con 0x7fb5ac09ec50 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb59ffff640 10 monclient: dump: 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.613 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 5 monclient: authenticate success, global_id 144602 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 4 set_mon_vals no callback set 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 10 monclient: shutdown 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 -- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 msgr2=0x7fb5ac0a3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 0x7fb5ac0a3e10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009dc0 tx=0x7fb5a401c850 comp rx=0 tx=0).stop 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 -- 172.21.15.43:0/2743572834 shutdown_connections 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac09ec50 0x7fb5ac0a3e10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fb5ac09e830 0x7fb5ac0a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/2743572834 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fb5ac09e450 0x7fb5ac136f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 -- 172.21.15.43:0/2743572834 >> 172.21.15.43:0/2743572834 conn(0x7fb5ac0a84c0 msgr2=0x7fb5ac0a49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 -- 172.21.15.43:0/2743572834 shutdown_connections 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 -- 172.21.15.43:0/2743572834 wait complete. 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 10 monclient: monmap: 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.616 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 librados: starting msgr at 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 librados: starting objecter 2024-04-19T01:05:26.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 Processor -- start 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 -- start start 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 librados: setting wanted keys 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 1 librados: calling monclient init 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.602+0000 7fb5b32a0640 10 monclient: init 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient: _add_conns ranks=[5,1,0,2,6,4,8,7,3] 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb5ac09e450 0x7fb5ac12a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): picked mon.noname-h con 0x7fb5ac09e450 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac09e830 0x7fb5ac12ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): picked mon.noname-f con 0x7fb5ac09e830 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:26.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 0x7fb5ac0703e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): picked mon.noname-a con 0x7fb5ac09ec50 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac1406e0 con 0x7fb5ac09ec50 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ac140860 con 0x7fb5ac09e830 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fb5ac1409e0 con 0x7fb5ac09e450 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): authenticate will time out at 2196.732424s 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 0x7fb5ac0703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 0x7fb5ac0703e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:57392/0 (socket says 172.21.15.43:57392) 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 -- 172.21.15.43:0/1907935860 learned_addr learned my addr 172.21.15.43:0/1907935860 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): get_auth_request con 0x7fb5ac09ec50 auth_method 0 2024-04-19T01:05:26.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb5ac09e450 0x7fb5ac12a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b083b640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac09e830 0x7fb5ac12ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): get_auth_request con 0x7fb5ac09e450 auth_method 0 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient(hunting): handle_auth_done global_id 184149 payload 824 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb5ac09e450 msgr2=0x7fb5ac12a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fb5ac09e450 0x7fb5ac12a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac09e830 msgr2=0x7fb5ac12ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac09e830 0x7fb5ac12ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 monclient: found mon.noname-a 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ac06bda0 con 0x7fb5ac09ec50 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b083b640 10 monclient: get_auth_request con 0x7fb5ac09e830 auth_method 0 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b083b640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5ac09e830 0x7fb5ac12ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:26.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b183d640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 0x7fb5ac0703e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009dc0 tx=0x7fb5a40195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 client.?.objecter ms_handle_connect 0x7fb5ac09ec50 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a40bf070 con 0x7fb5ac09ec50 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: dump: 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.622 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a40039a0 con 0x7fb5ac09ec50 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 5 monclient: authenticate success, global_id 184149 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient: _renew_subs 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ac06b9e0 con 0x7fb5ac09ec50 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a40ac050 con 0x7fb5ac09ec50 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59f7fe640 4 set_mon_vals no callback set 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 client.184149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:26.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: dump: 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.625 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient: _renew_subs 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ac132490 con 0x7fb5ac09ec50 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 librados: init done 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 0x7fb5ac070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb5ac06d400 con 0x7fb5ac133780 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fb5a40af070 con 0x7fb5ac09ec50 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 0x7fb5ac070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient: get_auth_request con 0x7fb5ac133780 auth_method 0 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb590038e10 0x7fb59003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 10 monclient(hunting): handle_auth_done global_id 144611 payload 824 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5ac069610 con 0x7fb5ac133780 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fb5a40b7070 con 0x7fb5ac09ec50 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 client.184149.objecter ms_dispatch 0x7fb5ac000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 3 client.184149.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 3 client.184149.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 20 client.184149.objecter dump_active .. 0 homeless 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b103c640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 0x7fb5ac070920 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8011600 tx=0x7fb5a8011ac0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 client.184149.objecter ms_handle_connect 0x7fb5ac133780 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 client.184149.objecter resend_mon_ops 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b083b640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb590038e10 0x7fb59003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b083b640 10 monclient: get_auth_request con 0x7fb590038e10 auth_method 0 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a800fba0 con 0x7fb5ac133780 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: dump: 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.628 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5a800f730 con 0x7fb5ac133780 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 msgr2=0x7fb5ac070920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 0x7fb5ac070920 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8011600 tx=0x7fb5a8011ac0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 msgr2=0x7fb5ac070920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac133780 0x7fb5ac070920 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb59dffb640 10 client.184149.objecter ms_handle_connect 0x7fb590038e10 2024-04-19T01:05:26.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.606+0000 7fb5b083b640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb590038e10 0x7fb59003b2d0 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac072550 tx=0x7fb5a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 0x7fb5ac071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fb5ac1376e0 con 0x7fb5ac070e60 2024-04-19T01:05:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 0x7fb5ac071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient: get_auth_request con 0x7fb5ac070e60 auth_method 0 2024-04-19T01:05:26.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 10 monclient(hunting): handle_auth_done global_id 144620 payload 824 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 1 -- 172.21.15.43:0/1907935860 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5ac0717b0 con 0x7fb5ac070e60 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 client.184149.objecter ms_handle_connect 0x7fb5ac070e60 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b103c640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 0x7fb5ac071270 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8008010 tx=0x7fb5a80046d0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 client.184149.objecter resend_mon_ops 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fb5a8006830 con 0x7fb5ac070e60 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 monclient: dump: 2024-04-19T01:05:26.633 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.634 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 <== mon.8 v2:172.21.15.43:3304/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7fb5a80063c0 con 0x7fb5ac070e60 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 msgr2=0x7fb5ac071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 0x7fb5ac071270 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8008010 tx=0x7fb5a80046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 msgr2=0x7fb5ac071270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb59dffb640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fb5ac070e60 0x7fb5ac071270 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:26.635 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 291244, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:26.636 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.637 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.638 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:26.639 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.640 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:26.641 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.642 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.643 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.643 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.643 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.644 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:26.645 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.646 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.647 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.648 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:26.649 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.650 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.651 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.652 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.653 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.653 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:26.653 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.654 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.655 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.656 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.657 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.657 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.657 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.657 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.658 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.659 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.660 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:05:26.661 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.661 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.661 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.661 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:26.661 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:26.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 10 librados: watch_flush enter 2024-04-19T01:05:26.663 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 10 librados: watch_flush exit 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 20 client.184149.objecter shutdown clearing up homeless session... 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 10 client.184149.objecter successfully canceled tick 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb590038e10 msgr2=0x7fb59003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fb590038e10 0x7fb59003b2d0 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac072550 tx=0x7fb5a0008040 comp rx=0 tx=0).stop 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 10 monclient: shutdown 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 msgr2=0x7fb5ac0703e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b183d640 1 -- 172.21.15.43:0/1907935860 reap_dead start 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 0x7fb5ac0703e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009dc0 tx=0x7fb5a40195f0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 shutdown_connections 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 --2- 172.21.15.43:0/1907935860 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fb5ac09ec50 0x7fb5ac0703e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.664 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 >> 172.21.15.43:0/1907935860 conn(0x7fb5ac0a84c0 msgr2=0x7fb5ac130e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 shutdown_connections 2024-04-19T01:05:26.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 -- 172.21.15.43:0/1907935860 wait complete. 2024-04-19T01:05:26.665 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.614+0000 7fb5b32a0640 1 librados: shutdown 2024-04-19T01:05:26.665 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:26.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon f 2024-04-19T01:05:26.665 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.f mon_status 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient: get_monmap_and_config 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient: monmap: 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:26.784889+0000 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:26.784889+0000 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:26.782 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:26.783 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 1 Processor -- start 2024-04-19T01:05:26.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 1 -- start start 2024-04-19T01:05:26.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient: init 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient: _add_conns ranks=[8,0,1,6,4,5,9,7,10,2,3] 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c138cd0 0x7f0a1c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient(hunting): picked mon.noname-e con 0x7f0a1c138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.782+0000 7f0a2254c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a1c0a5b80 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f0a1c1395f0 addr v1:172.21.15.161:6789/0 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a1c1366d0 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a1c0a3dc0 con 0x7f0a1c1395f0 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a1c0a4410 con 0x7f0a1c138cd0 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 10 monclient(hunting): authenticate will time out at 2196.912083s 2024-04-19T01:05:26.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c138cd0 0x7f0a1c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1b7fe640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f0a1c0a5b80 0x7f0a1c1358b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45504/0 (socket says 172.21.15.43:45504) 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1b7fe640 1 -- 172.21.15.43:0/1054960195 learned_addr learned my addr 172.21.15.43:0/1054960195 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): get_auth_request con 0x7f0a1c138cd0 auth_method 0 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 1 -- 172.21.15.43:0/1054960195 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4266974873 0 0) 0x7f0a1c1366d0 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient(hunting): _negotiate 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient(hunting): my global_id is 184158 2024-04-19T01:05:26.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 1 -- 172.21.15.43:0/1054960195 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0a04003530 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_done global_id 144629 payload 274 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1054960195 >> v1:172.21.15.161:6789/0 conn(0x7f0a1c1395f0 legacy=0x7f0a1c135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1054960195 >> v1:172.21.15.43:6789/0 conn(0x7f0a1c0a5b80 legacy=0x7f0a1c1358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 1 monclient: found mon.noname-e 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:26.788 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1054960195 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c0a40a0 con 0x7f0a1c138cd0 2024-04-19T01:05:26.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1bfff640 1 --2- 172.21.15.43:0/1054960195 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c138cd0 0x7f0a1c1390b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0097f0 tx=0x7f0a0c01c340 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a7b4b420f26badc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 1 -- 172.21.15.43:0/1054960195 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a0c0b2070 con 0x7f0a1c138cd0 2024-04-19T01:05:26.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:26.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: dump: 2024-04-19T01:05:26.796 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.797 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 5 monclient: authenticate success, global_id 144629 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 1 -- 172.21.15.43:0/1054960195 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0c0a9070 con 0x7f0a1c138cd0 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 4 set_mon_vals no callback set 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 1 -- 172.21.15.43:0/1054960195 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a0c0abc40 con 0x7f0a1c138cd0 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:26.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a1affd640 10 monclient: dump: 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.799 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 10 monclient: shutdown 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- 172.21.15.43:0/1054960195 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c138cd0 msgr2=0x7f0a1c1390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 --2- 172.21.15.43:0/1054960195 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c138cd0 0x7f0a1c1390b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0097f0 tx=0x7f0a0c01c340 comp rx=0 tx=0).stop 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- 172.21.15.43:0/1054960195 shutdown_connections 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 --2- 172.21.15.43:0/1054960195 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c138cd0 0x7f0a1c1390b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- 172.21.15.43:0/1054960195 >> 172.21.15.43:0/1054960195 conn(0x7f0a1c0ab2f0 msgr2=0x7f0a1c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- 172.21.15.43:0/1054960195 shutdown_connections 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.786+0000 7f0a2254c640 1 -- 172.21.15.43:0/1054960195 wait complete. 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: monmap: 2024-04-19T01:05:26.800 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.801 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 librados: starting msgr at 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 librados: starting objecter 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 Processor -- start 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- start start 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 librados: setting wanted keys 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 librados: calling monclient init 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: init 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: _add_conns ranks=[4,1,8,5,6,0,2,7,3] 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a1c0a5b80 0x7f0a1c1d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): picked mon.noname-c con 0x7f0a1c0a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c138cd0 0x7f0a1c0a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): picked mon.noname-f con 0x7f0a1c138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c1395f0 0x7f0a1c0a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): picked mon.noname-e con 0x7f0a1c1395f0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c0a2010 con 0x7f0a1c138cd0 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f0a1c0a1e90 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f0a1c144ac0 con 0x7f0a1c1395f0 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): authenticate will time out at 2196.917483s 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a1c0a5b80 0x7f0a1c1d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a1c0a5b80 0x7f0a1c1d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:34994/0 (socket says 172.21.15.43:34994) 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1673091873 learned_addr learned my addr 172.21.15.43:0/1673091873 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): get_auth_request con 0x7f0a1c0a5b80 auth_method 0 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c1395f0 0x7f0a1c0a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): get_auth_request con 0x7f0a1c1395f0 auth_method 0 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1b7fe640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c138cd0 0x7f0a1c0a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1b7fe640 10 monclient(hunting): get_auth_request con 0x7f0a1c138cd0 auth_method 0 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient(hunting): handle_auth_done global_id 184153 payload 824 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c1395f0 msgr2=0x7f0a1c0a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c1395f0 0x7f0a1c0a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c138cd0 msgr2=0x7f0a1c0a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c138cd0 0x7f0a1c0a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 monclient: found mon.noname-c 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:26.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1c09f310 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a20ae9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f0a1c1395f0 0x7f0a1c0a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1bfff640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a1c0a5b80 0x7f0a1c1d0a30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c019fd0 tx=0x7f0a0c019140 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 client.?.objecter ms_handle_connect 0x7f0a1c0a5b80 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a0c0b2040 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: dump: 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.806 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0c0bb070 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a0c0ad460 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 10 monclient: dump: 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.808 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a1a7fc640 4 set_mon_vals no callback set 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 5 monclient: authenticate success, global_id 184153 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: _renew_subs 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1c1d1150 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 client.184153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: _renew_subs 2024-04-19T01:05:26.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1c0a15a0 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 librados: init done 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 0x7f0a1c09be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c0a3590 con 0x7f0a1c09ba70 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f0a0c0ab070 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.790+0000 7f0a18ff9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a08038e10 0x7f0a0803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f0a0c0e2e30 con 0x7f0a1c0a5b80 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 client.184153.objecter ms_dispatch 0x7f0a1c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 3 client.184153.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 3 client.184153.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:26.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 0x7f0a1c09be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient: get_auth_request con 0x7f0a1c09ba70 auth_method 0 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 20 client.184153.objecter dump_active .. 0 homeless 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a20ae9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a08038e10 0x7f0a0803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a20ae9640 10 monclient: get_auth_request con 0x7f0a08038e10 auth_method 0 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_done global_id 144613 payload 824 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a1c0775a0 con 0x7f0a1c09ba70 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a1b7fe640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 0x7f0a1c09be50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c136500 tx=0x7f0a1000e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 client.184153.objecter ms_handle_connect 0x7f0a1c09ba70 2024-04-19T01:05:26.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 client.184153.objecter resend_mon_ops 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 client.184153.objecter ms_handle_connect 0x7f0a08038e10 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a20ae9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a08038e10 0x7f0a0803b2d0 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c141f60 tx=0x7f0a14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a10019070 con 0x7f0a1c09ba70 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 monclient: dump: 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.812 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a10005ad0 con 0x7f0a1c09ba70 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 msgr2=0x7f0a1c09be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 0x7f0a1c09be50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c136500 tx=0x7f0a1000e960 comp rx=0 tx=0).stop 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 msgr2=0x7f0a1c09be50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.794+0000 7f0a18ff9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09ba70 0x7f0a1c09be50 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c136500 tx=0x7f0a1000e960 comp rx=0 tx=0).stop 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 0x7f0a1c1320f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1c06d140 con 0x7f0a1c09c390 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 0x7f0a1c1320f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient: get_auth_request con 0x7f0a1c09c390 auth_method 0 2024-04-19T01:05:26.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 10 monclient(hunting): handle_auth_done global_id 144622 payload 824 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 1 -- 172.21.15.43:0/1673091873 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a1c0693b0 con 0x7f0a1c09c390 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 client.184153.objecter ms_handle_connect 0x7f0a1c09c390 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 client.184153.objecter resend_mon_ops 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a1b7fe640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 0x7f0a1c1320f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a10013000 tx=0x7f0a10005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.1 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f0a10019050 con 0x7f0a1c09c390 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 monclient: got monmap 1 from mon.f (according to old e1) 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 monclient: dump: 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.817 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 <== mon.1 v2:172.21.15.161:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f0a1001a040 con 0x7f0a1c09c390 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 msgr2=0x7f0a1c1320f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 0x7f0a1c1320f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a10013000 tx=0x7f0a10005800 comp rx=0 tx=0).stop 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 msgr2=0x7f0a1c1320f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a18ff9640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a1c09c390 0x7f0a1c1320f0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a10013000 tx=0x7f0a10005800 comp rx=0 tx=0).stop 2024-04-19T01:05:26.818 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 291433, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:26.819 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.820 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.821 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.822 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.823 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.824 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.825 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.826 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.827 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.828 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.829 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:26.830 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.831 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.832 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.833 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.834 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:26.835 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.836 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:26.837 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.838 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:26.839 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:26.840 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:26.841 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:26.842 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:26.842 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:26.843 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 10 librados: watch_flush enter 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 10 librados: watch_flush exit 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 20 client.184153.objecter shutdown clearing up homeless session... 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 10 client.184153.objecter successfully canceled tick 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a08038e10 msgr2=0x7f0a0803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f0a08038e10 0x7f0a0803b2d0 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c141f60 tx=0x7f0a14009040 comp rx=0 tx=0).stop 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 10 monclient: shutdown 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a1c0a5b80 msgr2=0x7f0a1c1d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 --2- 172.21.15.43:0/1673091873 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f0a1c0a5b80 0x7f0a1c1d0a30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c019fd0 tx=0x7f0a0c019140 comp rx=0 tx=0).stop 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a20ae9640 1 -- 172.21.15.43:0/1673091873 reap_dead start 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a20ae9640 1 -- 172.21.15.43:0/1673091873 reap_dead start 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 shutdown_connections 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 >> 172.21.15.43:0/1673091873 conn(0x7f0a1c0ab2f0 msgr2=0x7f0a1c130db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.844 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 shutdown_connections 2024-04-19T01:05:26.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 -- 172.21.15.43:0/1673091873 wait complete. 2024-04-19T01:05:26.845 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.802+0000 7f0a2254c640 1 librados: shutdown 2024-04-19T01:05:26.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:26.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon g 2024-04-19T01:05:26.845 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.g mon_status 2024-04-19T01:05:26.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient: get_monmap_and_config 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient: monmap: 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:26.959326+0000 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:26.959326+0000 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.957 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 Processor -- start 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 -- start start 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient: init 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient: _add_conns ranks=[0,5,6,7,9,3,2,10,4,8,1] 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab8c138a50 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c139540 0x7fab8c13dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): picked mon.noname-h con 0x7fab8c139540 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab8c0a5bb0 0x7fab8c140530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.968 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): picked mon.noname-d con 0x7fab8c0a5bb0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab8c0a0d40 con 0x7fab8c138a50 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fab8c0a1020 con 0x7fab8c139540 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fab8c0a0780 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8a59c640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fab8c138a50 0x7fab8c138e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45510/0 (socket says 172.21.15.43:45510) 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8a59c640 1 -- 172.21.15.43:0/3292720138 learned_addr learned my addr 172.21.15.43:0/3292720138 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab909f0640 10 monclient(hunting): authenticate will time out at 2197.086709s 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab8c0a5bb0 0x7fab8c140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): get_auth_request con 0x7fab8c0a5bb0 auth_method 0 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.969 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_done global_id 184173 payload 274 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 1 -- 172.21.15.43:0/3292720138 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c139540 msgr2=0x7fab8c13dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab89d9b640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c139540 0x7fab8c13dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8959a640 1 -- 172.21.15.43:0/3292720138 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833859666 0 0) 0x7fab8c0a0d40 con 0x7fab8c138a50 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c139540 0x7fab8c13dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 1 -- 172.21.15.43:0/3292720138 >> v1:172.21.15.43:6789/0 conn(0x7fab8c138a50 legacy=0x7fab8c138e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 1 monclient: found mon.noname-d 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8ad9d640 1 -- 172.21.15.43:0/3292720138 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8c0a0a60 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab8959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:05:26.971 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab89d9b640 10 monclient: get_auth_request con 0x7fab8c139540 auth_method 0 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.958+0000 7fab89d9b640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c139540 0x7fab8c13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8ad9d640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab8c0a5bb0 0x7fab8c140530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fab84009dc0 tx=0x7fab8401c850 comp rx=0 tx=0).ready entity=mon.6 client_cookie=44cd2fdae2cdf1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 1 -- 172.21.15.43:0/3292720138 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab840b2070 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: dump: 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.972 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 1 -- 172.21.15.43:0/3292720138 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab840a5a50 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 5 monclient: authenticate success, global_id 184173 2024-04-19T01:05:26.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 1 -- 172.21.15.43:0/3292720138 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab840a5c40 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab8959a640 10 monclient: dump: 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.974 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 4 set_mon_vals no callback set 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: shutdown 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- 172.21.15.43:0/3292720138 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab8c0a5bb0 msgr2=0x7fab8c140530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab8c0a5bb0 0x7fab8c140530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fab84009dc0 tx=0x7fab8401c850 comp rx=0 tx=0).stop 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- 172.21.15.43:0/3292720138 shutdown_connections 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab8c0a5bb0 0x7fab8c140530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 --2- 172.21.15.43:0/3292720138 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c139540 0x7fab8c13dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- 172.21.15.43:0/3292720138 >> 172.21.15.43:0/3292720138 conn(0x7fab8c0a7cb0 msgr2=0x7fab8c0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- 172.21.15.43:0/3292720138 shutdown_connections 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- 172.21.15.43:0/3292720138 wait complete. 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: build_initial_monmap 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: monmap: 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:26.976 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 librados: starting msgr at 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 librados: starting objecter 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 Processor -- start 2024-04-19T01:05:26.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- start start 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 librados: setting wanted keys 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 librados: calling monclient init 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: init 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient: _add_conns ranks=[5,1,7,0,8,3,4,6,2] 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c0a5bb0 0x7fab8c13ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.978 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): picked mon.noname-h con 0x7fab8c0a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab8c138a50 0x7fab8c1377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): picked mon.noname-f con 0x7fab8c138a50 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab8c139540 0x7fab8c13c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): picked mon.noname-i con 0x7fab8c139540 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab8c143620 con 0x7fab8c138a50 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fab8c1434a0 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fab8c1437a0 con 0x7fab8c139540 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.962+0000 7fab909f0640 10 monclient(hunting): authenticate will time out at 2197.091686s 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c0a5bb0 0x7fab8c13ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab8c139540 0x7fab8c13c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab8c138a50 0x7fab8c1377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c0a5bb0 0x7fab8c13ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3302/0 says I am v2:172.21.15.43:57386/0 (socket says 172.21.15.43:57386) 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 -- 172.21.15.43:0/2785443438 learned_addr learned my addr 172.21.15.43:0/2785443438 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): get_auth_request con 0x7fab8c0a5bb0 auth_method 0 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): get_auth_request con 0x7fab8c138a50 auth_method 0 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): get_auth_request con 0x7fab8c139540 auth_method 0 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient(hunting): handle_auth_done global_id 75137 payload 824 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab8c139540 msgr2=0x7fab8c13c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab8c139540 0x7fab8c13c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab8c138a50 msgr2=0x7fab8c1377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab8c138a50 0x7fab8c1377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient: _finish_hunting 0 2024-04-19T01:05:26.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 monclient: found mon.noname-h 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 10 monclient: _send_mon_message to mon.noname-h at v2:172.21.15.161:3302/0 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8c121e60 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab8c138a50 0x7fab8c1377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8a59c640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c0a5bb0 0x7fab8c13ecf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fab7c001cf0 tx=0x7fab7c00a430 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 client.?.objecter ms_handle_connect 0x7fab8c0a5bb0 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab7c00ad50 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: got monmap 1 from mon.noname-h (according to old e1) 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: dump: 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.983 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.984 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: _finish_auth 0 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 5 monclient: authenticate success, global_id 75137 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.5 v2:172.21.15.161:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab7c0070d0 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 monclient: _renew_subs 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8c09fc90 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.5 v2:172.21.15.161:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab7c0047f0 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:05:26.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 client.75137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab88d99640 4 set_mon_vals no callback set 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: dump: 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.986 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 monclient: _renew_subs 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 monclient: _send_mon_message to mon.h at v2:172.21.15.161:3302/0 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fab8c1d14a0 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 1 librados: init done 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 0x7fab8c13ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fab8c06d140 con 0x7fab8c13cab0 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 0x7fab8c13ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient: get_auth_request con 0x7fab8c13cab0 auth_method 0 2024-04-19T01:05:26.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.5 v2:172.21.15.161:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fab7c007280 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab6c038e10 0x7fab6c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.5 v2:172.21.15.161:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fab7c051460 con 0x7fab8c0a5bb0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 client.75137.objecter ms_dispatch 0x7fab8c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 3 client.75137.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 3 client.75137.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 20 client.75137.objecter dump_active .. 0 homeless 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab6c038e10 0x7fab6c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 10 monclient(hunting): handle_auth_done global_id 164235 payload 824 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab8c13d3d0 con 0x7fab8c13cab0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 10 monclient: get_auth_request con 0x7fab6c038e10 auth_method 0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 client.75137.objecter ms_handle_connect 0x7fab8c13cab0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 client.75137.objecter resend_mon_ops 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab89d9b640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 0x7fab8c13ce90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab8c13dc20 tx=0x7fab8000d590 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab8000fc60 con 0x7fab8c13cab0 2024-04-19T01:05:26.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 monclient: dump: 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.990 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab8ad9d640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab6c038e10 0x7fab6c03b2d0 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7fab840af000 tx=0x7fab840a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.966+0000 7fab7affd640 10 client.75137.objecter ms_handle_connect 0x7fab6c038e10 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fab8000fc60 con 0x7fab8c13cab0 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 msgr2=0x7fab8c13ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 0x7fab8c13ce90 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fab8c13dc20 tx=0x7fab8000d590 comp rx=0 tx=0).stop 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 msgr2=0x7fab8c13ce90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.991 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.970+0000 7fab7affd640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c13cab0 0x7fab8c13ce90 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab909f0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab909f0640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 0x7fab8c0a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab909f0640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fab8c076190 con 0x7fab8c0a3160 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 0x7fab8c0a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient: get_auth_request con 0x7fab8c0a3160 auth_method 0 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:26.993 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 10 monclient(hunting): handle_auth_done global_id 164244 payload 824 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 1 -- 172.21.15.43:0/2785443438 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fab8c13e9c0 con 0x7fab8c0a3160 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab89d9b640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 0x7fab8c0a3570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e010 tx=0x7fab800046d0 comp rx=0 tx=0).ready entity=mon.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab7affd640 10 client.75137.objecter ms_handle_connect 0x7fab8c0a3160 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.974+0000 7fab7affd640 10 client.75137.objecter resend_mon_ops 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.3 v2:172.21.15.161:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab800066c0 con 0x7fab8c0a3160 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 10 monclient: got monmap 1 from mon.g (according to old e1) 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 10 monclient: dump: 2024-04-19T01:05:26.994 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:26.995 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 <== mon.3 v2:172.21.15.161:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9446 (secure 0 0 0) 0x7fab800066c0 con 0x7fab8c0a3160 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 msgr2=0x7fab8c0a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 0x7fab8c0a3570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e010 tx=0x7fab800046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 msgr2=0x7fab8c0a3570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab7affd640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fab8c0a3160 0x7fab8c0a3570 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fab8000e010 tx=0x7fab800046d0 comp rx=0 tx=0).stop 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:26.996 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "leader", 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 158344, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.997 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:26.998 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:26.999 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:27.000 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.001 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:27.002 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.003 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.004 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.005 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:27.006 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.007 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.008 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.009 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.010 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.011 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.012 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.013 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.014 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.015 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:27.016 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.017 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "osd": [ 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.018 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.019 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 2 2024-04-19T01:05:27.019 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.019 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.019 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.019 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:27.019 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 10 librados: watch_flush enter 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 10 librados: watch_flush exit 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 20 client.75137.objecter shutdown clearing up homeless session... 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 10 client.75137.objecter successfully canceled tick 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab6c038e10 msgr2=0x7fab6c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab6c038e10 0x7fab6c03b2d0 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7fab840af000 tx=0x7fab840a7070 comp rx=0 tx=0).stop 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 10 monclient: shutdown 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.021 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c0a5bb0 msgr2=0x7fab8c13ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 --2- 172.21.15.43:0/2785443438 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab8c0a5bb0 0x7fab8c13ecf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fab7c001cf0 tx=0x7fab7c00a430 comp rx=0 tx=0).stop 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab8ad9d640 1 -- 172.21.15.43:0/2785443438 reap_dead start 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab8ad9d640 1 -- 172.21.15.43:0/2785443438 reap_dead start 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 shutdown_connections 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 >> 172.21.15.43:0/2785443438 conn(0x7fab8c0a7cb0 msgr2=0x7fab8c09da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 shutdown_connections 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 -- 172.21.15.43:0/2785443438 wait complete. 2024-04-19T01:05:27.022 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:26.978+0000 7fab909f0640 1 librados: shutdown 2024-04-19T01:05:27.022 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:27.022 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon h 2024-04-19T01:05:27.023 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.h mon_status 2024-04-19T01:05:27.131 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient: get_monmap_and_config 2024-04-19T01:05:27.220 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient: build_initial_monmap 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient: monmap: 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:27.134709+0000 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:27.134709+0000 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:27.221 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:27.222 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 Processor -- start 2024-04-19T01:05:27.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 -- start start 2024-04-19T01:05:27.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient: init 2024-04-19T01:05:27.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient: _add_conns ranks=[7,5,4,3,8,1,10,9,6,2,0] 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5930138cd0 0x7f59301390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): picked mon.noname-i con 0x7f5930138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f59300a5b80 0x7f59301395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): picked mon.noname-h con 0x7f59300a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 0x7f5930135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): picked mon.noname-c con 0x7f59301358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f59300a4410 con 0x7f59301358b0 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5930139b30 con 0x7f59300a5b80 2024-04-19T01:05:27.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f59300a3dc0 con 0x7f5930138cd0 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f5935049640 10 monclient(hunting): authenticate will time out at 2197.261794s 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 0x7f5930135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 0x7f5930135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:35010/0 (socket says 172.21.15.43:35010) 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 -- 172.21.15.43:0/2262032640 learned_addr learned my addr 172.21.15.43:0/2262032640 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): get_auth_request con 0x7f59301358b0 auth_method 0 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592ed9d640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5930138cd0 0x7f59301390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592e59c640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f59300a5b80 0x7f59301395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592ed9d640 10 monclient(hunting): get_auth_request con 0x7f5930138cd0 auth_method 0 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592e59c640 10 monclient(hunting): get_auth_request con 0x7f59300a5b80 auth_method 0 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient(hunting): handle_auth_done global_id 184162 payload 274 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 -- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5930138cd0 msgr2=0x7f59301390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5930138cd0 0x7f59301390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 -- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f59300a5b80 msgr2=0x7f59301395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f59300a5b80 0x7f59301395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient: _finish_hunting 0 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 monclient: found mon.noname-c 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.134+0000 7f592f59e640 1 -- 172.21.15.43:0/2262032640 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59300a40a0 con 0x7f59301358b0 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592f59e640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 0x7f5930135d30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f59280098a0 tx=0x7f592801c3e0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=a5b2a3cecf78bc58 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 1 -- 172.21.15.43:0/2262032640 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f59280b0070 con 0x7f59301358b0 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: dump: 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.237 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: _finish_auth 0 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: _check_auth_tickets 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 1 -- 172.21.15.43:0/2262032640 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59280aa810 con 0x7f59301358b0 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 1 -- 172.21.15.43:0/2262032640 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f59280a9300 con 0x7f59301358b0 2024-04-19T01:05:27.238 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 5 monclient: authenticate success, global_id 184162 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 4 set_mon_vals no callback set 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f592dd9b640 10 monclient: dump: 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.239 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: shutdown 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.240 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- 172.21.15.43:0/2262032640 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 msgr2=0x7f5930135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 0x7f5930135d30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f59280098a0 tx=0x7f592801c3e0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- 172.21.15.43:0/2262032640 shutdown_connections 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f59300a5b80 0x7f59301395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f59301358b0 0x7f5930135d30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- 172.21.15.43:0/2262032640 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5930138cd0 0x7f59301390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- 172.21.15.43:0/2262032640 >> 172.21.15.43:0/2262032640 conn(0x7f59300ab2f0 msgr2=0x7f59300ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- 172.21.15.43:0/2262032640 shutdown_connections 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- 172.21.15.43:0/2262032640 wait complete. 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: build_initial_monmap 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: monmap: 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:27.241 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.242 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 librados: starting msgr at 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 librados: starting objecter 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 Processor -- start 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- start start 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 librados: setting wanted keys 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 librados: calling monclient init 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: init 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient: _add_conns ranks=[8,7,1,4,0,5,6,2,3] 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f59300a5b80 0x7f59301d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): picked mon.noname-e con 0x7f59300a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f59301358b0 0x7f593013ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): picked mon.noname-i con 0x7f59301358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5930138cd0 0x7f593013f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): picked mon.noname-f con 0x7f5930138cd0 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f59300a2010 con 0x7f5930138cd0 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f59300a1e90 con 0x7f59301358b0 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f5930142aa0 con 0x7f59300a5b80 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.138+0000 7f5935049640 10 monclient(hunting): authenticate will time out at 2197.267922s 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f59300a5b80 0x7f59301d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f59300a5b80 0x7f59301d0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57944/0 (socket says 172.21.15.43:57944) 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 -- 172.21.15.43:0/1065292859 learned_addr learned my addr 172.21.15.43:0/1065292859 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:27.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): get_auth_request con 0x7f59300a5b80 auth_method 0 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f59301358b0 0x7f593013ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5930138cd0 0x7f593013f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 10 monclient(hunting): get_auth_request con 0x7f5930138cd0 auth_method 0 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): get_auth_request con 0x7f59301358b0 auth_method 0 2024-04-19T01:05:27.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient(hunting): handle_auth_done global_id 144647 payload 824 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f59301358b0 msgr2=0x7f593013ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f59301358b0 0x7f593013ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5930138cd0 msgr2=0x7f593013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5930138cd0 0x7f593013f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 monclient: found mon.noname-e 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59300a3590 con 0x7f59300a5b80 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592ed9d640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f59300a5b80 0x7f59301d0c90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f591800d5f0 tx=0x7f591800dab0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 client.?.objecter ms_handle_connect 0x7f59300a5b80 2024-04-19T01:05:27.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5918012a60 con 0x7f59300a5b80 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: dump: 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.247 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 5 monclient: authenticate success, global_id 144647 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5918019070 con 0x7f59300a5b80 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 monclient: _renew_subs 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59300a3040 con 0x7f59300a5b80 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:27.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5918012c40 con 0x7f59300a5b80 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592d59a640 4 set_mon_vals no callback set 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 client.144647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: dump: 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 monclient: _renew_subs 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f593013b410 con 0x7f59300a5b80 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 1 librados: init done 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 0x7f59301334c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f593006d140 con 0x7f5930139f60 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f59180125f0 con 0x7f59300a5b80 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5914038e10 0x7f591403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f591800f040 con 0x7f59300a5b80 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 client.144647.objecter ms_dispatch 0x7f5930000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 3 client.144647.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 3 client.144647.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 0x7f59301334c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 20 client.144647.objecter dump_active .. 0 homeless 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient: get_auth_request con 0x7f5930139f60 auth_method 0 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5914038e10 0x7f591403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 10 monclient: get_auth_request con 0x7f5914038e10 auth_method 0 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 10 monclient(hunting): handle_auth_done global_id 75155 payload 824 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59300693b0 con 0x7f5930139f60 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 client.144647.objecter ms_handle_connect 0x7f5930139f60 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 client.144647.objecter resend_mon_ops 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592e59c640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 0x7f59301334c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5930134ad0 tx=0x7f5920007700 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 client.144647.objecter ms_handle_connect 0x7f5914038e10 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f592f59e640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5914038e10 0x7f591403b2d0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f59280ad000 tx=0x7f59280a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5920014880 con 0x7f5930139f60 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: dump: 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.253 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5920019070 con 0x7f5930139f60 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 msgr2=0x7f59301334c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 0x7f59301334c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5930134ad0 tx=0x7f5920007700 comp rx=0 tx=0).stop 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 msgr2=0x7f59301334c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.142+0000 7f59277fe640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930139f60 0x7f59301334c0 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5930134ad0 tx=0x7f5920007700 comp rx=0 tx=0).stop 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f5935049640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f5935049640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 0x7f5930133e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f5935049640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5930134350 con 0x7f5930133a00 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 0x7f5930133e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient: get_auth_request con 0x7f5930133a00 auth_method 0 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 10 monclient(hunting): handle_auth_done global_id 75164 payload 824 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 1 -- 172.21.15.43:0/1065292859 --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59300008e0 con 0x7f5930133a00 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f59277fe640 10 client.144647.objecter ms_handle_connect 0x7f5930133a00 2024-04-19T01:05:27.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f59277fe640 10 client.144647.objecter resend_mon_ops 2024-04-19T01:05:27.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f592e59c640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 0x7f5930133e10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5920016040 tx=0x7f5920004a50 comp rx=0 tx=0).ready entity=mon.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.5 v2:172.21.15.161:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5920014e40 con 0x7f5930133a00 2024-04-19T01:05:27.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f59277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f59277fe640 10 monclient: got monmap 1 from mon.h (according to old e1) 2024-04-19T01:05:27.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.150+0000 7f59277fe640 10 monclient: dump: 2024-04-19T01:05:27.256 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.257 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 <== mon.5 v2:172.21.15.161:3302/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9274 (secure 0 0 0) 0x7f59200149d0 con 0x7f5930133a00 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 msgr2=0x7f5930133e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 0x7f5930133e10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5920016040 tx=0x7f5920004a50 comp rx=0 tx=0).stop 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 msgr2=0x7f5930133e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f59277fe640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5930133a00 0x7f5930133e10 secure :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5920016040 tx=0x7f5920004a50 comp rx=0 tx=0).stop 2024-04-19T01:05:27.258 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 706770, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:27.259 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 1, 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.260 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.261 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:27.262 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.263 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.264 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:27.265 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.266 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:27.267 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.268 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.269 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.270 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.271 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.272 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.273 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.274 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.275 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.276 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.277 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:27.278 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.279 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.280 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:27.280 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.280 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:27.281 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 10 librados: watch_flush enter 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 10 librados: watch_flush exit 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 20 client.144647.objecter shutdown clearing up homeless session... 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 10 client.144647.objecter successfully canceled tick 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5914038e10 msgr2=0x7f591403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5914038e10 0x7f591403b2d0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f59280ad000 tx=0x7f59280a6000 comp rx=0 tx=0).stop 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 10 monclient: shutdown 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f59300a5b80 msgr2=0x7f59301d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 --2- 172.21.15.43:0/1065292859 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f59300a5b80 0x7f59301d0c90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f591800d5f0 tx=0x7f591800dab0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.283 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f592f59e640 1 -- 172.21.15.43:0/1065292859 reap_dead start 2024-04-19T01:05:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f592f59e640 1 -- 172.21.15.43:0/1065292859 reap_dead start 2024-04-19T01:05:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 shutdown_connections 2024-04-19T01:05:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 >> 172.21.15.43:0/1065292859 conn(0x7f59300ab2f0 msgr2=0x7f59300a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 shutdown_connections 2024-04-19T01:05:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 -- 172.21.15.43:0/1065292859 wait complete. 2024-04-19T01:05:27.284 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.154+0000 7f5935049640 1 librados: shutdown 2024-04-19T01:05:27.284 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:27.284 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 9 for mon i 2024-04-19T01:05:27.285 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.i mon_status 2024-04-19T01:05:27.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient: get_monmap_and_config 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient: build_initial_monmap 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient: monmap: 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:27.418571+0000 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:27.418571+0000 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:27.416 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:27.417 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:27.417 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:27.417 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:27.417 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:27.417 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 Processor -- start 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 -- start start 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient: init 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient: _add_conns ranks=[0,3,2,7,6,9,8,5,10,1,4] 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18d4138cd0 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d40a5b80 0x7f18d413e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): picked mon.noname-g con 0x7f18d40a5b80 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18d41397c0 0x7f18d41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): picked mon.noname-b con 0x7f18d41397c0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18d40a3dc0 con 0x7f18d4138cd0 2024-04-19T01:05:27.419 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f18d40a4410 con 0x7f18d41397c0 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f18d40a40a0 con 0x7f18d40a5b80 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18da66d640 10 monclient(hunting): authenticate will time out at 2197.545705s 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d3fff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f18d4138cd0 0x7f18d41390b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:45524/0 (socket says 172.21.15.43:45524) 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d3fff640 1 -- 172.21.15.43:0/188607647 learned_addr learned my addr 172.21.15.43:0/188607647 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18d41397c0 0x7f18d41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): get_auth_request con 0x7f18d41397c0 auth_method 0 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d37fe640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d40a5b80 0x7f18d413e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d37fe640 10 monclient(hunting): get_auth_request con 0x7f18d40a5b80 auth_method 0 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_done global_id 144605 payload 274 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d2ffd640 1 -- 172.21.15.43:0/188607647 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603373760 0 0) 0x7f18d40a3dc0 con 0x7f18d4138cd0 2024-04-19T01:05:27.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 1 -- 172.21.15.43:0/188607647 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d40a5b80 msgr2=0x7f18d413e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d40a5b80 0x7f18d413e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 1 -- 172.21.15.43:0/188607647 >> v1:172.21.15.43:6789/0 conn(0x7f18d4138cd0 legacy=0x7f18d41390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient: _finish_hunting 0 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 1 monclient: found mon.noname-b 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d8c0a640 1 -- 172.21.15.43:0/188607647 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d40a2ae0 con 0x7f18d41397c0 2024-04-19T01:05:27.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.418+0000 7f18d2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18d41397c0 0x7f18d41407c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18c40097f0 tx=0x7f18c401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=663ec2ba2bcb28e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 1 -- 172.21.15.43:0/188607647 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18c40b0070 con 0x7f18d41397c0 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: dump: 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.430 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.431 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: _finish_auth 0 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 1 -- 172.21.15.43:0/188607647 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c40aa810 con 0x7f18d41397c0 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 5 monclient: authenticate success, global_id 144605 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 1 -- 172.21.15.43:0/188607647 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18c40a9300 con 0x7f18d41397c0 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 4 set_mon_vals no callback set 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d2ffd640 10 monclient: dump: 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.432 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.433 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: shutdown 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- 172.21.15.43:0/188607647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18d41397c0 msgr2=0x7f18d41407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18d41397c0 0x7f18d41407c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f18c40097f0 tx=0x7f18c401c3a0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- 172.21.15.43:0/188607647 shutdown_connections 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f18d41397c0 0x7f18d41407c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 --2- 172.21.15.43:0/188607647 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f18d40a5b80 0x7f18d413e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.434 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- 172.21.15.43:0/188607647 >> 172.21.15.43:0/188607647 conn(0x7f18d40ab2f0 msgr2=0x7f18d40ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- 172.21.15.43:0/188607647 shutdown_connections 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- 172.21.15.43:0/188607647 wait complete. 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: build_initial_monmap 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: monmap: 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:27.435 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 librados: starting msgr at 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 librados: starting objecter 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 Processor -- start 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- start start 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 librados: setting wanted keys 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 librados: calling monclient init 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: init 2024-04-19T01:05:27.436 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient: _add_conns ranks=[5,6,8,1,2,4,7,0,3] 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18d40a5b80 0x7f18d40a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): picked mon.noname-h con 0x7f18d40a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18d4138cd0 0x7f18d4135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): picked mon.noname-d con 0x7f18d4138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18d41397c0 0x7f18d41361b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): picked mon.noname-e con 0x7f18d41397c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f18d4143730 con 0x7f18d40a5b80 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f18d40a20a0 con 0x7f18d4138cd0 2024-04-19T01:05:27.437 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f18d41438b0 con 0x7f18d41397c0 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18da66d640 10 monclient(hunting): authenticate will time out at 2197.551054s 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18d41397c0 0x7f18d41361b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18d4138cd0 0x7f18d4135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18d41397c0 0x7f18d41361b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57950/0 (socket says 172.21.15.43:57950) 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 1 -- 172.21.15.43:0/218732989 learned_addr learned my addr 172.21.15.43:0/218732989 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): get_auth_request con 0x7f18d4138cd0 auth_method 0 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): get_auth_request con 0x7f18d41397c0 auth_method 0 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.438 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d3fff640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18d40a5b80 0x7f18d40a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d3fff640 10 monclient(hunting): get_auth_request con 0x7f18d40a5b80 auth_method 0 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient(hunting): handle_auth_done global_id 184182 payload 824 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18d41397c0 msgr2=0x7f18d41361b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.439 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18d41397c0 0x7f18d41361b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18d40a5b80 msgr2=0x7f18d40a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f18d40a5b80 0x7f18d40a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient: _finish_hunting 0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 monclient: found mon.noname-d 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d40a3590 con 0x7f18d4138cd0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d8c0a640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f18d41397c0 0x7f18d41361b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d37fe640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18d4138cd0 0x7f18d4135c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00d5a0 tx=0x7f18bc00da60 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d0ff9640 10 client.?.objecter ms_handle_connect 0x7f18d4138cd0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.422+0000 7f18d0ff9640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18bc019070 con 0x7f18d4138cd0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:27.440 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: dump: 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.441 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: _finish_auth 0 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: _check_auth_tickets 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc004240 con 0x7f18d4138cd0 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18bc0043f0 con 0x7f18d4138cd0 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 5 monclient: authenticate success, global_id 184182 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 monclient: _renew_subs 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d40a3040 con 0x7f18d4138cd0 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d27fc640 4 set_mon_vals no callback set 2024-04-19T01:05:27.442 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: dump: 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.443 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 client.184182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 monclient: _renew_subs 2024-04-19T01:05:27.444 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d40a07e0 con 0x7f18d4138cd0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 1 librados: init done 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 0x7f18d409ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f18d406d140 con 0x7f18d409c8f0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f18bc003620 con 0x7f18d4138cd0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f18b4038e40 0x7f18b403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 0x7f18d409ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f18bc00f040 con 0x7f18d4138cd0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 client.184182.objecter ms_dispatch 0x7f18d4000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 3 client.184182.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 3 client.184182.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient: get_auth_request con 0x7f18d409c8f0 auth_method 0 2024-04-19T01:05:27.445 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 20 client.184182.objecter dump_active .. 0 homeless 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d8c0a640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f18b4038e40 0x7f18b403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d8c0a640 10 monclient: get_auth_request con 0x7f18b4038e40 auth_method 0 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 10 monclient(hunting): handle_auth_done global_id 184147 payload 824 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18d40693b0 con 0x7f18d409c8f0 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 client.184182.objecter ms_handle_connect 0x7f18d409c8f0 2024-04-19T01:05:27.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 client.184182.objecter resend_mon_ops 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d3fff640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 0x7f18d409ccd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000f90 tx=0x7f18cc00c4b0 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 client.184182.objecter ms_handle_connect 0x7f18b4038e40 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d8c0a640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f18b4038e40 0x7f18b403b300 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f18c401c3e0 tx=0x7f18c40a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18cc005df0 con 0x7f18d409c8f0 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: dump: 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.447 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18cc018070 con 0x7f18d409c8f0 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 msgr2=0x7f18d409ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.448 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 0x7f18d409ccd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000f90 tx=0x7f18cc00c4b0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 msgr2=0x7f18d409ccd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:27.449 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.426+0000 7f18d0ff9640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409c8f0 0x7f18d409ccd0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18cc000f90 tx=0x7f18cc00c4b0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-19T01:05:27.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 0x7f18d413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f18d4068240 con 0x7f18d409d210 2024-04-19T01:05:27.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 0x7f18d413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient: get_auth_request con 0x7f18d409d210 auth_method 0 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 10 monclient(hunting): handle_auth_done global_id 184156 payload 824 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 1 -- 172.21.15.43:0/218732989 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18d41366f0 con 0x7f18d409d210 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d3fff640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 0x7f18d413bcf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f18cc017000 tx=0x7f18cc001260 comp rx=0 tx=0).ready entity=mon.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 client.184182.objecter ms_handle_connect 0x7f18d409d210 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 client.184182.objecter resend_mon_ops 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.7 v2:172.21.15.161:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f18cc00ce30 con 0x7f18d409d210 2024-04-19T01:05:27.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 monclient: got monmap 1 from mon.i (according to old e1) 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 monclient: dump: 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.452 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 <== mon.7 v2:172.21.15.161:3303/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9273 (secure 0 0 0) 0x7f18cc00ce30 con 0x7f18d409d210 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 msgr2=0x7f18d413bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 0x7f18d413bcf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f18cc017000 tx=0x7f18cc001260 comp rx=0 tx=0).stop 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 msgr2=0x7f18d413bcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18d0ff9640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f18d409d210 0x7f18d413bcf0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f18cc017000 tx=0x7f18cc001260 comp rx=0 tx=0).stop 2024-04-19T01:05:27.453 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:27.454 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:27.454 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:27.454 INFO:teuthology.orchestra.run.smithi043.stdout: "state": "peon", 2024-04-19T01:05:27.454 INFO:teuthology.orchestra.run.smithi043.stdout: "election_epoch": 132, 2024-04-19T01:05:27.454 INFO:teuthology.orchestra.run.smithi043.stdout: "uptime": 17051, 2024-04-19T01:05:27.454 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum": [ 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 0, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 1, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 2, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 3, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 4, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 5, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 6, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 7, 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: 8 2024-04-19T01:05:27.455 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_age": 2, 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "required_con": "2450521708007866372", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "required_mon": [ 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.456 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_con": "4540701547738038271", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "quorum_mon": [ 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.457 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "outside_quorum": [], 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "extra_probe_peers": [], 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "sync_provider": [], 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "monmap": { 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "epoch": 1, 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "fsid": "1280a06a-2032-4fc0-a20f-543d29f598e9", 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "modified": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "created": "2024-04-19T00:43:51.655686Z", 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release": 19, 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "min_mon_release_name": "squid", 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "election_strategy": 3, 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "disallowed_leaders: ": "", 2024-04-19T01:05:27.458 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false, 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "tiebreaker_mon": "", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "removed_ranks: ": "", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "features": { 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "persistent": [ 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "kraken", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "luminous", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "mimic", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "osdmap-prune", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "nautilus", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "octopus", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "pacific", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "elector-pinging", 2024-04-19T01:05:27.459 INFO:teuthology.orchestra.run.smithi043.stdout: "quincy", 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "reef", 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "squid" 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "optional": [] 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "mons": [ 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 0, 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "a", 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.460 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3300", 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789", 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6789/0", 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6789/0", 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.461 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 1, 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "f", 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3300", 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.462 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789", 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6789/0", 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6789/0", 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 2, 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "b", 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.463 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3301", 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790", 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.464 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6790/0", 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6790/0", 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 3, 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "g", 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.465 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3301", 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790", 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6790/0", 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6790/0", 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.466 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 4, 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "c", 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3302", 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791", 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.467 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6791/0", 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6791/0", 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 5, 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "h", 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.468 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3302", 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791", 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6791/0", 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6791/0", 2024-04-19T01:05:27.469 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 6, 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "d", 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3303", 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.470 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792", 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6792/0", 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6792/0", 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.471 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 7, 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "i", 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:3303", 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792", 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.472 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.161:6792/0", 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.161:6792/0", 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "rank": 8, 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "name": "e", 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addrs": { 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addrvec": [ 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v2", 2024-04-19T01:05:27.473 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:3304", 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "type": "v1", 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793", 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "nonce": 0 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "addr": "172.21.15.43:6793/0", 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "public_addr": "172.21.15.43:6793/0", 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "priority": 0, 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "weight": 0, 2024-04-19T01:05:27.474 INFO:teuthology.orchestra.run.smithi043.stdout: "crush_location": "{}" 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: "feature_map": { 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: "mon": [ 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: ], 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: "client": [ 2024-04-19T01:05:27.475 INFO:teuthology.orchestra.run.smithi043.stdout: { 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: "features": "0x3f03cffffffdffff", 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: "release": "luminous", 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: "num": 1 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: } 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: ] 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: }, 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout: "stretch_mode": false 2024-04-19T01:05:27.476 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 10 librados: watch_flush enter 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 10 librados: watch_flush exit 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 20 client.184182.objecter shutdown clearing up homeless session... 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.434+0000 7f18da66d640 10 client.184182.objecter successfully canceled tick 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f18b4038e40 msgr2=0x7f18b403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f18b4038e40 0x7f18b403b300 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f18c401c3e0 tx=0x7f18c40a6000 comp rx=0 tx=0).stop 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 10 monclient: shutdown 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18d4138cd0 msgr2=0x7f18d4135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 --2- 172.21.15.43:0/218732989 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f18d4138cd0 0x7f18d4135c70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00d5a0 tx=0x7f18bc00da60 comp rx=0 tx=0).stop 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18d8c0a640 1 -- 172.21.15.43:0/218732989 reap_dead start 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18d8c0a640 1 -- 172.21.15.43:0/218732989 reap_dead start 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 shutdown_connections 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 >> 172.21.15.43:0/218732989 conn(0x7f18d40ab2f0 msgr2=0x7f18d41377e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 shutdown_connections 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 -- 172.21.15.43:0/218732989 wait complete. 2024-04-19T01:05:27.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.438+0000 7f18da66d640 1 librados: shutdown 2024-04-19T01:05:27.479 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 9 2024-04-19T01:05:27.479 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-19T01:05:27.480 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.586+0000 7fdb580f5640 10 monclient: get_monmap_and_config 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.586+0000 7fdb580f5640 10 monclient: build_initial_monmap 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.586+0000 7fdb580f5640 10 monclient: monmap: 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:27.590038+0000 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:27.590038+0000 2024-04-19T01:05:27.587 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:27.588 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:27.589 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:27.589 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:27.589 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 Processor -- start 2024-04-19T01:05:27.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 -- start start 2024-04-19T01:05:27.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient: init 2024-04-19T01:05:27.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:27.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient: _add_conns ranks=[3,2,6,7,10,8,0,4,5,1,9] 2024-04-19T01:05:27.599 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb50076190 0x7fdb500a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): picked mon.noname-g con 0x7fdb50076190 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb500a2e20 0x7fdb500a3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): picked mon.noname-b con 0x7fdb500a2e20 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdb50138a50 0x7fdb50138eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): picked mon.noname-d con 0x7fdb50138a50 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fdb500a3740 con 0x7fdb500a2e20 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fdb50076570 con 0x7fdb50076190 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fdb500a11d0 con 0x7fdb50138a50 2024-04-19T01:05:27.600 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb580f5640 10 monclient(hunting): authenticate will time out at 2197.717229s 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb500a2e20 0x7fdb500a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdb50138a50 0x7fdb50138eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb500a2e20 0x7fdb500a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47194/0 (socket says 172.21.15.43:47194) 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 1 -- 172.21.15.43:0/3163294235 learned_addr learned my addr 172.21.15.43:0/3163294235 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): get_auth_request con 0x7fdb500a2e20 auth_method 0 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): get_auth_request con 0x7fdb50138a50 auth_method 0 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.601 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55e91640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb50076190 0x7fdb500a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55690640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55e91640 10 monclient(hunting): get_auth_request con 0x7fdb50076190 auth_method 0 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55e91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55e91640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55e91640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient(hunting): handle_auth_done global_id 184191 payload 274 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 -- 172.21.15.43:0/3163294235 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb50076190 msgr2=0x7fdb500a28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:27.602 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb50076190 0x7fdb500a28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 -- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb500a2e20 msgr2=0x7fdb500a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb500a2e20 0x7fdb500a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient: _finish_hunting 0 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 monclient: found mon.noname-d 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 -- 172.21.15.43:0/3163294235 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb500a14b0 con 0x7fdb50138a50 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb55e91640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb50076190 0x7fdb500a28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.590+0000 7fdb56692640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdb50138a50 0x7fdb50138eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009820 tx=0x7fdb4c01c410 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4c212d48fbc4c6ca server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.610 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 1 -- 172.21.15.43:0/3163294235 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb4c0b0070 con 0x7fdb50138a50 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: dump: 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.611 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: _finish_auth 0 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: _check_auth_tickets 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 5 monclient: authenticate success, global_id 184191 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 1 -- 172.21.15.43:0/3163294235 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c0aa810 con 0x7fdb50138a50 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:27.612 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 1 -- 172.21.15.43:0/3163294235 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb4c0a9300 con 0x7fdb50138a50 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 4 set_mon_vals no callback set 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb54e8f640 10 monclient: dump: 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.613 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: shutdown 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.614 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdb50138a50 msgr2=0x7fdb50138eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdb50138a50 0x7fdb50138eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009820 tx=0x7fdb4c01c410 comp rx=0 tx=0).stop 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3163294235 shutdown_connections 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fdb50138a50 0x7fdb50138eb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fdb500a2e20 0x7fdb500a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3163294235 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fdb50076190 0x7fdb500a28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3163294235 >> 172.21.15.43:0/3163294235 conn(0x7fdb500a7fb0 msgr2=0x7fdb500a83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3163294235 shutdown_connections 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3163294235 wait complete. 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: build_initial_monmap 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: monmap: 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:27.615 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 librados: starting msgr at 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 librados: starting objecter 2024-04-19T01:05:27.616 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 Processor -- start 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- start start 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 librados: setting wanted keys 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 librados: calling monclient init 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: init 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: _add_conns ranks=[5,1,8,7,0,4,2,6,3] 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb50076190 0x7fdb501d0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): picked mon.noname-h con 0x7fdb50076190 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb500a2e20 0x7fdb501d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): picked mon.noname-f con 0x7fdb500a2e20 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 0x7fdb501d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): picked mon.noname-e con 0x7fdb50138a50 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdb50142760 con 0x7fdb500a2e20 2024-04-19T01:05:27.617 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fdb501425e0 con 0x7fdb50076190 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fdb501428e0 con 0x7fdb50138a50 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient(hunting): authenticate will time out at 2197.722529s 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 0x7fdb501d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 0x7fdb501d13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57962/0 (socket says 172.21.15.43:57962) 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 -- 172.21.15.43:0/3414549631 learned_addr learned my addr 172.21.15.43:0/3414549631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): get_auth_request con 0x7fdb50138a50 auth_method 0 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:27.618 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient(hunting): handle_auth_done global_id 144665 payload 824 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 -- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb50076190 msgr2=0x7fdb501d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb55e91640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb50076190 0x7fdb501d0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb55690640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb500a2e20 0x7fdb501d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb50076190 0x7fdb501d0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 -- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb500a2e20 msgr2=0x7fdb501d0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb500a2e20 0x7fdb501d0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient: _finish_hunting 0 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 monclient: found mon.noname-e 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 -- 172.21.15.43:0/3414549631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb500a04c0 con 0x7fdb50138a50 2024-04-19T01:05:27.619 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb56692640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 0x7fdb501d13f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009820 tx=0x7fdb4c0191b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 client.?.objecter ms_handle_connect 0x7fdb50138a50 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb4c0b0070 con 0x7fdb50138a50 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: dump: 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.620 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c0b1040 con 0x7fdb50138a50 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 5 monclient: authenticate success, global_id 144665 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: _renew_subs 2024-04-19T01:05:27.621 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb50132c20 con 0x7fdb50138a50 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fdb4c0aae10 con 0x7fdb50138a50 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb47fff640 4 set_mon_vals no callback set 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 client.144665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb467fc640 10 monclient: dump: 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:27.622 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: _renew_subs 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.623 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb501357b0 con 0x7fdb50138a50 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.594+0000 7fdb580f5640 1 librados: init done 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fdb4c004030 con 0x7fdb50138a50 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb580f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb580f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb580f5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb500008e0 con 0x7fdb50138a50 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb34038e10 0x7fdb3403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fdb4c0e2dd0 con 0x7fdb50138a50 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 10 client.144665.objecter ms_dispatch 0x7fdb50000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:27.624 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 3 client.144665.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 3 client.144665.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 20 client.144665.objecter dump_active .. 0 homeless 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb55e91640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb34038e10 0x7fdb3403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb55e91640 10 monclient: get_auth_request con 0x7fdb34038e10 auth_method 0 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb467fc640 10 client.144665.objecter ms_handle_connect 0x7fdb34038e10 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.598+0000 7fdb55e91640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb34038e10 0x7fdb3403b2d0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009f70 tx=0x7fdb4800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:27.625 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.602+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdb500a04c0 con 0x7fdb50138a50 2024-04-19T01:05:27.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.602+0000 7fdb467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:27.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.602+0000 7fdb467fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:27.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.822+0000 7fdb580f5640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-19T01:05:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.822+0000 7fdb580f5640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-19T01:05:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.822+0000 7fdb580f5640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:27.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.822+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fdb501348c0 con 0x7fdb50138a50 2024-04-19T01:05:27.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.826+0000 7fdb467fc640 1 -- 172.21.15.43:0/3414549631 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fdb4c016090 con 0x7fdb50138a50 2024-04-19T01:05:27.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.826+0000 7fdb467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-19T01:05:27.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.826+0000 7fdb467fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:27.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=12,logm=47} crc {auth=3429521128,config=2586702516,health=2883617892,kv=4196696685,logm=1761868381}) 2024-04-19T01:05:27.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.829 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=12,logm=45} crc {auth=2087065151,config=2586702516,health=1388669092,kv=4196696685,logm=4055993762}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=627241025,config=2586702516,health=2722854062,kv=4196696685,logm=3136440680}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {auth=29,config=2,health=11,kv=11,logm=47} crc {auth=613543750,config=2586702516,health=1152775331,kv=414189483,logm=1554512494}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {auth=28,config=2,health=12,kv=11,logm=47} crc {auth=876841631,config=2586702516,health=1388669092,kv=167605683,logm=709834894}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=3346618769,config=2586702516,health=1388669092,kv=4196696685,logm=3382315403}) 2024-04-19T01:05:27.830 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=1505869813,config=2586702516,health=1388669092,kv=4196696685,logm=4051391546}) 2024-04-19T01:05:27.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {auth=29,config=1,health=12,kv=12,logm=46} crc {auth=2087065151,config=1228700967,health=1388669092,kv=4196696685,logm=2251257361}) 2024-04-19T01:05:27.831 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.830+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {auth=30,config=2,health=11,kv=12,logm=45} crc {auth=2455919262,config=2586702516,health=197628583,kv=4196696685,logm=2246717498}) 2024-04-19T01:05:27.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 10 librados: watch_flush enter 2024-04-19T01:05:27.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 10 librados: watch_flush exit 2024-04-19T01:05:27.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 20 client.144665.objecter shutdown clearing up homeless session... 2024-04-19T01:05:27.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 10 client.144665.objecter successfully canceled tick 2024-04-19T01:05:27.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb34038e10 msgr2=0x7fdb3403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb34038e10 0x7fdb3403b2d0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7fdb48009f70 tx=0x7fdb4800b040 comp rx=0 tx=0).stop 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 10 monclient: shutdown 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 msgr2=0x7fdb501d13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 0x7fdb501d13f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009820 tx=0x7fdb4c0191b0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 shutdown_connections 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fdb34038e10 0x7fdb3403b2d0 unknown :-1 s=CLOSED pgs=3670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fdb50138a50 0x7fdb501d13f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdb500a2e20 0x7fdb501d0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 --2- 172.21.15.43:0/3414549631 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fdb50076190 0x7fdb501d0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 >> 172.21.15.43:0/3414549631 conn(0x7fdb500a7fb0 msgr2=0x7fdb500a6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:27.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 shutdown_connections 2024-04-19T01:05:27.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 -- 172.21.15.43:0/3414549631 wait complete. 2024-04-19T01:05:27.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:27.830+0000 7fdb580f5640 1 librados: shutdown 2024-04-19T01:05:27.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.836 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2748946933}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2601035439}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3963808211}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1328459635}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4099444057}) 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.837 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.834+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1018632817}) 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=107617522}) 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3659946089}) 2024-04-19T01:05:27.838 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.838+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2751527369}) 2024-04-19T01:05:27.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091114670}) 2024-04-19T01:05:27.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.843 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159251020}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=473775471}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2173523182}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1712259604}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3179067062}) 2024-04-19T01:05:27.844 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3593417876}) 2024-04-19T01:05:27.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2093619354}) 2024-04-19T01:05:27.845 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.842+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1959143076}) 2024-04-19T01:05:27.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-19T01:05:27.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4278854235}) 2024-04-19T01:05:27.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.864 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128474238}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4155248149}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3563586538}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4283105990}) 2024-04-19T01:05:27.865 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4089735271}) 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=35786659}) 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1521639615}) 2024-04-19T01:05:27.866 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.858+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1985513065}) 2024-04-19T01:05:27.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354994368}) 2024-04-19T01:05:27.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.871 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428817121}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451516513}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2753849381}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2317650860}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2176360986}) 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.872 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3964967365}) 2024-04-19T01:05:27.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2022427758}) 2024-04-19T01:05:27.873 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.870+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=652101380}) 2024-04-19T01:05:27.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1741468414}) 2024-04-19T01:05:27.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.875 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1303430112}) 2024-04-19T01:05:27.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017062599}) 2024-04-19T01:05:27.876 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.877 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3461561328}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2500451561}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2371224330}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=605234010}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1231249971}) 2024-04-19T01:05:27.878 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.874+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1649711544}) 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562173790}) 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839716031}) 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.885 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828261068}) 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=679094485}) 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2572548432}) 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.886 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=515453739}) 2024-04-19T01:05:27.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=476207398}) 2024-04-19T01:05:27.887 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2605835916}) 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.878+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2045043961}) 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1938524336}) 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.888 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902232472}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263023449}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1690787550}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=474580828}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=903726979}) 2024-04-19T01:05:27.889 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2602666274}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=41619454}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3638718462}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.882+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019954531}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081548565}) 2024-04-19T01:05:27.890 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=287273420}) 2024-04-19T01:05:27.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3010265669}) 2024-04-19T01:05:27.891 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2274629967}) 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1622590799}) 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2639543542}) 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.892 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=522227825}) 2024-04-19T01:05:27.893 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=301006900}) 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3231079335}) 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=612283065}) 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.902 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219385774}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1679159305}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=161687122}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1920769455}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=132160743}) 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.903 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2225833154}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.886+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1433658716}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186794382}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809625089}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3567641141}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=252763563}) 2024-04-19T01:05:27.904 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1503759210}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=646948690}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2112070651}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=265753252}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.890+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1257411173}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.905 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1067829004}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3572928612}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617736933}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3831331900}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=29134348}) 2024-04-19T01:05:27.906 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2128551135}) 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2056912989}) 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2134570861}) 2024-04-19T01:05:27.907 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.894+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4020326036}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029089085}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057797874}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2651740252}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1808530551}) 2024-04-19T01:05:27.908 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2326679851}) 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3215070}) 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.909 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1618571162}) 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=560915831}) 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.898+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3338097513}) 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587420388}) 2024-04-19T01:05:27.910 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3827211741}) 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3913530704}) 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.911 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3068371395}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3180163976}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1634444776}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4030849428}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4063735255}) 2024-04-19T01:05:27.912 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.902+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2266012292}) 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3755062880}) 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=326173816}) 2024-04-19T01:05:27.920 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=528526772}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2540381192}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=675953661}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=689310806}) 2024-04-19T01:05:27.921 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1568536037}) 2024-04-19T01:05:27.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.922 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2823205060}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2942675197}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3830045264}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022356570}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467256650}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4135163394}) 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.923 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3887411561}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3376435364}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4270384759}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2615631868}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.906+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=958775264}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053804681}) 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.924 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882051868}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=774679781}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3133693645}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3808472831}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2437815052}) 2024-04-19T01:05:27.925 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3507560291}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3980269060}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.910+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2268878487}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615605901}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2732869434}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.926 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557724368}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=367303236}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2332785861}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=913518748}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2243902724}) 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.927 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=890081826}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.914+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4089393096}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=578647974}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3967257485}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2027483094}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1180311992}) 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.928 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.929 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3939863657}) 2024-04-19T01:05:27.929 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.929 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.929 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1538116875}) 2024-04-19T01:05:27.929 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.930 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.930 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=352716792}) 2024-04-19T01:05:27.930 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.930 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535233285}) 2024-04-19T01:05:27.930 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=951555420}) 2024-04-19T01:05:27.930 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140998747}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2622053553}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1262772271}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.918+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.931 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=751905571}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=459472502}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2606427010}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4279137178}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2853266008}) 2024-04-19T01:05:27.932 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1742285885}) 2024-04-19T01:05:27.950 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.950 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.950 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174696041}) 2024-04-19T01:05:27.950 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862760502}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243646010}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3827386640}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1304118901}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.951 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=188858817}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=446057431}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=864603310}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.922+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2944846057}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=595329486}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1305195932}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.952 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2320126540}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=272618316}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1600798733}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2237437995}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3691288252}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1664331023}) 2024-04-19T01:05:27.953 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.926+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2696410648}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704567096}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3697414132}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3669326507}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2353070962}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.954 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1970091457}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1639484355}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=616410277}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3196096406}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3548872331}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3515259203}) 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.955 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559048095}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=295300880}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=712813127}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=840830776}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1147862496}) 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.956 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1277400747}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1903862322}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.930+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4051305647}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916565916}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3449423090}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2890707781}) 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.957 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1359115487}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2271497235}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1255552746}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1154754283}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2235151308}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2255227833}) 2024-04-19T01:05:27.958 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.934+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3456168914}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3820744342}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1151408339}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1118455701}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.959 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2477432005}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1439645144}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1985560540}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3940055873}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2102096916}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968446217}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.960 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513782458}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3915917066}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1856507885}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1410207631}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=604801255}) 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.961 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3763756843}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3000069485}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.938+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2236757729}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=325785351}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3625963351}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3527248532}) 2024-04-19T01:05:27.962 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1430215349}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3108637963}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4238070759}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1437198577}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3472400155}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.942+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4058101526}) 2024-04-19T01:05:27.963 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1134183319}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409707540}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31549916}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3895322450}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=107045968}) 2024-04-19T01:05:27.964 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3246133871}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4173660026}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2758870454}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.946+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1707469259}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2446913855}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484319200}) 2024-04-19T01:05:27.965 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033145295}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2140742424}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4279428848}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2479949158}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=763943897}) 2024-04-19T01:05:27.966 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:27.967 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1731311249}) 2024-04-19T01:05:27.967 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3621333959}) 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.950+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651632169}) 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945625704}) 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.014 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=864220747}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2156506524}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3080869951}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2130072549}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1610287442}) 2024-04-19T01:05:28.015 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600100923}) 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=478229853}) 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.954+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3714817022}) 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.016 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=580352767}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3614920450}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=871932533}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=442052174}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1343274649}) 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.017 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2373826295}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1103488362}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2917053818}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3984278604}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=992756713}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.958+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.018 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831263349}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1495081927}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=618807661}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1317935983}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2871623713}) 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.019 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3869966058}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3975752625}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2834445262}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=19992828}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631463671}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3330491113}) 2024-04-19T01:05:28.020 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1936072961}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=582019738}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1250570799}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952083347}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.962+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3355217091}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.021 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832532168}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105288724}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424052201}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3587932177}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=350504560}) 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.022 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3511608226}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3797663083}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1089879166}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2345362678}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39432032}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740864782}) 2024-04-19T01:05:28.023 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.966+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=418256557}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3188455467}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=744057506}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1596856582}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.024 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3521877003}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1531848696}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=101545394}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035995664}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777426500}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393474353}) 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.025 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3746421455}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1294199167}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=565016290}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1198610629}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2095763138}) 2024-04-19T01:05:28.026 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.970+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1979581590}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665787425}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896565050}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127452981}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4221626965}) 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.027 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2761293338}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1653019689}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=923787655}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1409730976}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.974+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1689776829}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432222067}) 2024-04-19T01:05:28.028 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3136865887}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346946136}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=224793810}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1662125958}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.029 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4038212046}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=465677804}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=346455800}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2999058835}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=484174274}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1871972468}) 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.030 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477470013}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3979973806}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3590511142}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2263966065}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=294799493}) 2024-04-19T01:05:28.031 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3782861185}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.978+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2328430422}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538467962}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856585128}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032267922}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2510242518}) 2024-04-19T01:05:28.032 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2878138407}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2113733132}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4044980265}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=442135747}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2666253535}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.033 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=510542975}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2778007552}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4282795434}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3370425770}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1163758471}) 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.034 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=934911522}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1960335683}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3173472909}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.982+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=768311454}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523963786}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799072302}) 2024-04-19T01:05:28.035 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3202342848}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=760509354}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2010280247}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2596455215}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4175492549}) 2024-04-19T01:05:28.036 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3514422841}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.986+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3738380677}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398216493}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514767405}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353277544}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.037 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=671674509}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3234396602}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=479955721}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=843463124}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2294185583}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3727566394}) 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.038 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761396672}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3385799760}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3028559235}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1986338279}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1301002616}) 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.039 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1193729991}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=666837369}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1063793787}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.990+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3052561765}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1807916299}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.040 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058654371}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443428048}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2545081809}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3469487625}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3028943406}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.041 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1008692766}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3923610903}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.994+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3130478637}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574082904}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=477727601}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3476112053}) 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.042 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3602299567}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4122013199}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1614723155}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=623594364}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=207758177}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:27.998+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2090839878}) 2024-04-19T01:05:28.043 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3162924638}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2212221414}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951393789}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=692140198}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.044 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4138325769}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3355800715}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2252018973}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3830594354}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=173713060}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=47755631}) 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.045 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505047823}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4196785121}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3845427518}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3927934318}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1331699917}) 2024-04-19T01:05:28.046 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2269862379}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2523657111}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.002+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2543743013}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496069590}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3669875832}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244146913}) 2024-04-19T01:05:28.047 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2626730189}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1674826345}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3585437050}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2673370891}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1093865707}) 2024-04-19T01:05:28.048 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.006+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3646323100}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801941137}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272882447}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004773094}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2253899669}) 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.049 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3057312556}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2172976848}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3941405893}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2460491948}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.010+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2961514310}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264225241}) 2024-04-19T01:05:28.050 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1524605721}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2924557150}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=160908051}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2352995620}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.051 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2419823693}) 2024-04-19T01:05:28.052 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.052 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.052 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1046273004}) 2024-04-19T01:05:28.052 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.052 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4166159444}) 2024-04-19T01:05:28.052 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.014+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2689998948}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552361049}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3107716504}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2810796251}) 2024-04-19T01:05:28.234 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2678834505}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1607440586}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2037928379}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1821875533}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3952247099}) 2024-04-19T01:05:28.235 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1446625335}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3001103864}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071003847}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883494093}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4224937756}) 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.236 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2083475204}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1134726396}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2543364363}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=860413528}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.018+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3800753990}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132540173}) 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.237 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622429140}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1500878553}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2969794989}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2133303230}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4057813767}) 2024-04-19T01:05:28.238 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1520715285}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1914633512}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=681489851}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759649823}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3125630463}) 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.239 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2872993355}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3503927789}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1259358560}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=193047147}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3577837757}) 2024-04-19T01:05:28.240 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2298450716}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.022+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=477060152}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1396042008}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=363113485}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367706015}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.241 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=525820878}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2595037198}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1770728856}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1269446746}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1271490886}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2384893233}) 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.242 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210527531}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1286478567}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333014687}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.026+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4117965455}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.243 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4124828285}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1349499975}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2735964638}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1799435378}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2417414851}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1198101416}) 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.244 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224086737}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3266906699}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1287366978}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3196965042}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2999704889}) 2024-04-19T01:05:28.245 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2612386381}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=993747940}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.030+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3227611094}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490195639}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670127056}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.246 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1821742791}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=996934746}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=980054559}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3413454917}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1218611360}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2645568057}) 2024-04-19T01:05:28.247 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1807917102}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085620537}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2027449046}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757385435}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2634460361}) 2024-04-19T01:05:28.248 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1736020868}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3084379470}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1979055839}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=502530717}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.034+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2042302863}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.249 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345961773}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1800827459}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=188592665}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3297605867}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3342668964}) 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.250 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1270194591}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2493700664}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2938513743}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3009134630}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116858399}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029921507}) 2024-04-19T01:05:28.251 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044763311}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=363555893}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1736429851}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4194069551}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.252 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3386702153}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4288965771}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.038+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4261725248}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3431354029}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=476979098}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3270741039}) 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.253 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1948631796}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=769591191}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3718159404}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4120868785}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=776670783}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=447032410}) 2024-04-19T01:05:28.254 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881289814}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=572260068}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2807366910}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4209766007}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.255 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.042+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=653603524}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1532584168}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3012629129}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3037533181}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3469161445}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917252864}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.256 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341713055}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61192840}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3739802820}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1372681665}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3493492076}) 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.257 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3216773618}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2778083185}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.046+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1735430830}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494299638}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1681496224}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.258 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3124665386}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2964899078}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=892835230}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1499418557}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3406213373}) 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.259 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=432303394}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2091317301}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555747822}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223569447}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1999842403}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=442482741}) 2024-04-19T01:05:28.260 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2440022656}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1379178304}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1495038511}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2682360613}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.050+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3566323377}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2497215182}) 2024-04-19T01:05:28.261 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058167302}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2284257831}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1101092273}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2997810268}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.262 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4181909785}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=217998587}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1623222099}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=66283138}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481730632}) 2024-04-19T01:05:28.263 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=376909974}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452386253}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2785660412}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2645059784}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.264 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2979240334}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=101423911}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=177932530}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.054+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2437247996}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468931831}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766321711}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.265 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3430153130}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3026896785}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1457624750}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3779551556}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=201996274}) 2024-04-19T01:05:28.266 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=868114173}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3292078769}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899935270}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3390639384}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1444327186}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.058+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3431066174}) 2024-04-19T01:05:28.267 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4281585655}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1043472870}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2180188566}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=779215294}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3507329266}) 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.268 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3878250318}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013641360}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706806351}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1374477397}) 2024-04-19T01:05:28.269 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4244630200}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=109899555}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=66224826}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4160592485}) 2024-04-19T01:05:28.270 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.062+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=338878420}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=527586028}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930694245}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862924435}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4102363338}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.271 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2663070673}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4108278847}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2226373471}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1920851868}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3723950024}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394323794}) 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.272 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3365538760}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904759978}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3760312942}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3625558621}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2706197235}) 2024-04-19T01:05:28.273 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2824693857}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3093821715}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.066+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2932922587}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=582010014}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4248970246}) 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.274 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652132157}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=59496730}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3763059575}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2787782710}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=339806810}) 2024-04-19T01:05:28.275 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=636711302}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3795258936}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219934810}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=67298016}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169989286}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.276 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=747427285}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2681464200}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3239461646}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=621528401}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3261901787}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.070+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3591763030}) 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.277 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397366465}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371767880}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=155004631}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1978354377}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.278 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=933341968}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3575127427}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4175306512}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2638413468}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3005411197}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372062451}) 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.279 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=628753121}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814547607}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3377726576}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3929092671}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1182447640}) 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.280 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1771905703}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2343551962}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.074+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4053025725}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872128655}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016959750}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2424279447}) 2024-04-19T01:05:28.281 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3696149844}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2623981639}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3245249639}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2978858492}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1406436966}) 2024-04-19T01:05:28.282 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2450026891}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679371827}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832071171}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348391649}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1538347401}) 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.283 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2805283818}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1717132035}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1607683181}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=831516}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.078+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2884447997}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2844501556}) 2024-04-19T01:05:28.284 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787753458}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573178684}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2062675167}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3142134577}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1524389654}) 2024-04-19T01:05:28.285 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=69798355}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3154899634}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3698916452}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372301873}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1488289158}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.286 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765849901}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4012311034}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2224209838}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1657324407}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3916049518}) 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.287 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2932192841}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.082+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4288081401}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034066705}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3485485172}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3879825255}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2313138037}) 2024-04-19T01:05:28.288 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2892836386}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1855561758}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1716841792}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4092007421}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1742118447}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.289 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936317705}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3857349811}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766291018}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2881830488}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3025199598}) 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.290 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=71025749}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.086+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3103887106}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2243618240}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3517938480}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164764570}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925729811}) 2024-04-19T01:05:28.291 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122053147}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=240129252}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3782265551}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2296598095}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.292 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1880203086}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3123965843}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3338966757}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686003852}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127031686}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649991130}) 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.293 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=984431941}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.090+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2933832853}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2321106532}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4292720457}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2530778320}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1243975591}) 2024-04-19T01:05:28.294 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568917297}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3163248029}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080893773}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3820951168}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.295 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=350715048}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3284050535}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1733423973}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=171697509}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3733317987}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4098473450}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.094+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.296 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1328817965}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672245140}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1176718262}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1511437856}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3685686842}) 2024-04-19T01:05:28.297 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3408041191}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=154753305}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=669079802}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025255123}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943145077}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.298 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3817499529}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1291779456}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3201466915}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=378807734}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=332035240}) 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.299 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=600610466}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.098+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4088281301}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714880934}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2108486728}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=14646267}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3656379708}) 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.300 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2733728284}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3077037963}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2606424146}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=398580014}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3079159249}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3915213847}) 2024-04-19T01:05:28.301 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1795488361}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3141426350}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=482574261}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=484518552}) 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.302 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1491393571}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2671451659}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=272554519}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.102+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=497575033}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495191104}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367680829}) 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.303 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=187313123}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=392897367}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1779535057}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4288578982}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=820156574}) 2024-04-19T01:05:28.304 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=531227631}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1380689428}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696477514}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1844448789}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402011499}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.305 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=274742249}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3787873075}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1287499404}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1782657172}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=667019064}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.106+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2014671360}) 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.306 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709869038}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259403509}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306750217}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=638174650}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=816797667}) 2024-04-19T01:05:28.307 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1423099240}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=431172978}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2038748350}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.110+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=4272770366}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211553721}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.308 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699511920}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=904802114}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2397433735}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3412838488}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1810112099}) 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.309 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2274986929}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2501212540}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2945660198}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2387985596}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1756654596}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3607060300}) 2024-04-19T01:05:28.310 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1135335998}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1689558639}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2961075677}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1742548270}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1191469389}) 2024-04-19T01:05:28.311 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.114+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=75144880}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756949769}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017812574}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103920839}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2402556456}) 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.312 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=575688233}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3982318743}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2741668914}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3929618438}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2597139580}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1310545247}) 2024-04-19T01:05:28.313 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3561039971}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102467791}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3714348931}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3304322164}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.314 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3984466966}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2135772674}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3249144177}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.118+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3886617744}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550391979}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136639588}) 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.315 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3739235192}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2223121309}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3501511415}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=118896491}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=292704297}) 2024-04-19T01:05:28.316 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1746000889}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1647800343}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255235856}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170939078}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555022296}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.317 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2901554959}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=966146155}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4154015786}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=660658741}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=687376110}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.122+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2772041929}) 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.318 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=973722379}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705506084}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3832470969}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1781385912}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1610243800}) 2024-04-19T01:05:28.319 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2586000027}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3921439623}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3160260018}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3358230567}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3078808547}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.320 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638189349}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447495772}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=707767371}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=474184101}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=559192580}) 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.321 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1228044425}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1414204202}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.126+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2223872144}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3232391410}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3705687681}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2300954900}) 2024-04-19T01:05:28.322 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2047011341}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3430781681}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2574729612}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2387742372}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=535422360}) 2024-04-19T01:05:28.323 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2540658850}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922803456}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609540814}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=942799927}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1368498213}) 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.324 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1190849279}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2939402571}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4294313143}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2385770729}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.130+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3619330632}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484560065}) 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.325 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1117489991}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2315340461}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=959524717}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1427089556}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.326 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=705901425}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2404146228}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2325692459}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1974291173}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1586802346}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=452872512}) 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.327 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229933177}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2412143632}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3164341544}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2163677215}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1986872289}) 2024-04-19T01:05:28.328 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3650400107}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.134+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=173491704}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821025693}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757778661}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300808394}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1410753036}) 2024-04-19T01:05:28.329 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=545127993}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2764118474}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2222779295}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4280943759}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2297012919}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.330 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3544669824}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=706160353}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=225624414}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3701380890}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4217848371}) 2024-04-19T01:05:28.331 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1297214649}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4051271667}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3766373421}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.138+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2537194634}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433044058}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.332 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768071142}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829192953}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2119542567}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2999633813}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3000181171}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.333 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2589440285}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2986859806}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2002320570}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999376987}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2316739587}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494973246}) 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.334 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3280901499}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2368971829}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1178890592}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1461612081}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2488998232}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.142+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3533602418}) 2024-04-19T01:05:28.335 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101023843}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181919553}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1063368100}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2156649174}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.336 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=992466611}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2862518588}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3713664882}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3250286006}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3488116945}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432428285}) 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.337 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476045235}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3280308048}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3470291835}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2458521115}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=172578568}) 2024-04-19T01:05:28.338 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4045208463}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3806691075}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.146+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3841487280}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638772898}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035286152}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.339 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880096160}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1323780296}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4294505128}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3544477803}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1643014038}) 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.340 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1882218177}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=124308593}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727392093}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=76893533}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2489403921}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=659511631}) 2024-04-19T01:05:28.341 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2902431085}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2414488128}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1711927758}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1733858179}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.150+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1154919720}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.342 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66131641}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011467866}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543362538}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3102937122}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=860208399}) 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.343 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3372899067}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1463072414}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2386742446}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3717873425}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3869931155}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352316809}) 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.344 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166221463}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1370620043}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1128789468}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1230576060}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.345 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3690550224}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=640780085}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.154+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2266970107}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=897607927}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2515290830}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774797483}) 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.346 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=769151884}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1913815058}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=698242309}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1013293772}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2508072793}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.158+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=482119970}) 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.347 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040655579}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=755815945}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206384031}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2309267890}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4009477050}) 2024-04-19T01:05:28.348 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1443312420}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1039527605}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=317900249}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3039183366}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=874841114}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1697575102}) 2024-04-19T01:05:28.349 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554714276}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=144293796}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=204067514}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2894102657}) 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.350 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3083935668}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1295874430}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.162+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2380395962}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1456548570}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86532727}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556221195}) 2024-04-19T01:05:28.351 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=678058964}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=890901987}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=777470161}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3311146470}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=794223066}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=384005715}) 2024-04-19T01:05:28.352 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2890668379}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545795275}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2303256251}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=513880697}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.353 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2374379204}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.166+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1270652207}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3797801697}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4257123765}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3448631823}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3599143642}) 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.354 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159327342}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=560053999}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1642438702}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2580172871}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1413489011}) 2024-04-19T01:05:28.355 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=273567354}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=373618735}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.170+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1038585085}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=43667194}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318289595}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.356 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=904947961}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=803640846}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=4069214604}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1284525566}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3149350582}) 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.357 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=93445843}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3784621592}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049776981}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2494623490}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1396929924}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2487869506}) 2024-04-19T01:05:28.358 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=2598469411}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=858574490}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1750010440}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3600279900}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.174+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=854088144}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.359 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2179056338}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=436313342}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3638649853}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3256328636}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3387296892}) 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.360 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2217876981}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=959760359}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3731038511}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3861990585}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206798237}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111851524}) 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.361 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550360460}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4281599022}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.178+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=802781132}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1083822163}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2621704881}) 2024-04-19T01:05:28.362 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1136676061}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3540311742}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359713182}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3782350776}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517110504}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.363 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3986920041}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3435956237}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=431967587}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3330461618}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2372628771}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.182+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=2444129109}) 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.364 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469387193}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3219324316}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977823637}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=263144529}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1134020904}) 2024-04-19T01:05:28.365 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=1363519321}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3706744981}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=4033808886}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3879601119}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022533941}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.366 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1570246459}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918838301}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3513260915}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1923815860}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=4286897211}) 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.367 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=3227885580}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2663716588}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.186+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3596277653}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1753162140}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2920410070}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2411624103}) 2024-04-19T01:05:28.368 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=4180994025}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=646007513}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2913681724}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=1156796833}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1671806663}) 2024-04-19T01:05:28.369 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3345205659}) 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741700442}) 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.190+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399527543}) 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.370 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=180338673}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=2082778150}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3802891597}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=117895183}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.371 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=25303162}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=2738971296}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=83354513}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1844100890}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294730691}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845089683}) 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.372 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=3781467474}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3779323230}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3967273355}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=2903091564}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1038825620}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.194+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=3242576591}) 2024-04-19T01:05:28.373 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946610473}) 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512833115}) 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.374 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.375 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722625686}) 2024-04-19T01:05:28.375 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.375 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.375 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=1347592338}) 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=1101622776}) 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=2164542752}) 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.376 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=605566732}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=3334964346}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1332820667}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=494563967}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176155685}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151701174}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.377 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=100} crc {logm=376517177}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=100} crc {logm=3349282032}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=100} crc {logm=3914798350}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=100} crc {logm=4287119606}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=100} crc {logm=1350123232}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.198+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=100} crc {logm=1623200679}) 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.378 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=82,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=291834071,mdsmap=2840952586,mgr=1871624564,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3541950639}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=5} crc {logm=4136514007,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgrstat=3028842219}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=83,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3159306283,mdsmap=2840952586,mgr=3013536616,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3541950639}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {logm=81,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=839253121,mdsmap=2840952586,mgr=3781018680,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2401274070}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=7} crc {logm=3896557327,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgrstat=2266441022}) 2024-04-19T01:05:28.379 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {logm=79,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8} crc {logm=261121222,mdsmap=2840952586,mgr=2051240854,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=2808390425}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2713191538,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=512190905}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {logm=78,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2529978488,mdsmap=1401145748,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3883107961}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.202+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=3230368878,mdsmap=2840952586,mgr=4237663773,mgr_command_descs=3636332126,mgr_metadata=2750671369,mgrstat=3541950639}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=41} crc {mon_config_key=936563741,monmap=4080741109,osd_metadata=1368215097,osd_pg_creating=3438928825,osd_snap=3030079819,osdmap=2685655040}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.380 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=40} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=100236187,osdmap=3769682403}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=40} crc {mon_config_key=2716152751,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=760958808,osdmap=2795492971}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=39} crc {mon_config_key=3719646478,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=3360780286,osdmap=3514280306}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=40} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=2331100710,osdmap=1344594650}) 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.381 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.382 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=41} crc {mon_config_key=2604816478,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=3916460321,osdmap=2293856471}) 2024-04-19T01:05:28.382 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.382 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=34,osdmap=40} crc {mon_config_key=4212284759,monmap=4080741110,osd_metadata=1206563275,osd_pg_creating=3438928825,osd_snap=460563125,osdmap=2097595327}) 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=38} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=1015963332,osdmap=222481606}) 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.206+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=37} crc {mon_config_key=4212284759,monmap=4080741109,osd_metadata=3468572908,osd_pg_creating=3438928825,osd_snap=4046508059,osdmap=2851559753}) 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=984940594}) 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.383 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=15} crc {osdmap=2890827650}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=14} crc {osdmap=2850022980}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.214+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.4 ScrubResult(keys {osdmap=15} crc {osdmap=2890827650}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.5 ScrubResult(keys {osdmap=15} crc {osdmap=2890827650}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.6 ScrubResult(keys {osdmap=15} crc {osdmap=2890827650}) 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.384 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.385 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.7 ScrubResult(keys {osdmap=15} crc {osdmap=2890827650}) 2024-04-19T01:05:28.385 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-19T01:05:28.385 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.3 ScrubResult(keys {osdmap=14} crc {osdmap=4178247430}) 2024-04-19T01:05:28.385 INFO:tasks.ceph.mon.g.smithi161.stderr:2024-04-19T01:05:28.218+0000 7f2bbebbc640 -1 log_channel(cluster) log [ERR] : mon.8 ScrubResult(keys {osdmap=14} crc {osdmap=1340466358}) 2024-04-19T01:05:28.849 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 9 2024-04-19T01:05:28.850 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient: get_monmap_and_config 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient: build_initial_monmap 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient: monmap: 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:28.984746+0000 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:28.984746+0000 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:28.982 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:28.983 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:28.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 1 Processor -- start 2024-04-19T01:05:28.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 1 -- start start 2024-04-19T01:05:28.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient: init 2024-04-19T01:05:28.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:28.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient: _add_conns ranks=[10,4,2,3,1,6,7,0,5,9,8] 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f52340a1e20 0x7f52340a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient(hunting): picked mon.noname-f con 0x7f52340a1e20 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 0x7f52340a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 10 monclient(hunting): picked mon.noname-c con 0x7f52340a1430 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.982+0000 7f523bbfa640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52340a2200 0x7f52340a44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 10 monclient(hunting): picked mon.noname-b con 0x7f52340a2200 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f52340a0ae0 con 0x7f52340a1e20 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f52340a1a60 con 0x7f52340a2200 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f523406c040 con 0x7f52340a1430 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:28.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523bbfa640 10 monclient(hunting): authenticate will time out at 2199.112050s 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 0x7f52340a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 0x7f52340a3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:35026/0 (socket says 172.21.15.43:35026) 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 -- 172.21.15.43:0/3535547079 learned_addr learned my addr 172.21.15.43:0/3535547079 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): get_auth_request con 0x7f52340a1430 auth_method 0 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 1 --2- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52340a2200 0x7f52340a44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): get_auth_request con 0x7f52340a2200 auth_method 0 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:28.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239996640 1 --2- 172.21.15.43:0/3535547079 >> v2:172.21.15.161:3300/0 conn(0x7f52340a1e20 0x7f52340a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f523a197640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239996640 10 monclient(hunting): get_auth_request con 0x7f52340a1e20 auth_method 0 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239996640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239996640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239996640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient(hunting): handle_auth_done global_id 184171 payload 274 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 -- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52340a2200 msgr2=0x7f52340a44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 --2- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52340a2200 0x7f52340a44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 -- 172.21.15.43:0/3535547079 >> v2:172.21.15.161:3300/0 conn(0x7f52340a1e20 msgr2=0x7f52340a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 --2- 172.21.15.43:0/3535547079 >> v2:172.21.15.161:3300/0 conn(0x7f52340a1e20 0x7f52340a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:28.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient: _finish_hunting 0 2024-04-19T01:05:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 monclient: found mon.noname-c 2024-04-19T01:05:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 -- 172.21.15.43:0/3535547079 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523406c480 con 0x7f52340a1430 2024-04-19T01:05:28.990 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:28.986+0000 7f5239195640 1 --2- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 0x7f52340a3f60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5224009870 tx=0x7f522401c440 comp rx=0 tx=0).ready entity=mon.4 client_cookie=e9bc740d7fe0364b server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 1 -- 172.21.15.43:0/3535547079 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52240b2070 con 0x7f52340a1430 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 10 monclient: dump: 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.062 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 10 monclient: _finish_auth 0 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 10 monclient: _check_auth_tickets 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 1 -- 172.21.15.43:0/3535547079 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52240a9070 con 0x7f52340a1430 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f5238994640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 5 monclient: authenticate success, global_id 184171 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:29.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 4 set_mon_vals no callback set 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 10 monclient: shutdown 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 -- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 msgr2=0x7f52340a3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 0x7f52340a3f60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5224009870 tx=0x7f522401c440 comp rx=0 tx=0).stop 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 -- 172.21.15.43:0/3535547079 shutdown_connections 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f52340a2200 0x7f52340a44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3535547079 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f52340a1430 0x7f52340a3f60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3535547079 >> v2:172.21.15.161:3300/0 conn(0x7f52340a1e20 0x7f52340a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.067 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.062+0000 7f523bbfa640 1 -- 172.21.15.43:0/3535547079 >> 172.21.15.43:0/3535547079 conn(0x7f52340a86a0 msgr2=0x7f52340a4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 -- 172.21.15.43:0/3535547079 shutdown_connections 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 -- 172.21.15.43:0/3535547079 wait complete. 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient: build_initial_monmap 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient: monmap: 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:29.068 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 librados: starting msgr at 2024-04-19T01:05:29.069 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 librados: starting objecter 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 Processor -- start 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 -- start start 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 librados: setting wanted keys 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 librados: calling monclient init 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient: init 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient: _add_conns ranks=[5,6,3,1,2,8,7,4,0] 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f52340a1e20 0x7f52341d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): picked mon.noname-h con 0x7f52340a1e20 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 0x7f523413ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): picked mon.noname-d con 0x7f52340a2200 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f523413f130 0x7f5234126650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): picked mon.noname-g con 0x7f523413f130 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f52341425b0 con 0x7f523413f130 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5234142430 con 0x7f52340a1e20 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f5234142730 con 0x7f52340a2200 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523bbfa640 10 monclient(hunting): authenticate will time out at 2199.194589s 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 0x7f523413ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f52340a1e20 0x7f52341d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 0x7f523413ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:36724/0 (socket says 172.21.15.43:36724) 2024-04-19T01:05:29.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 -- 172.21.15.43:0/3690796440 learned_addr learned my addr 172.21.15.43:0/3690796440 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): get_auth_request con 0x7f52340a2200 auth_method 0 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): get_auth_request con 0x7f52340a1e20 auth_method 0 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f523413f130 0x7f5234126650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): get_auth_request con 0x7f523413f130 auth_method 0 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239996640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f523a197640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient(hunting): handle_auth_done global_id 184200 payload 824 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 -- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f52340a1e20 msgr2=0x7f52341d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f52340a1e20 0x7f52341d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 -- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f523413f130 msgr2=0x7f5234126650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f523413f130 0x7f5234126650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient: _finish_hunting 0 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 monclient: found mon.noname-d 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 -- 172.21.15.43:0/3690796440 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f523412a590 con 0x7f52340a2200 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f5239195640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 0x7f523413ebf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52240b0040 tx=0x7f52240191e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f522a7fc640 10 client.?.objecter ms_handle_connect 0x7f52340a2200 2024-04-19T01:05:29.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.066+0000 7f522a7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:29.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52240b2040 con 0x7f52340a2200 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: dump: 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.077 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52240bb070 con 0x7f52340a2200 2024-04-19T01:05:29.078 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 5 monclient: authenticate success, global_id 184200 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 monclient: _renew_subs 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5234073330 con 0x7f52340a2200 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f52240044b0 con 0x7f52340a2200 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522bfff640 4 set_mon_vals no callback set 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 monclient: dump: 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.079 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 client.184200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 monclient: _renew_subs 2024-04-19T01:05:29.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f52341282a0 con 0x7f52340a2200 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 1 librados: init done 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5234068440 con 0x7f52340a2200 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f52240ab070 con 0x7f52340a2200 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5218038e10 0x7f521803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f52240e2a00 con 0x7f52340a2200 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 10 client.184200.objecter ms_dispatch 0x7f5234000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 3 client.184200.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 3 client.184200.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:29.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f522a7fc640 20 client.184200.objecter dump_active .. 0 homeless 2024-04-19T01:05:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f5239996640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5218038e10 0x7f521803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.074+0000 7f5239996640 10 monclient: get_auth_request con 0x7f5218038e10 auth_method 0 2024-04-19T01:05:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.078+0000 7f522a7fc640 10 client.184200.objecter ms_handle_connect 0x7f5218038e10 2024-04-19T01:05:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.078+0000 7f5239996640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5218038e10 0x7f521803b2d0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7f5230007a60 tx=0x7f5230008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.082+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5224016150 con 0x7f52340a2200 2024-04-19T01:05:29.082 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.082+0000 7f522a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.083 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.082+0000 7f522a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:29.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-19T01:05:29.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-19T01:05:29.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5234076a20 con 0x7f52340a2200 2024-04-19T01:05:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f522a7fc640 1 -- 172.21.15.43:0/3690796440 <== mon.6 v2:172.21.15.43:3303/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+3193 (secure 0 0 0) 0x7f52240abc40 con 0x7f52340a2200 2024-04-19T01:05:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f522a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-19T01:05:29.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f522a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:29.305 INFO:teuthology.orchestra.run.smithi043.stdout:{"election_epoch":132,"quorum":[0,1,2,3,4,5,6,7,8],"quorum_names":["a","f","b","g","c","h","d","i","e"],"quorum_leader_name":"g","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","modified":"2024-04-19T00:43:51.655686Z","created":"2024-04-19T00:43:51.655686Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":3,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3300","nonce":0},{"type":"v1","addr":"172.21.15.43:6789","nonce":0}]},"addr":"172.21.15.43:6789/0","public_addr":"172.21.15.43:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"f","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3300","nonce":0},{"type":"v1","addr":"172.21.15.161:6789","nonce":0}]},"addr":"172.21.15.161:6789/0","public_addr":"172.21.15.161:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3301","nonce":0},{"type":"v1","addr":"172.21.15.43:6790","nonce":0}]},"addr":"172.21.15.43:6790/0","public_addr":"172.21.15.43:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"g","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3301","nonce":0},{"type":"v1","addr":"172.21.15.161:6790","nonce":0}]},"addr":"172.21.15.161:6790/0","public_addr":"172.21.15.161:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3302","nonce":0},{"type":"v1","addr":"172.21.15.43:6791","nonce":0}]},"addr":"172.21.15.43:6791/0","public_addr":"172.21.15.43:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":5,"name":"h","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3302","nonce":0},{"type":"v1","addr":"172.21.15.161:6791","nonce":0}]},"addr":"172.21.15.161:6791/0","public_addr":"172.21.15.161:6791/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":6,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3303","nonce":0},{"type":"v1","addr":"172.21.15.43:6792","nonce":0}]},"addr":"172.21.15.43:6792/0","public_addr":"172.21.15.43:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":7,"name":"i","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:3303","nonce":0},{"type":"v1","addr":"172.21.15.161:6792","nonce":0}]},"addr":"172.21.15.161:6792/0","public_addr":"172.21.15.161:6792/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":8,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:3304","nonce":0},{"type":"v1","addr":"172.21.15.43:6793","nonce":0}]},"addr":"172.21.15.43:6793/0","public_addr":"172.21.15.43:6793/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 librados: watch_flush enter 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 librados: watch_flush exit 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 20 client.184200.objecter shutdown clearing up homeless session... 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 client.184200.objecter successfully canceled tick 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5218038e10 msgr2=0x7f521803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5218038e10 0x7f521803b2d0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7f5230007a60 tx=0x7f5230008040 comp rx=0 tx=0).stop 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 10 monclient: shutdown 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 msgr2=0x7f523413ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.306+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 0x7f523413ebf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f52240b0040 tx=0x7f52240191e0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 shutdown_connections 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5218038e10 0x7f521803b2d0 unknown :-1 s=CLOSED pgs=3671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f523413f130 0x7f5234126650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f52340a2200 0x7f523413ebf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 --2- 172.21.15.43:0/3690796440 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f52340a1e20 0x7f52341d07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 >> 172.21.15.43:0/3690796440 conn(0x7f52340a86a0 msgr2=0x7f52340711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 shutdown_connections 2024-04-19T01:05:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 -- 172.21.15.43:0/3690796440 wait complete. 2024-04-19T01:05:29.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.310+0000 7f523bbfa640 1 librados: shutdown 2024-04-19T01:05:29.325 INFO:tasks.mon_thrash.ceph_manager:quorum is size 9 2024-04-19T01:05:29.326 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-19T01:05:29.335 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-19T01:05:29.335 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient: get_monmap_and_config 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient: build_initial_monmap 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient: monmap: 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:29.455934+0000 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:29.455934+0000 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:29.453 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:29.454 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 Processor -- start 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 -- start start 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient: init 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient: _add_conns ranks=[7,5,2,10,0,3,9,6,8,4,1] 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efe98138cd0 0x7efe981390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): picked mon.noname-i con 0x7efe98138cd0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:29.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe980a5b80 0x7efe981395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): picked mon.noname-h con 0x7efe980a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 0x7efe98135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): picked mon.noname-b con 0x7efe981358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7efe980a4410 con 0x7efe981358b0 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7efe98139b30 con 0x7efe980a5b80 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7efe980a3dc0 con 0x7efe98138cd0 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efea07ed640 10 monclient(hunting): authenticate will time out at 2199.583626s 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 0x7efe98135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 0x7efe98135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47220/0 (socket says 172.21.15.43:47220) 2024-04-19T01:05:29.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 1 -- 172.21.15.43:0/1803977317 learned_addr learned my addr 172.21.15.43:0/1803977317 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): get_auth_request con 0x7efe981358b0 auth_method 0 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9e589640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efe98138cd0 0x7efe981390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9dd88640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe980a5b80 0x7efe981395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.454+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9e589640 10 monclient(hunting): get_auth_request con 0x7efe98138cd0 auth_method 0 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9e589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9e589640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9e589640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9dd88640 10 monclient(hunting): get_auth_request con 0x7efe980a5b80 auth_method 0 2024-04-19T01:05:29.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9dd88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9dd88640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9dd88640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_done global_id 144632 payload 274 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 -- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efe98138cd0 msgr2=0x7efe981390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efe98138cd0 0x7efe981390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 -- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe980a5b80 msgr2=0x7efe981395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:29.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe980a5b80 0x7efe981395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 10 monclient: _finish_hunting 0 2024-04-19T01:05:29.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 monclient: found mon.noname-b 2024-04-19T01:05:29.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:29.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:29.461 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 -- 172.21.15.43:0/1803977317 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe980a40a0 con 0x7efe981358b0 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9ed8a640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 0x7efe98135d30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efe90009e80 tx=0x7efe9001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b840fbbc857bca server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 1 -- 172.21.15.43:0/1803977317 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe900090d0 con 0x7efe981358b0 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: dump: 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.467 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: _finish_auth 0 2024-04-19T01:05:29.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: _check_auth_tickets 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 1 -- 172.21.15.43:0/1803977317 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe900a5a10 con 0x7efe981358b0 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 1 -- 172.21.15.43:0/1803977317 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe900a5c00 con 0x7efe981358b0 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 5 monclient: authenticate success, global_id 144632 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 4 set_mon_vals no callback set 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efe9d587640 10 monclient: dump: 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.469 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.470 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 10 monclient: shutdown 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 -- 172.21.15.43:0/1803977317 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 msgr2=0x7efe98135d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 0x7efe98135d30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efe90009e80 tx=0x7efe9001c8b0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 -- 172.21.15.43:0/1803977317 shutdown_connections 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7efe981358b0 0x7efe98135d30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe980a5b80 0x7efe981395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 --2- 172.21.15.43:0/1803977317 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7efe98138cd0 0x7efe981390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 -- 172.21.15.43:0/1803977317 >> 172.21.15.43:0/1803977317 conn(0x7efe980ab2f0 msgr2=0x7efe980ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 -- 172.21.15.43:0/1803977317 shutdown_connections 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.458+0000 7efea07ed640 1 -- 172.21.15.43:0/1803977317 wait complete. 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: build_initial_monmap 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: monmap: 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:29.472 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 librados: starting msgr at 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 librados: starting objecter 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 Processor -- start 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- start start 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 librados: setting wanted keys 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 librados: calling monclient init 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: init 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _add_conns ranks=[8,5,6,1,0,7,3,2,4] 2024-04-19T01:05:29.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 0x7efe981d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): picked mon.noname-e con 0x7efe980a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe981358b0 0x7efe980a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): picked mon.noname-h con 0x7efe981358b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efe98138cd0 0x7efe980a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): picked mon.noname-d con 0x7efe98138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7efe980a2010 con 0x7efe981358b0 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7efe980a1e90 con 0x7efe98138cd0 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7efe98142aa0 con 0x7efe980a5b80 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:29.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient(hunting): authenticate will time out at 2199.589538s 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 0x7efe981d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 0x7efe981d0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57964/0 (socket says 172.21.15.43:57964) 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efe98138cd0 0x7efe980a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 1 -- 172.21.15.43:0/3387656808 learned_addr learned my addr 172.21.15.43:0/3387656808 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): get_auth_request con 0x7efe980a5b80 auth_method 0 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): get_auth_request con 0x7efe98138cd0 auth_method 0 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe981358b0 0x7efe980a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): get_auth_request con 0x7efe981358b0 auth_method 0 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9dd88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient(hunting): handle_auth_done global_id 184209 payload 824 2024-04-19T01:05:29.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 -- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 msgr2=0x7efe981d0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 0x7efe981d0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 -- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe981358b0 msgr2=0x7efe980a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe981358b0 0x7efe980a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient: _finish_hunting 0 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 monclient: found mon.noname-d 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:29.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 -- 172.21.15.43:0/3387656808 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe980a1510 con 0x7efe98138cd0 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9e589640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 0x7efe981d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9ed8a640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efe98138cd0 0x7efe980a0fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efe980a4590 tx=0x7efe900196b0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 client.?.objecter ms_handle_connect 0x7efe98138cd0 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe900bc020 con 0x7efe98138cd0 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: dump: 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.478 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: _finish_auth 0 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:29.479 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe90003e10 con 0x7efe98138cd0 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 5 monclient: authenticate success, global_id 184209 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _renew_subs 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe980a3040 con 0x7efe98138cd0 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7efe900a57b0 con 0x7efe98138cd0 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe9cd86640 4 set_mon_vals no callback set 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 monclient: dump: 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.480 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.481 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 client.184209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _renew_subs 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7efe9809bdb0 con 0x7efe98138cd0 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 librados: init done 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7efe900bf020 con 0x7efe98138cd0 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe980693b0 con 0x7efe98138cd0 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efe7c038e10 0x7efe7c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7efe900e3c60 con 0x7efe98138cd0 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 10 client.184209.objecter ms_dispatch 0x7efe98000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 3 client.184209.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 3 client.184209.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.462+0000 7efe8effd640 20 client.184209.objecter dump_active .. 0 homeless 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.466+0000 7efe9e589640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efe7c038e10 0x7efe7c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.466+0000 7efe9e589640 10 monclient: get_auth_request con 0x7efe7c038e10 auth_method 0 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.466+0000 7efe8effd640 10 client.184209.objecter ms_handle_connect 0x7efe7c038e10 2024-04-19T01:05:29.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.466+0000 7efe9e589640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efe7c038e10 0x7efe7c03b2d0 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7efe94007df0 tx=0x7efe94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.484 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.470+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efe900c6070 con 0x7efe98138cd0 2024-04-19T01:05:29.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.470+0000 7efe8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.485 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.470+0000 7efe8effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:29.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.630+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe9809c040 con 0x7efe7c038e10 2024-04-19T01:05:29.639 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.638+0000 7efe8effd640 1 -- 172.21.15.43:0/3387656808 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113902 (secure 0 0 0) 0x7efe9809c040 con 0x7efe7c038e10 2024-04-19T01:05:29.640 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T01:05:29.640 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T01:05:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 10 librados: watch_flush enter 2024-04-19T01:05:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 10 librados: watch_flush exit 2024-04-19T01:05:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 20 client.184209.objecter shutdown clearing up homeless session... 2024-04-19T01:05:29.642 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 10 client.184209.objecter successfully canceled tick 2024-04-19T01:05:29.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efe7c038e10 msgr2=0x7efe7c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efe7c038e10 0x7efe7c03b2d0 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7efe94007df0 tx=0x7efe94008040 comp rx=0 tx=0).stop 2024-04-19T01:05:29.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 10 monclient: shutdown 2024-04-19T01:05:29.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:29.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efe98138cd0 msgr2=0x7efe980a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.643 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efe98138cd0 0x7efe980a0fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efe980a4590 tx=0x7efe900196b0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 shutdown_connections 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7efe98138cd0 0x7efe980a0fd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7efe981358b0 0x7efe980a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7efe7c038e10 0x7efe7c03b2d0 unknown :-1 s=CLOSED pgs=3672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 --2- 172.21.15.43:0/3387656808 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7efe980a5b80 0x7efe981d0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 >> 172.21.15.43:0/3387656808 conn(0x7efe980ab2f0 msgr2=0x7efe980a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 shutdown_connections 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 -- 172.21.15.43:0/3387656808 wait complete. 2024-04-19T01:05:29.645 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.642+0000 7efea07ed640 1 librados: shutdown 2024-04-19T01:05:29.659 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1328,"stamp":"2024-04-19T01:05:29.332359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":448576,"kb_used_data":24112,"kb_used_omap":9,"kb_used_meta":424374,"kb_avail":565782464,"statfs":{"total":579820584960,"available":579361243136,"internally_reserved":0,"allocated":24690688,"data_stored":10993320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":434559649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004097"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683169+0000","last_change":"2024-04-19T00:45:27.797265+0000","last_active":"2024-04-19T01:03:40.683169+0000","last_peered":"2024-04-19T01:03:40.683169+0000","last_clean":"2024-04-19T01:03:40.683169+0000","last_became_active":"2024-04-19T00:45:27.796802+0000","last_became_peered":"2024-04-19T00:45:27.796802+0000","last_unstale":"2024-04-19T01:03:40.683169+0000","last_undegraded":"2024-04-19T01:03:40.683169+0000","last_fullsized":"2024-04-19T01:03:40.683169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:39:20.387401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777710+0000","last_change":"2024-04-19T00:45:27.717179+0000","last_active":"2024-04-19T01:03:41.777710+0000","last_peered":"2024-04-19T01:03:41.777710+0000","last_clean":"2024-04-19T01:03:41.777710+0000","last_became_active":"2024-04-19T00:45:27.715591+0000","last_became_peered":"2024-04-19T00:45:27.715591+0000","last_unstale":"2024-04-19T01:03:41.777710+0000","last_undegraded":"2024-04-19T01:03:41.777710+0000","last_fullsized":"2024-04-19T01:03:41.777710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:12:05.981056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404350+0000","last_change":"2024-04-19T00:45:27.745246+0000","last_active":"2024-04-19T01:03:42.404350+0000","last_peered":"2024-04-19T01:03:42.404350+0000","last_clean":"2024-04-19T01:03:42.404350+0000","last_became_active":"2024-04-19T00:45:27.744986+0000","last_became_peered":"2024-04-19T00:45:27.744986+0000","last_unstale":"2024-04-19T01:03:42.404350+0000","last_undegraded":"2024-04-19T01:03:42.404350+0000","last_fullsized":"2024-04-19T01:03:42.404350+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:47:38.164550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683470+0000","last_change":"2024-04-19T00:45:27.767266+0000","last_active":"2024-04-19T01:03:40.683470+0000","last_peered":"2024-04-19T01:03:40.683470+0000","last_clean":"2024-04-19T01:03:40.683470+0000","last_became_active":"2024-04-19T00:45:27.767040+0000","last_became_peered":"2024-04-19T00:45:27.767040+0000","last_unstale":"2024-04-19T01:03:40.683470+0000","last_undegraded":"2024-04-19T01:03:40.683470+0000","last_fullsized":"2024-04-19T01:03:40.683470+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:43:32.043488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777937+0000","last_change":"2024-04-19T00:45:27.717564+0000","last_active":"2024-04-19T01:03:41.777937+0000","last_peered":"2024-04-19T01:03:41.777937+0000","last_clean":"2024-04-19T01:03:41.777937+0000","last_became_active":"2024-04-19T00:45:27.717286+0000","last_became_peered":"2024-04-19T00:45:27.717286+0000","last_unstale":"2024-04-19T01:03:41.777937+0000","last_undegraded":"2024-04-19T01:03:41.777937+0000","last_fullsized":"2024-04-19T01:03:41.777937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:43:17.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683854+0000","last_change":"2024-04-19T00:45:27.799104+0000","last_active":"2024-04-19T01:03:40.683854+0000","last_peered":"2024-04-19T01:03:40.683854+0000","last_clean":"2024-04-19T01:03:40.683854+0000","last_became_active":"2024-04-19T00:45:27.795414+0000","last_became_peered":"2024-04-19T00:45:27.795414+0000","last_unstale":"2024-04-19T01:03:40.683854+0000","last_undegraded":"2024-04-19T01:03:40.683854+0000","last_fullsized":"2024-04-19T01:03:40.683854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:42:29.849497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404539+0000","last_change":"2024-04-19T00:45:27.795299+0000","last_active":"2024-04-19T01:03:42.404539+0000","last_peered":"2024-04-19T01:03:42.404539+0000","last_clean":"2024-04-19T01:03:42.404539+0000","last_became_active":"2024-04-19T00:45:27.790389+0000","last_became_peered":"2024-04-19T00:45:27.790389+0000","last_unstale":"2024-04-19T01:03:42.404539+0000","last_undegraded":"2024-04-19T01:03:42.404539+0000","last_fullsized":"2024-04-19T01:03:42.404539+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:54:10.325529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403468+0000","last_change":"2024-04-19T00:45:27.738033+0000","last_active":"2024-04-19T01:03:42.403468+0000","last_peered":"2024-04-19T01:03:42.403468+0000","last_clean":"2024-04-19T01:03:42.403468+0000","last_became_active":"2024-04-19T00:45:27.735932+0000","last_became_peered":"2024-04-19T00:45:27.735932+0000","last_unstale":"2024-04-19T01:03:42.403468+0000","last_undegraded":"2024-04-19T01:03:42.403468+0000","last_fullsized":"2024-04-19T01:03:42.403468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:35:41.884097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1364,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403517+0000","last_change":"2024-04-19T00:45:27.765213+0000","last_active":"2024-04-19T01:03:42.403517+0000","last_peered":"2024-04-19T01:03:42.403517+0000","last_clean":"2024-04-19T01:03:42.403517+0000","last_became_active":"2024-04-19T00:45:27.752978+0000","last_became_peered":"2024-04-19T00:45:27.752978+0000","last_unstale":"2024-04-19T01:03:42.403517+0000","last_undegraded":"2024-04-19T01:03:42.403517+0000","last_fullsized":"2024-04-19T01:03:42.403517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:55:11.690354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"17'7","reported_seq":1376,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403206+0000","last_change":"2024-04-19T00:45:27.805132+0000","last_active":"2024-04-19T01:03:42.403206+0000","last_peered":"2024-04-19T01:03:42.403206+0000","last_clean":"2024-04-19T01:03:42.403206+0000","last_became_active":"2024-04-19T00:45:27.798161+0000","last_became_peered":"2024-04-19T00:45:27.798161+0000","last_unstale":"2024-04-19T01:03:42.403206+0000","last_undegraded":"2024-04-19T01:03:42.403206+0000","last_fullsized":"2024-04-19T01:03:42.403206+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:51:44.398144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683194+0000","last_change":"2024-04-19T00:45:27.799286+0000","last_active":"2024-04-19T01:03:40.683194+0000","last_peered":"2024-04-19T01:03:40.683194+0000","last_clean":"2024-04-19T01:03:40.683194+0000","last_became_active":"2024-04-19T00:45:27.797033+0000","last_became_peered":"2024-04-19T00:45:27.797033+0000","last_unstale":"2024-04-19T01:03:40.683194+0000","last_undegraded":"2024-04-19T01:03:40.683194+0000","last_fullsized":"2024-04-19T01:03:40.683194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:04:40.412420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403776+0000","last_change":"2024-04-19T00:45:27.760256+0000","last_active":"2024-04-19T01:03:42.403776+0000","last_peered":"2024-04-19T01:03:42.403776+0000","last_clean":"2024-04-19T01:03:42.403776+0000","last_became_active":"2024-04-19T00:45:27.757597+0000","last_became_peered":"2024-04-19T00:45:27.757597+0000","last_unstale":"2024-04-19T01:03:42.403776+0000","last_undegraded":"2024-04-19T01:03:42.403776+0000","last_fullsized":"2024-04-19T01:03:42.403776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:41:10.030457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978226+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T01:03:41.978226+0000","last_peered":"2024-04-19T01:03:41.978226+0000","last_clean":"2024-04-19T01:03:41.978226+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T01:03:41.978226+0000","last_undegraded":"2024-04-19T01:03:41.978226+0000","last_fullsized":"2024-04-19T01:03:41.978226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:18:46.046655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777648+0000","last_change":"2024-04-19T00:45:27.742732+0000","last_active":"2024-04-19T01:03:41.777648+0000","last_peered":"2024-04-19T01:03:41.777648+0000","last_clean":"2024-04-19T01:03:41.777648+0000","last_became_active":"2024-04-19T00:45:27.740409+0000","last_became_peered":"2024-04-19T00:45:27.740409+0000","last_unstale":"2024-04-19T01:03:41.777648+0000","last_undegraded":"2024-04-19T01:03:41.777648+0000","last_fullsized":"2024-04-19T01:03:41.777648+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:13:15.654443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.778149+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T01:03:41.778149+0000","last_peered":"2024-04-19T01:03:41.778149+0000","last_clean":"2024-04-19T01:03:41.778149+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T01:03:41.778149+0000","last_undegraded":"2024-04-19T01:03:41.778149+0000","last_fullsized":"2024-04-19T01:03:41.778149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777602+0000","last_change":"2024-04-19T00:45:27.679388+0000","last_active":"2024-04-19T01:03:41.777602+0000","last_peered":"2024-04-19T01:03:41.777602+0000","last_clean":"2024-04-19T01:03:41.777602+0000","last_became_active":"2024-04-19T00:45:27.677388+0000","last_became_peered":"2024-04-19T00:45:27.677388+0000","last_unstale":"2024-04-19T01:03:41.777602+0000","last_undegraded":"2024-04-19T01:03:41.777602+0000","last_fullsized":"2024-04-19T01:03:41.777602+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:57:14.716992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683573+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T01:03:40.683573+0000","last_peered":"2024-04-19T01:03:40.683573+0000","last_clean":"2024-04-19T01:03:40.683573+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T01:03:40.683573+0000","last_undegraded":"2024-04-19T01:03:40.683573+0000","last_fullsized":"2024-04-19T01:03:40.683573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:48:30.196169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404298+0000","last_change":"2024-04-19T00:45:27.785476+0000","last_active":"2024-04-19T01:03:42.404298+0000","last_peered":"2024-04-19T01:03:42.404298+0000","last_clean":"2024-04-19T01:03:42.404298+0000","last_became_active":"2024-04-19T00:45:27.785226+0000","last_became_peered":"2024-04-19T00:45:27.785226+0000","last_unstale":"2024-04-19T01:03:42.404298+0000","last_undegraded":"2024-04-19T01:03:42.404298+0000","last_fullsized":"2024-04-19T01:03:42.404298+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:32:30.269561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403439+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T01:03:42.403439+0000","last_peered":"2024-04-19T01:03:42.403439+0000","last_clean":"2024-04-19T01:03:42.403439+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T01:03:42.403439+0000","last_undegraded":"2024-04-19T01:03:42.403439+0000","last_fullsized":"2024-04-19T01:03:42.403439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":1537,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977832+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T01:03:41.977832+0000","last_peered":"2024-04-19T01:03:41.977832+0000","last_clean":"2024-04-19T01:03:41.977832+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T01:03:41.977832+0000","last_undegraded":"2024-04-19T01:03:41.977832+0000","last_fullsized":"2024-04-19T01:03:41.977832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:02:45.312062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403239+0000","last_change":"2024-04-19T00:45:27.821135+0000","last_active":"2024-04-19T01:03:42.403239+0000","last_peered":"2024-04-19T01:03:42.403239+0000","last_clean":"2024-04-19T01:03:42.403239+0000","last_became_active":"2024-04-19T00:45:27.813538+0000","last_became_peered":"2024-04-19T00:45:27.813538+0000","last_unstale":"2024-04-19T01:03:42.403239+0000","last_undegraded":"2024-04-19T01:03:42.403239+0000","last_fullsized":"2024-04-19T01:03:42.403239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:20:22.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403483+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T01:03:42.403483+0000","last_peered":"2024-04-19T01:03:42.403483+0000","last_clean":"2024-04-19T01:03:42.403483+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T01:03:42.403483+0000","last_undegraded":"2024-04-19T01:03:42.403483+0000","last_fullsized":"2024-04-19T01:03:42.403483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777775+0000","last_change":"2024-04-19T00:45:27.763529+0000","last_active":"2024-04-19T01:03:41.777775+0000","last_peered":"2024-04-19T01:03:41.777775+0000","last_clean":"2024-04-19T01:03:41.777775+0000","last_became_active":"2024-04-19T00:45:27.763230+0000","last_became_peered":"2024-04-19T00:45:27.763230+0000","last_unstale":"2024-04-19T01:03:41.777775+0000","last_undegraded":"2024-04-19T01:03:41.777775+0000","last_fullsized":"2024-04-19T01:03:41.777775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:22:57.898965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683667+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T01:03:40.683667+0000","last_peered":"2024-04-19T01:03:40.683667+0000","last_clean":"2024-04-19T01:03:40.683667+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T01:03:40.683667+0000","last_undegraded":"2024-04-19T01:03:40.683667+0000","last_fullsized":"2024-04-19T01:03:40.683667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:04:03.225536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404316+0000","last_change":"2024-04-19T00:45:27.647807+0000","last_active":"2024-04-19T01:03:42.404316+0000","last_peered":"2024-04-19T01:03:42.404316+0000","last_clean":"2024-04-19T01:03:42.404316+0000","last_became_active":"2024-04-19T00:45:27.647106+0000","last_became_peered":"2024-04-19T00:45:27.647106+0000","last_unstale":"2024-04-19T01:03:42.404316+0000","last_undegraded":"2024-04-19T01:03:42.404316+0000","last_fullsized":"2024-04-19T01:03:42.404316+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:21:15.104447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978360+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T01:03:41.978360+0000","last_peered":"2024-04-19T01:03:41.978360+0000","last_clean":"2024-04-19T01:03:41.978360+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T01:03:41.978360+0000","last_undegraded":"2024-04-19T01:03:41.978360+0000","last_fullsized":"2024-04-19T01:03:41.978360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:36:04.048348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404715+0000","last_change":"2024-04-19T00:45:27.659480+0000","last_active":"2024-04-19T01:03:42.404715+0000","last_peered":"2024-04-19T01:03:42.404715+0000","last_clean":"2024-04-19T01:03:42.404715+0000","last_became_active":"2024-04-19T00:45:27.659197+0000","last_became_peered":"2024-04-19T00:45:27.659197+0000","last_unstale":"2024-04-19T01:03:42.404715+0000","last_undegraded":"2024-04-19T01:03:42.404715+0000","last_fullsized":"2024-04-19T01:03:42.404715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:55:55.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777965+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T01:03:41.777965+0000","last_peered":"2024-04-19T01:03:41.777965+0000","last_clean":"2024-04-19T01:03:41.777965+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T01:03:41.777965+0000","last_undegraded":"2024-04-19T01:03:41.777965+0000","last_fullsized":"2024-04-19T01:03:41.777965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404412+0000","last_change":"2024-04-19T00:45:27.661883+0000","last_active":"2024-04-19T01:03:42.404412+0000","last_peered":"2024-04-19T01:03:42.404412+0000","last_clean":"2024-04-19T01:03:42.404412+0000","last_became_active":"2024-04-19T00:45:27.661535+0000","last_became_peered":"2024-04-19T00:45:27.661535+0000","last_unstale":"2024-04-19T01:03:42.404412+0000","last_undegraded":"2024-04-19T01:03:42.404412+0000","last_fullsized":"2024-04-19T01:03:42.404412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:46:16.741442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404690+0000","last_change":"2024-04-19T00:45:27.722242+0000","last_active":"2024-04-19T01:03:42.404690+0000","last_peered":"2024-04-19T01:03:42.404690+0000","last_clean":"2024-04-19T01:03:42.404690+0000","last_became_active":"2024-04-19T00:45:27.719717+0000","last_became_peered":"2024-04-19T00:45:27.719717+0000","last_unstale":"2024-04-19T01:03:42.404690+0000","last_undegraded":"2024-04-19T01:03:42.404690+0000","last_fullsized":"2024-04-19T01:03:42.404690+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:28:43.370123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403826+0000","last_change":"2024-04-19T00:45:27.725269+0000","last_active":"2024-04-19T01:03:42.403826+0000","last_peered":"2024-04-19T01:03:42.403826+0000","last_clean":"2024-04-19T01:03:42.403826+0000","last_became_active":"2024-04-19T00:45:27.719793+0000","last_became_peered":"2024-04-19T00:45:27.719793+0000","last_unstale":"2024-04-19T01:03:42.403826+0000","last_undegraded":"2024-04-19T01:03:42.403826+0000","last_fullsized":"2024-04-19T01:03:42.403826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:05:34.645736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404841+0000","last_change":"2024-04-19T00:45:27.770635+0000","last_active":"2024-04-19T01:03:42.404841+0000","last_peered":"2024-04-19T01:03:42.404841+0000","last_clean":"2024-04-19T01:03:42.404841+0000","last_became_active":"2024-04-19T00:45:27.769635+0000","last_became_peered":"2024-04-19T00:45:27.769635+0000","last_unstale":"2024-04-19T01:03:42.404841+0000","last_undegraded":"2024-04-19T01:03:42.404841+0000","last_fullsized":"2024-04-19T01:03:42.404841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:30:12.981643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777809+0000","last_change":"2024-04-19T00:45:27.752249+0000","last_active":"2024-04-19T01:03:41.777809+0000","last_peered":"2024-04-19T01:03:41.777809+0000","last_clean":"2024-04-19T01:03:41.777809+0000","last_became_active":"2024-04-19T00:45:27.751895+0000","last_became_peered":"2024-04-19T00:45:27.751895+0000","last_unstale":"2024-04-19T01:03:41.777809+0000","last_undegraded":"2024-04-19T01:03:41.777809+0000","last_fullsized":"2024-04-19T01:03:41.777809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:19:10.348532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403676+0000","last_change":"2024-04-19T00:45:27.813207+0000","last_active":"2024-04-19T01:03:42.403676+0000","last_peered":"2024-04-19T01:03:42.403676+0000","last_clean":"2024-04-19T01:03:42.403676+0000","last_became_active":"2024-04-19T00:45:27.812520+0000","last_became_peered":"2024-04-19T00:45:27.812520+0000","last_unstale":"2024-04-19T01:03:42.403676+0000","last_undegraded":"2024-04-19T01:03:42.403676+0000","last_fullsized":"2024-04-19T01:03:42.403676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:21:29.786709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977870+0000","last_change":"2024-04-19T00:45:27.796325+0000","last_active":"2024-04-19T01:03:41.977870+0000","last_peered":"2024-04-19T01:03:41.977870+0000","last_clean":"2024-04-19T01:03:41.977870+0000","last_became_active":"2024-04-19T00:45:27.795698+0000","last_became_peered":"2024-04-19T00:45:27.795698+0000","last_unstale":"2024-04-19T01:03:41.977870+0000","last_undegraded":"2024-04-19T01:03:41.977870+0000","last_fullsized":"2024-04-19T01:03:41.977870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:16:23.414716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978229+0000","last_change":"2024-04-19T00:45:27.750891+0000","last_active":"2024-04-19T01:03:41.978229+0000","last_peered":"2024-04-19T01:03:41.978229+0000","last_clean":"2024-04-19T01:03:41.978229+0000","last_became_active":"2024-04-19T00:45:27.750623+0000","last_became_peered":"2024-04-19T00:45:27.750623+0000","last_unstale":"2024-04-19T01:03:41.978229+0000","last_undegraded":"2024-04-19T01:03:41.978229+0000","last_fullsized":"2024-04-19T01:03:41.978229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:14:11.322418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977779+0000","last_change":"2024-04-19T00:45:27.758851+0000","last_active":"2024-04-19T01:03:41.977779+0000","last_peered":"2024-04-19T01:03:41.977779+0000","last_clean":"2024-04-19T01:03:41.977779+0000","last_became_active":"2024-04-19T00:45:27.758559+0000","last_became_peered":"2024-04-19T00:45:27.758559+0000","last_unstale":"2024-04-19T01:03:41.977779+0000","last_undegraded":"2024-04-19T01:03:41.977779+0000","last_fullsized":"2024-04-19T01:03:41.977779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:04:39.957325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978326+0000","last_change":"2024-04-19T00:45:27.701325+0000","last_active":"2024-04-19T01:03:41.978326+0000","last_peered":"2024-04-19T01:03:41.978326+0000","last_clean":"2024-04-19T01:03:41.978326+0000","last_became_active":"2024-04-19T00:45:27.697796+0000","last_became_peered":"2024-04-19T00:45:27.697796+0000","last_unstale":"2024-04-19T01:03:41.978326+0000","last_undegraded":"2024-04-19T01:03:41.978326+0000","last_fullsized":"2024-04-19T01:03:41.978326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:38:52.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403581+0000","last_change":"2024-04-19T00:45:27.815387+0000","last_active":"2024-04-19T01:03:42.403581+0000","last_peered":"2024-04-19T01:03:42.403581+0000","last_clean":"2024-04-19T01:03:42.403581+0000","last_became_active":"2024-04-19T00:45:27.814761+0000","last_became_peered":"2024-04-19T00:45:27.814761+0000","last_unstale":"2024-04-19T01:03:42.403581+0000","last_undegraded":"2024-04-19T01:03:42.403581+0000","last_fullsized":"2024-04-19T01:03:42.403581+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:29:10.459966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978040+0000","last_change":"2024-04-19T00:45:27.781966+0000","last_active":"2024-04-19T01:03:41.978040+0000","last_peered":"2024-04-19T01:03:41.978040+0000","last_clean":"2024-04-19T01:03:41.978040+0000","last_became_active":"2024-04-19T00:45:27.779682+0000","last_became_peered":"2024-04-19T00:45:27.779682+0000","last_unstale":"2024-04-19T01:03:41.978040+0000","last_undegraded":"2024-04-19T01:03:41.978040+0000","last_fullsized":"2024-04-19T01:03:41.978040+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:45:17.983232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683489+0000","last_change":"2024-04-19T00:45:27.790124+0000","last_active":"2024-04-19T01:03:40.683489+0000","last_peered":"2024-04-19T01:03:40.683489+0000","last_clean":"2024-04-19T01:03:40.683489+0000","last_became_active":"2024-04-19T00:45:27.788325+0000","last_became_peered":"2024-04-19T00:45:27.788325+0000","last_unstale":"2024-04-19T01:03:40.683489+0000","last_undegraded":"2024-04-19T01:03:40.683489+0000","last_fullsized":"2024-04-19T01:03:40.683489+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:16:28.463405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607809,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607808,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607807,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75800,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296040,"statfs":{"total":96636764160,"available":96559144960,"internally_reserved":0,"allocated":4468736,"data_stored":2184837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640512,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79440,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292400,"statfs":{"total":96636764160,"available":96555417600,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71644,"kb_used_data":4368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300196,"statfs":{"total":96636764160,"available":96563400704,"internally_reserved":0,"allocated":4472832,"data_stored":2184842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640510,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71124,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300716,"statfs":{"total":96636764160,"available":96563933184,"internally_reserved":0,"allocated":3940352,"data_stored":1655914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:29.661 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:29.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.786+0000 7fc7fd34f640 10 monclient: get_monmap_and_config 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.786+0000 7fc7fd34f640 10 monclient: build_initial_monmap 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.786+0000 7fc7fd34f640 10 monclient: monmap: 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:29.789709+0000 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:29.789709+0000 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:29.787 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:29.788 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:29.788 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 Processor -- start 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 -- start start 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient: init 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient: _add_conns ranks=[8,4,2,5,1,9,6,0,7,10,3] 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc7f8138cd0 0x7fc7f81390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): picked mon.noname-e con 0x7fc7f8138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f80a5b80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 0x7fc7f8135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f81358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:29.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f80a4410 con 0x7fc7f81358b0 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc7f8139b30 con 0x7fc7f80a5b80 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fc7f80a3dc0 con 0x7fc7f8138cd0 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7fd34f640 10 monclient(hunting): authenticate will time out at 2199.917127s 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 0x7fc7f8135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 0x7fc7f8135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47236/0 (socket says 172.21.15.43:47236) 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 1 -- 172.21.15.43:0/3698735137 learned_addr learned my addr 172.21.15.43:0/3698735137 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): get_auth_request con 0x7fc7f81358b0 auth_method 0 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc7f8138cd0 0x7fc7f81390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): get_auth_request con 0x7fc7f80a5b80 auth_method 0 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7f8138cd0 auth_method 0 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient(hunting): handle_auth_done global_id 184180 payload 274 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 -- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc7f8138cd0 msgr2=0x7fc7f81390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc7f8138cd0 0x7fc7f81390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 -- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 msgr2=0x7fc7f8135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 0x7fc7f8135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient: _finish_hunting 0 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 monclient: found mon.noname-c 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:29.794 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 -- 172.21.15.43:0/3698735137 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f80a40a0 con 0x7fc7f80a5b80 2024-04-19T01:05:29.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f6ffd640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc7f8138cd0 0x7fc7f81390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:29.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7f77fe640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 0x7fc7f8135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:29.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.790+0000 7fc7effff640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81395f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00097f0 tx=0x7fc7e001c340 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b65e80889319c936 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 1 -- 172.21.15.43:0/3698735137 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7e00b0070 con 0x7fc7f80a5b80 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: dump: 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.800 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:29.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 1 -- 172.21.15.43:0/3698735137 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7e00aa810 con 0x7fc7f80a5b80 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 5 monclient: authenticate success, global_id 184180 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 1 -- 172.21.15.43:0/3698735137 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7e00a9300 con 0x7fc7f80a5b80 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 4 set_mon_vals no callback set 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f67fc640 10 monclient: dump: 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.802 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: shutdown 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:29.803 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f80a5b80 msgr2=0x7fc7f81395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81395f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc7e00097f0 tx=0x7fc7e001c340 comp rx=0 tx=0).stop 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/3698735137 shutdown_connections 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc7f81358b0 0x7fc7f8135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81395f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/3698735137 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc7f8138cd0 0x7fc7f81390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/3698735137 >> 172.21.15.43:0/3698735137 conn(0x7fc7f80ab2f0 msgr2=0x7fc7f80ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/3698735137 shutdown_connections 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/3698735137 wait complete. 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: build_initial_monmap 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: monmap: 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:29.804 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 librados: starting msgr at 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 librados: starting objecter 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 Processor -- start 2024-04-19T01:05:29.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- start start 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 librados: setting wanted keys 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 librados: calling monclient init 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: init 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: _add_conns ranks=[5,7,4,0,3,2,1,6,8] 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): picked mon.noname-h con 0x7fc7f80a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc7f81358b0 0x7fc7f81d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): picked mon.noname-i con 0x7fc7f81358b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 0x7fc7f81d1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f8138cd0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc7f80a2010 con 0x7fc7f8138cd0 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc7f80a1e90 con 0x7fc7f80a5b80 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fc7f8142aa0 con 0x7fc7f81358b0 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient(hunting): authenticate will time out at 2199.922304s 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 0x7fc7f81d1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 0x7fc7f81d1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:35052/0 (socket says 172.21.15.43:35052) 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 -- 172.21.15.43:0/4047049062 learned_addr learned my addr 172.21.15.43:0/4047049062 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): get_auth_request con 0x7fc7f8138cd0 auth_method 0 2024-04-19T01:05:29.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:29.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7effff640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc7f81358b0 0x7fc7f81d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f6ffd640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f6ffd640 10 monclient(hunting): get_auth_request con 0x7fc7f80a5b80 auth_method 0 2024-04-19T01:05:29.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7effff640 10 monclient(hunting): get_auth_request con 0x7fc7f81358b0 auth_method 0 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7effff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7effff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient(hunting): handle_auth_done global_id 184189 payload 824 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 -- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc7f81358b0 msgr2=0x7fc7f81d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc7f81358b0 0x7fc7f81d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 -- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc7f80a5b80 msgr2=0x7fc7f81d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient: _finish_hunting 0 2024-04-19T01:05:29.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 monclient: found mon.noname-c 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 -- 172.21.15.43:0/4047049062 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f80a3040 con 0x7fc7f8138cd0 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f77fe640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 0x7fc7f81d1520 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000efc0 tx=0x7fc7f000c490 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 client.?.objecter ms_handle_connect 0x7fc7f8138cd0 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7f000cbe0 con 0x7fc7f8138cd0 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: dump: 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.814 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.815 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 5 monclient: authenticate success, global_id 184189 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: _renew_subs 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f000cd80 con 0x7fc7f8138cd0 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:29.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7f809e360 con 0x7fc7f8138cd0 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc7f0014d60 con 0x7fc7f8138cd0 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7f5ffb640 4 set_mon_vals no callback set 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 client.184189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7ef7fe640 10 monclient: dump: 2024-04-19T01:05:29.817 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:29.818 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: _renew_subs 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7f809f0e0 con 0x7fc7f8138cd0 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.794+0000 7fc7fd34f640 1 librados: init done 2024-04-19T01:05:29.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc7f001b070 con 0x7fc7f8138cd0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7fd34f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7fd34f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7fd34f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7f80693b0 con 0x7fc7f8138cd0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc7d8038e10 0x7fc7d803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fc7f0019020 con 0x7fc7f8138cd0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 10 client.184189.objecter ms_dispatch 0x7fc7f8000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 3 client.184189.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 3 client.184189.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 20 client.184189.objecter dump_active .. 0 homeless 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7f6ffd640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc7d8038e10 0x7fc7d803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7f6ffd640 10 monclient: get_auth_request con 0x7fc7d8038e10 auth_method 0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7f6ffd640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc7d8038e10 0x7fc7d803b2d0 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80099d0 tx=0x7fc7e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:29.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.798+0000 7fc7ef7fe640 10 client.184189.objecter ms_handle_connect 0x7fc7d8038e10 2024-04-19T01:05:29.821 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.802+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc7f0010040 con 0x7fc7f8138cd0 2024-04-19T01:05:29.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.802+0000 7fc7ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:29.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.802+0000 7fc7ef7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:29.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.962+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc7f8076190 con 0x7fc7d8038e10 2024-04-19T01:05:29.970 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.970+0000 7fc7ef7fe640 1 -- 172.21.15.43:0/4047049062 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113902 (secure 0 0 0) 0x7fc7f8076190 con 0x7fc7d8038e10 2024-04-19T01:05:29.972 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T01:05:29.973 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 10 librados: watch_flush enter 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 10 librados: watch_flush exit 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 20 client.184189.objecter shutdown clearing up homeless session... 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 10 client.184189.objecter successfully canceled tick 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc7d8038e10 msgr2=0x7fc7d803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc7d8038e10 0x7fc7d803b2d0 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7fc7e80099d0 tx=0x7fc7e8008040 comp rx=0 tx=0).stop 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 10 monclient: shutdown 2024-04-19T01:05:29.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 msgr2=0x7fc7f81d1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 0x7fc7f81d1520 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7f000efc0 tx=0x7fc7f000c490 comp rx=0 tx=0).stop 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 shutdown_connections 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc7d8038e10 0x7fc7d803b2d0 unknown :-1 s=CLOSED pgs=3673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc7f8138cd0 0x7fc7f81d1520 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fc7f81358b0 0x7fc7f81d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 --2- 172.21.15.43:0/4047049062 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc7f80a5b80 0x7fc7f81d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 >> 172.21.15.43:0/4047049062 conn(0x7fc7f80ab2f0 msgr2=0x7fc7f8137460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 shutdown_connections 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 -- 172.21.15.43:0/4047049062 wait complete. 2024-04-19T01:05:29.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:29.974+0000 7fc7fd34f640 1 librados: shutdown 2024-04-19T01:05:29.996 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1328,"stamp":"2024-04-19T01:05:29.332359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":448576,"kb_used_data":24112,"kb_used_omap":9,"kb_used_meta":424374,"kb_avail":565782464,"statfs":{"total":579820584960,"available":579361243136,"internally_reserved":0,"allocated":24690688,"data_stored":10993320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":434559649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004097"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683169+0000","last_change":"2024-04-19T00:45:27.797265+0000","last_active":"2024-04-19T01:03:40.683169+0000","last_peered":"2024-04-19T01:03:40.683169+0000","last_clean":"2024-04-19T01:03:40.683169+0000","last_became_active":"2024-04-19T00:45:27.796802+0000","last_became_peered":"2024-04-19T00:45:27.796802+0000","last_unstale":"2024-04-19T01:03:40.683169+0000","last_undegraded":"2024-04-19T01:03:40.683169+0000","last_fullsized":"2024-04-19T01:03:40.683169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:39:20.387401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777710+0000","last_change":"2024-04-19T00:45:27.717179+0000","last_active":"2024-04-19T01:03:41.777710+0000","last_peered":"2024-04-19T01:03:41.777710+0000","last_clean":"2024-04-19T01:03:41.777710+0000","last_became_active":"2024-04-19T00:45:27.715591+0000","last_became_peered":"2024-04-19T00:45:27.715591+0000","last_unstale":"2024-04-19T01:03:41.777710+0000","last_undegraded":"2024-04-19T01:03:41.777710+0000","last_fullsized":"2024-04-19T01:03:41.777710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:12:05.981056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404350+0000","last_change":"2024-04-19T00:45:27.745246+0000","last_active":"2024-04-19T01:03:42.404350+0000","last_peered":"2024-04-19T01:03:42.404350+0000","last_clean":"2024-04-19T01:03:42.404350+0000","last_became_active":"2024-04-19T00:45:27.744986+0000","last_became_peered":"2024-04-19T00:45:27.744986+0000","last_unstale":"2024-04-19T01:03:42.404350+0000","last_undegraded":"2024-04-19T01:03:42.404350+0000","last_fullsized":"2024-04-19T01:03:42.404350+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:47:38.164550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683470+0000","last_change":"2024-04-19T00:45:27.767266+0000","last_active":"2024-04-19T01:03:40.683470+0000","last_peered":"2024-04-19T01:03:40.683470+0000","last_clean":"2024-04-19T01:03:40.683470+0000","last_became_active":"2024-04-19T00:45:27.767040+0000","last_became_peered":"2024-04-19T00:45:27.767040+0000","last_unstale":"2024-04-19T01:03:40.683470+0000","last_undegraded":"2024-04-19T01:03:40.683470+0000","last_fullsized":"2024-04-19T01:03:40.683470+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:43:32.043488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777937+0000","last_change":"2024-04-19T00:45:27.717564+0000","last_active":"2024-04-19T01:03:41.777937+0000","last_peered":"2024-04-19T01:03:41.777937+0000","last_clean":"2024-04-19T01:03:41.777937+0000","last_became_active":"2024-04-19T00:45:27.717286+0000","last_became_peered":"2024-04-19T00:45:27.717286+0000","last_unstale":"2024-04-19T01:03:41.777937+0000","last_undegraded":"2024-04-19T01:03:41.777937+0000","last_fullsized":"2024-04-19T01:03:41.777937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:43:17.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683854+0000","last_change":"2024-04-19T00:45:27.799104+0000","last_active":"2024-04-19T01:03:40.683854+0000","last_peered":"2024-04-19T01:03:40.683854+0000","last_clean":"2024-04-19T01:03:40.683854+0000","last_became_active":"2024-04-19T00:45:27.795414+0000","last_became_peered":"2024-04-19T00:45:27.795414+0000","last_unstale":"2024-04-19T01:03:40.683854+0000","last_undegraded":"2024-04-19T01:03:40.683854+0000","last_fullsized":"2024-04-19T01:03:40.683854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:42:29.849497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404539+0000","last_change":"2024-04-19T00:45:27.795299+0000","last_active":"2024-04-19T01:03:42.404539+0000","last_peered":"2024-04-19T01:03:42.404539+0000","last_clean":"2024-04-19T01:03:42.404539+0000","last_became_active":"2024-04-19T00:45:27.790389+0000","last_became_peered":"2024-04-19T00:45:27.790389+0000","last_unstale":"2024-04-19T01:03:42.404539+0000","last_undegraded":"2024-04-19T01:03:42.404539+0000","last_fullsized":"2024-04-19T01:03:42.404539+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:54:10.325529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403468+0000","last_change":"2024-04-19T00:45:27.738033+0000","last_active":"2024-04-19T01:03:42.403468+0000","last_peered":"2024-04-19T01:03:42.403468+0000","last_clean":"2024-04-19T01:03:42.403468+0000","last_became_active":"2024-04-19T00:45:27.735932+0000","last_became_peered":"2024-04-19T00:45:27.735932+0000","last_unstale":"2024-04-19T01:03:42.403468+0000","last_undegraded":"2024-04-19T01:03:42.403468+0000","last_fullsized":"2024-04-19T01:03:42.403468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:35:41.884097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1364,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403517+0000","last_change":"2024-04-19T00:45:27.765213+0000","last_active":"2024-04-19T01:03:42.403517+0000","last_peered":"2024-04-19T01:03:42.403517+0000","last_clean":"2024-04-19T01:03:42.403517+0000","last_became_active":"2024-04-19T00:45:27.752978+0000","last_became_peered":"2024-04-19T00:45:27.752978+0000","last_unstale":"2024-04-19T01:03:42.403517+0000","last_undegraded":"2024-04-19T01:03:42.403517+0000","last_fullsized":"2024-04-19T01:03:42.403517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:55:11.690354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"17'7","reported_seq":1376,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403206+0000","last_change":"2024-04-19T00:45:27.805132+0000","last_active":"2024-04-19T01:03:42.403206+0000","last_peered":"2024-04-19T01:03:42.403206+0000","last_clean":"2024-04-19T01:03:42.403206+0000","last_became_active":"2024-04-19T00:45:27.798161+0000","last_became_peered":"2024-04-19T00:45:27.798161+0000","last_unstale":"2024-04-19T01:03:42.403206+0000","last_undegraded":"2024-04-19T01:03:42.403206+0000","last_fullsized":"2024-04-19T01:03:42.403206+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:51:44.398144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683194+0000","last_change":"2024-04-19T00:45:27.799286+0000","last_active":"2024-04-19T01:03:40.683194+0000","last_peered":"2024-04-19T01:03:40.683194+0000","last_clean":"2024-04-19T01:03:40.683194+0000","last_became_active":"2024-04-19T00:45:27.797033+0000","last_became_peered":"2024-04-19T00:45:27.797033+0000","last_unstale":"2024-04-19T01:03:40.683194+0000","last_undegraded":"2024-04-19T01:03:40.683194+0000","last_fullsized":"2024-04-19T01:03:40.683194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:04:40.412420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403776+0000","last_change":"2024-04-19T00:45:27.760256+0000","last_active":"2024-04-19T01:03:42.403776+0000","last_peered":"2024-04-19T01:03:42.403776+0000","last_clean":"2024-04-19T01:03:42.403776+0000","last_became_active":"2024-04-19T00:45:27.757597+0000","last_became_peered":"2024-04-19T00:45:27.757597+0000","last_unstale":"2024-04-19T01:03:42.403776+0000","last_undegraded":"2024-04-19T01:03:42.403776+0000","last_fullsized":"2024-04-19T01:03:42.403776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:41:10.030457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978226+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T01:03:41.978226+0000","last_peered":"2024-04-19T01:03:41.978226+0000","last_clean":"2024-04-19T01:03:41.978226+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T01:03:41.978226+0000","last_undegraded":"2024-04-19T01:03:41.978226+0000","last_fullsized":"2024-04-19T01:03:41.978226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:18:46.046655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777648+0000","last_change":"2024-04-19T00:45:27.742732+0000","last_active":"2024-04-19T01:03:41.777648+0000","last_peered":"2024-04-19T01:03:41.777648+0000","last_clean":"2024-04-19T01:03:41.777648+0000","last_became_active":"2024-04-19T00:45:27.740409+0000","last_became_peered":"2024-04-19T00:45:27.740409+0000","last_unstale":"2024-04-19T01:03:41.777648+0000","last_undegraded":"2024-04-19T01:03:41.777648+0000","last_fullsized":"2024-04-19T01:03:41.777648+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:13:15.654443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.778149+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T01:03:41.778149+0000","last_peered":"2024-04-19T01:03:41.778149+0000","last_clean":"2024-04-19T01:03:41.778149+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T01:03:41.778149+0000","last_undegraded":"2024-04-19T01:03:41.778149+0000","last_fullsized":"2024-04-19T01:03:41.778149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777602+0000","last_change":"2024-04-19T00:45:27.679388+0000","last_active":"2024-04-19T01:03:41.777602+0000","last_peered":"2024-04-19T01:03:41.777602+0000","last_clean":"2024-04-19T01:03:41.777602+0000","last_became_active":"2024-04-19T00:45:27.677388+0000","last_became_peered":"2024-04-19T00:45:27.677388+0000","last_unstale":"2024-04-19T01:03:41.777602+0000","last_undegraded":"2024-04-19T01:03:41.777602+0000","last_fullsized":"2024-04-19T01:03:41.777602+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:57:14.716992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683573+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T01:03:40.683573+0000","last_peered":"2024-04-19T01:03:40.683573+0000","last_clean":"2024-04-19T01:03:40.683573+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T01:03:40.683573+0000","last_undegraded":"2024-04-19T01:03:40.683573+0000","last_fullsized":"2024-04-19T01:03:40.683573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:48:30.196169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404298+0000","last_change":"2024-04-19T00:45:27.785476+0000","last_active":"2024-04-19T01:03:42.404298+0000","last_peered":"2024-04-19T01:03:42.404298+0000","last_clean":"2024-04-19T01:03:42.404298+0000","last_became_active":"2024-04-19T00:45:27.785226+0000","last_became_peered":"2024-04-19T00:45:27.785226+0000","last_unstale":"2024-04-19T01:03:42.404298+0000","last_undegraded":"2024-04-19T01:03:42.404298+0000","last_fullsized":"2024-04-19T01:03:42.404298+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:32:30.269561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403439+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T01:03:42.403439+0000","last_peered":"2024-04-19T01:03:42.403439+0000","last_clean":"2024-04-19T01:03:42.403439+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T01:03:42.403439+0000","last_undegraded":"2024-04-19T01:03:42.403439+0000","last_fullsized":"2024-04-19T01:03:42.403439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":1537,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977832+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T01:03:41.977832+0000","last_peered":"2024-04-19T01:03:41.977832+0000","last_clean":"2024-04-19T01:03:41.977832+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T01:03:41.977832+0000","last_undegraded":"2024-04-19T01:03:41.977832+0000","last_fullsized":"2024-04-19T01:03:41.977832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:02:45.312062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403239+0000","last_change":"2024-04-19T00:45:27.821135+0000","last_active":"2024-04-19T01:03:42.403239+0000","last_peered":"2024-04-19T01:03:42.403239+0000","last_clean":"2024-04-19T01:03:42.403239+0000","last_became_active":"2024-04-19T00:45:27.813538+0000","last_became_peered":"2024-04-19T00:45:27.813538+0000","last_unstale":"2024-04-19T01:03:42.403239+0000","last_undegraded":"2024-04-19T01:03:42.403239+0000","last_fullsized":"2024-04-19T01:03:42.403239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:20:22.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403483+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T01:03:42.403483+0000","last_peered":"2024-04-19T01:03:42.403483+0000","last_clean":"2024-04-19T01:03:42.403483+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T01:03:42.403483+0000","last_undegraded":"2024-04-19T01:03:42.403483+0000","last_fullsized":"2024-04-19T01:03:42.403483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777775+0000","last_change":"2024-04-19T00:45:27.763529+0000","last_active":"2024-04-19T01:03:41.777775+0000","last_peered":"2024-04-19T01:03:41.777775+0000","last_clean":"2024-04-19T01:03:41.777775+0000","last_became_active":"2024-04-19T00:45:27.763230+0000","last_became_peered":"2024-04-19T00:45:27.763230+0000","last_unstale":"2024-04-19T01:03:41.777775+0000","last_undegraded":"2024-04-19T01:03:41.777775+0000","last_fullsized":"2024-04-19T01:03:41.777775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:22:57.898965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683667+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T01:03:40.683667+0000","last_peered":"2024-04-19T01:03:40.683667+0000","last_clean":"2024-04-19T01:03:40.683667+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T01:03:40.683667+0000","last_undegraded":"2024-04-19T01:03:40.683667+0000","last_fullsized":"2024-04-19T01:03:40.683667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:04:03.225536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404316+0000","last_change":"2024-04-19T00:45:27.647807+0000","last_active":"2024-04-19T01:03:42.404316+0000","last_peered":"2024-04-19T01:03:42.404316+0000","last_clean":"2024-04-19T01:03:42.404316+0000","last_became_active":"2024-04-19T00:45:27.647106+0000","last_became_peered":"2024-04-19T00:45:27.647106+0000","last_unstale":"2024-04-19T01:03:42.404316+0000","last_undegraded":"2024-04-19T01:03:42.404316+0000","last_fullsized":"2024-04-19T01:03:42.404316+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:21:15.104447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978360+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T01:03:41.978360+0000","last_peered":"2024-04-19T01:03:41.978360+0000","last_clean":"2024-04-19T01:03:41.978360+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T01:03:41.978360+0000","last_undegraded":"2024-04-19T01:03:41.978360+0000","last_fullsized":"2024-04-19T01:03:41.978360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:36:04.048348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404715+0000","last_change":"2024-04-19T00:45:27.659480+0000","last_active":"2024-04-19T01:03:42.404715+0000","last_peered":"2024-04-19T01:03:42.404715+0000","last_clean":"2024-04-19T01:03:42.404715+0000","last_became_active":"2024-04-19T00:45:27.659197+0000","last_became_peered":"2024-04-19T00:45:27.659197+0000","last_unstale":"2024-04-19T01:03:42.404715+0000","last_undegraded":"2024-04-19T01:03:42.404715+0000","last_fullsized":"2024-04-19T01:03:42.404715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:55:55.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777965+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T01:03:41.777965+0000","last_peered":"2024-04-19T01:03:41.777965+0000","last_clean":"2024-04-19T01:03:41.777965+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T01:03:41.777965+0000","last_undegraded":"2024-04-19T01:03:41.777965+0000","last_fullsized":"2024-04-19T01:03:41.777965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404412+0000","last_change":"2024-04-19T00:45:27.661883+0000","last_active":"2024-04-19T01:03:42.404412+0000","last_peered":"2024-04-19T01:03:42.404412+0000","last_clean":"2024-04-19T01:03:42.404412+0000","last_became_active":"2024-04-19T00:45:27.661535+0000","last_became_peered":"2024-04-19T00:45:27.661535+0000","last_unstale":"2024-04-19T01:03:42.404412+0000","last_undegraded":"2024-04-19T01:03:42.404412+0000","last_fullsized":"2024-04-19T01:03:42.404412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:46:16.741442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404690+0000","last_change":"2024-04-19T00:45:27.722242+0000","last_active":"2024-04-19T01:03:42.404690+0000","last_peered":"2024-04-19T01:03:42.404690+0000","last_clean":"2024-04-19T01:03:42.404690+0000","last_became_active":"2024-04-19T00:45:27.719717+0000","last_became_peered":"2024-04-19T00:45:27.719717+0000","last_unstale":"2024-04-19T01:03:42.404690+0000","last_undegraded":"2024-04-19T01:03:42.404690+0000","last_fullsized":"2024-04-19T01:03:42.404690+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:28:43.370123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403826+0000","last_change":"2024-04-19T00:45:27.725269+0000","last_active":"2024-04-19T01:03:42.403826+0000","last_peered":"2024-04-19T01:03:42.403826+0000","last_clean":"2024-04-19T01:03:42.403826+0000","last_became_active":"2024-04-19T00:45:27.719793+0000","last_became_peered":"2024-04-19T00:45:27.719793+0000","last_unstale":"2024-04-19T01:03:42.403826+0000","last_undegraded":"2024-04-19T01:03:42.403826+0000","last_fullsized":"2024-04-19T01:03:42.403826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:05:34.645736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404841+0000","last_change":"2024-04-19T00:45:27.770635+0000","last_active":"2024-04-19T01:03:42.404841+0000","last_peered":"2024-04-19T01:03:42.404841+0000","last_clean":"2024-04-19T01:03:42.404841+0000","last_became_active":"2024-04-19T00:45:27.769635+0000","last_became_peered":"2024-04-19T00:45:27.769635+0000","last_unstale":"2024-04-19T01:03:42.404841+0000","last_undegraded":"2024-04-19T01:03:42.404841+0000","last_fullsized":"2024-04-19T01:03:42.404841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:30:12.981643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777809+0000","last_change":"2024-04-19T00:45:27.752249+0000","last_active":"2024-04-19T01:03:41.777809+0000","last_peered":"2024-04-19T01:03:41.777809+0000","last_clean":"2024-04-19T01:03:41.777809+0000","last_became_active":"2024-04-19T00:45:27.751895+0000","last_became_peered":"2024-04-19T00:45:27.751895+0000","last_unstale":"2024-04-19T01:03:41.777809+0000","last_undegraded":"2024-04-19T01:03:41.777809+0000","last_fullsized":"2024-04-19T01:03:41.777809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:19:10.348532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403676+0000","last_change":"2024-04-19T00:45:27.813207+0000","last_active":"2024-04-19T01:03:42.403676+0000","last_peered":"2024-04-19T01:03:42.403676+0000","last_clean":"2024-04-19T01:03:42.403676+0000","last_became_active":"2024-04-19T00:45:27.812520+0000","last_became_peered":"2024-04-19T00:45:27.812520+0000","last_unstale":"2024-04-19T01:03:42.403676+0000","last_undegraded":"2024-04-19T01:03:42.403676+0000","last_fullsized":"2024-04-19T01:03:42.403676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:21:29.786709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977870+0000","last_change":"2024-04-19T00:45:27.796325+0000","last_active":"2024-04-19T01:03:41.977870+0000","last_peered":"2024-04-19T01:03:41.977870+0000","last_clean":"2024-04-19T01:03:41.977870+0000","last_became_active":"2024-04-19T00:45:27.795698+0000","last_became_peered":"2024-04-19T00:45:27.795698+0000","last_unstale":"2024-04-19T01:03:41.977870+0000","last_undegraded":"2024-04-19T01:03:41.977870+0000","last_fullsized":"2024-04-19T01:03:41.977870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:16:23.414716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978229+0000","last_change":"2024-04-19T00:45:27.750891+0000","last_active":"2024-04-19T01:03:41.978229+0000","last_peered":"2024-04-19T01:03:41.978229+0000","last_clean":"2024-04-19T01:03:41.978229+0000","last_became_active":"2024-04-19T00:45:27.750623+0000","last_became_peered":"2024-04-19T00:45:27.750623+0000","last_unstale":"2024-04-19T01:03:41.978229+0000","last_undegraded":"2024-04-19T01:03:41.978229+0000","last_fullsized":"2024-04-19T01:03:41.978229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:14:11.322418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977779+0000","last_change":"2024-04-19T00:45:27.758851+0000","last_active":"2024-04-19T01:03:41.977779+0000","last_peered":"2024-04-19T01:03:41.977779+0000","last_clean":"2024-04-19T01:03:41.977779+0000","last_became_active":"2024-04-19T00:45:27.758559+0000","last_became_peered":"2024-04-19T00:45:27.758559+0000","last_unstale":"2024-04-19T01:03:41.977779+0000","last_undegraded":"2024-04-19T01:03:41.977779+0000","last_fullsized":"2024-04-19T01:03:41.977779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:04:39.957325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978326+0000","last_change":"2024-04-19T00:45:27.701325+0000","last_active":"2024-04-19T01:03:41.978326+0000","last_peered":"2024-04-19T01:03:41.978326+0000","last_clean":"2024-04-19T01:03:41.978326+0000","last_became_active":"2024-04-19T00:45:27.697796+0000","last_became_peered":"2024-04-19T00:45:27.697796+0000","last_unstale":"2024-04-19T01:03:41.978326+0000","last_undegraded":"2024-04-19T01:03:41.978326+0000","last_fullsized":"2024-04-19T01:03:41.978326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:38:52.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403581+0000","last_change":"2024-04-19T00:45:27.815387+0000","last_active":"2024-04-19T01:03:42.403581+0000","last_peered":"2024-04-19T01:03:42.403581+0000","last_clean":"2024-04-19T01:03:42.403581+0000","last_became_active":"2024-04-19T00:45:27.814761+0000","last_became_peered":"2024-04-19T00:45:27.814761+0000","last_unstale":"2024-04-19T01:03:42.403581+0000","last_undegraded":"2024-04-19T01:03:42.403581+0000","last_fullsized":"2024-04-19T01:03:42.403581+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:29:10.459966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978040+0000","last_change":"2024-04-19T00:45:27.781966+0000","last_active":"2024-04-19T01:03:41.978040+0000","last_peered":"2024-04-19T01:03:41.978040+0000","last_clean":"2024-04-19T01:03:41.978040+0000","last_became_active":"2024-04-19T00:45:27.779682+0000","last_became_peered":"2024-04-19T00:45:27.779682+0000","last_unstale":"2024-04-19T01:03:41.978040+0000","last_undegraded":"2024-04-19T01:03:41.978040+0000","last_fullsized":"2024-04-19T01:03:41.978040+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:45:17.983232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683489+0000","last_change":"2024-04-19T00:45:27.790124+0000","last_active":"2024-04-19T01:03:40.683489+0000","last_peered":"2024-04-19T01:03:40.683489+0000","last_clean":"2024-04-19T01:03:40.683489+0000","last_became_active":"2024-04-19T00:45:27.788325+0000","last_became_peered":"2024-04-19T00:45:27.788325+0000","last_unstale":"2024-04-19T01:03:40.683489+0000","last_undegraded":"2024-04-19T01:03:40.683489+0000","last_fullsized":"2024-04-19T01:03:40.683489+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:16:28.463405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607809,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607808,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607807,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75800,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296040,"statfs":{"total":96636764160,"available":96559144960,"internally_reserved":0,"allocated":4468736,"data_stored":2184837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640512,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79440,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292400,"statfs":{"total":96636764160,"available":96555417600,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71644,"kb_used_data":4368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300196,"statfs":{"total":96636764160,"available":96563400704,"internally_reserved":0,"allocated":4472832,"data_stored":2184842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640510,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71124,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300716,"statfs":{"total":96636764160,"available":96563933184,"internally_reserved":0,"allocated":3940352,"data_stored":1655914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:29.998 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-19T01:05:29.998 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 10 monclient: get_monmap_and_config 2024-04-19T01:05:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 10 monclient: build_initial_monmap 2024-04-19T01:05:30.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 10 monclient: monmap: 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:30.124927+0000 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:30.124927+0000 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:30.135 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:30.136 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 1 Processor -- start 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 1 -- start start 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 10 monclient: init 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.122+0000 7f4f39ac1640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient: _add_conns ranks=[3,6,2,10,7,8,9,5,4,0,1] 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f341390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): picked mon.noname-g con 0x7f4f34138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:30.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 0x7f4f341395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): picked mon.noname-d con 0x7f4f340a5b80 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 0x7f4f34135d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): picked mon.noname-b con 0x7f4f341358b0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f4f340a4410 con 0x7f4f341358b0 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4f34139b30 con 0x7f4f34138cd0 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f4f340a3dc0 con 0x7f4f340a5b80 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient(hunting): authenticate will time out at 2200.252137s 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 0x7f4f341395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 0x7f4f34135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 0x7f4f341395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:36748/0 (socket says 172.21.15.43:36748) 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 0x7f4f34135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:47244/0 (socket says 172.21.15.43:47244) 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 -- 172.21.15.43:0/3848314316 learned_addr learned my addr 172.21.15.43:0/3848314316 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): get_auth_request con 0x7f4f340a5b80 auth_method 0 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): get_auth_request con 0x7f4f341358b0 auth_method 0 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.139 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f337fe640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f341390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f337fe640 10 monclient(hunting): get_auth_request con 0x7f4f34138cd0 auth_method 0 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f337fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_done global_id 184218 payload 274 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 -- 172.21.15.43:0/3848314316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 msgr2=0x7f4f341390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f341390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 -- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 msgr2=0x7f4f34135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 0x7f4f34135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.140 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 monclient: found mon.noname-d 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 -- 172.21.15.43:0/3848314316 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f340a40a0 con 0x7f4f340a5b80 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f33fff640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 0x7f4f34135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f32ffd640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 0x7f4f341395f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0098a0 tx=0x7f4f1c01c3e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e22c10286d04ddb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 1 -- 172.21.15.43:0/3848314316 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f1c0b0070 con 0x7f4f340a5b80 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: dump: 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.141 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.142 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.143 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:30.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:30.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:30.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 1 -- 172.21.15.43:0/3848314316 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f1c0aa810 con 0x7f4f340a5b80 2024-04-19T01:05:30.143 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 5 monclient: authenticate success, global_id 184218 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 1 -- 172.21.15.43:0/3848314316 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f1c0a9300 con 0x7f4f340a5b80 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 4 set_mon_vals no callback set 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f327fc640 10 monclient: dump: 2024-04-19T01:05:30.144 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.145 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.146 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 10 monclient: shutdown 2024-04-19T01:05:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 msgr2=0x7f4f341395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.146 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 0x7f4f341395f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0098a0 tx=0x7f4f1c01c3e0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- 172.21.15.43:0/3848314316 shutdown_connections 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f4f341358b0 0x7f4f34135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f4f340a5b80 0x7f4f341395f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/3848314316 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f341390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- 172.21.15.43:0/3848314316 >> 172.21.15.43:0/3848314316 conn(0x7f4f340ab2f0 msgr2=0x7f4f340ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- 172.21.15.43:0/3848314316 shutdown_connections 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.126+0000 7f4f39ac1640 1 -- 172.21.15.43:0/3848314316 wait complete. 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: build_initial_monmap 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: monmap: 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:30.147 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:30.148 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 librados: starting msgr at 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 librados: starting objecter 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 Processor -- start 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 -- start start 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 librados: setting wanted keys 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 librados: calling monclient init 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: init 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: _add_conns ranks=[8,0,3,7,6,4,1,2,5] 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 0x7f4f341d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): picked mon.noname-e con 0x7f4f340a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f341358b0 0x7f4f340a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.149 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): picked mon.noname-a con 0x7f4f341358b0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f340a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): picked mon.noname-g con 0x7f4f34138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f4f340a2010 con 0x7f4f341358b0 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f4f340a1e90 con 0x7f4f34138cd0 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f4f34142aa0 con 0x7f4f340a5b80 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:30.150 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient(hunting): authenticate will time out at 2200.258072s 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 0x7f4f341d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f341358b0 0x7f4f340a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 0x7f4f341d0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:57992/0 (socket says 172.21.15.43:57992) 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 -- 172.21.15.43:0/2818791675 learned_addr learned my addr 172.21.15.43:0/2818791675 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): get_auth_request con 0x7f4f340a5b80 auth_method 0 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): get_auth_request con 0x7f4f341358b0 auth_method 0 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.151 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f33fff640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f340a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f33fff640 10 monclient(hunting): get_auth_request con 0x7f4f34138cd0 auth_method 0 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.152 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f33fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient(hunting): handle_auth_done global_id 144692 payload 824 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 -- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 msgr2=0x7f4f340a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f340a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 -- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f341358b0 msgr2=0x7f4f340a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f341358b0 0x7f4f340a0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient: _finish_hunting 0 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 monclient: found mon.noname-e 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 -- 172.21.15.43:0/2818791675 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3409d4f0 con 0x7f4f340a5b80 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f32ffd640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f341358b0 0x7f4f340a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f337fe640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 0x7f4f341d0d30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400d620 tx=0x7f4f2400dae0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:30.153 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 client.?.objecter ms_handle_connect 0x7f4f340a5b80 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f24007380 con 0x7f4f340a5b80 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: dump: 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.154 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: _finish_auth 0 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:30.155 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f24019070 con 0x7f4f340a5b80 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f4f24014430 con 0x7f4f340a5b80 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f2bfff640 10 monclient: dump: 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.156 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.157 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f31ffb640 4 set_mon_vals no callback set 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 5 monclient: authenticate success, global_id 144692 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: _renew_subs 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f340a3590 con 0x7f4f340a5b80 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 client.144692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: _renew_subs 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f34134fe0 con 0x7f4f340a5b80 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.130+0000 7f4f39ac1640 1 librados: init done 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f4f2400b620 con 0x7f4f340a5b80 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f39ac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f39ac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f39ac1640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.158 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f340693b0 con 0x7f4f340a5b80 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4f14038e10 0x7f4f1403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f4f2400f040 con 0x7f4f340a5b80 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 10 client.144692.objecter ms_dispatch 0x7f4f34000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 3 client.144692.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 3 client.144692.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 20 client.144692.objecter dump_active .. 0 homeless 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f32ffd640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4f14038e10 0x7f4f1403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f32ffd640 10 monclient: get_auth_request con 0x7f4f14038e10 auth_method 0 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f32ffd640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4f14038e10 0x7f4f1403b2d0 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0ad000 tx=0x7f4f1c0a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:30.159 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.134+0000 7f4f2bfff640 10 client.144692.objecter ms_handle_connect 0x7f4f14038e10 2024-04-19T01:05:30.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.138+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4f24014950 con 0x7f4f340a5b80 2024-04-19T01:05:30.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.138+0000 7f4f2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:30.160 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.138+0000 7f4f2bfff640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:30.294 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.294+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4f3409d050 con 0x7f4f14038e10 2024-04-19T01:05:30.302 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.302+0000 7f4f2bfff640 1 -- 172.21.15.43:0/2818791675 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113902 (secure 0 0 0) 0x7f4f1c004140 con 0x7f4f14038e10 2024-04-19T01:05:30.303 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T01:05:30.304 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 10 librados: watch_flush enter 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 10 librados: watch_flush exit 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 20 client.144692.objecter shutdown clearing up homeless session... 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 10 client.144692.objecter successfully canceled tick 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4f14038e10 msgr2=0x7f4f1403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4f14038e10 0x7f4f1403b2d0 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0ad000 tx=0x7f4f1c0a4f90 comp rx=0 tx=0).stop 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 10 monclient: shutdown 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 msgr2=0x7f4f341d0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 0x7f4f341d0d30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4f2400d620 tx=0x7f4f2400dae0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 shutdown_connections 2024-04-19T01:05:30.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f4f34138cd0 0x7f4f340a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f4f341358b0 0x7f4f340a0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f4f14038e10 0x7f4f1403b2d0 unknown :-1 s=CLOSED pgs=3674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 --2- 172.21.15.43:0/2818791675 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f4f340a5b80 0x7f4f341d0d30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 >> 172.21.15.43:0/2818791675 conn(0x7f4f340ab2f0 msgr2=0x7f4f340a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 shutdown_connections 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 -- 172.21.15.43:0/2818791675 wait complete. 2024-04-19T01:05:30.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.306+0000 7f4f39ac1640 1 librados: shutdown 2024-04-19T01:05:30.321 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1328,"stamp":"2024-04-19T01:05:29.332359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":448576,"kb_used_data":24112,"kb_used_omap":9,"kb_used_meta":424374,"kb_avail":565782464,"statfs":{"total":579820584960,"available":579361243136,"internally_reserved":0,"allocated":24690688,"data_stored":10993320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":434559649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004097"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683169+0000","last_change":"2024-04-19T00:45:27.797265+0000","last_active":"2024-04-19T01:03:40.683169+0000","last_peered":"2024-04-19T01:03:40.683169+0000","last_clean":"2024-04-19T01:03:40.683169+0000","last_became_active":"2024-04-19T00:45:27.796802+0000","last_became_peered":"2024-04-19T00:45:27.796802+0000","last_unstale":"2024-04-19T01:03:40.683169+0000","last_undegraded":"2024-04-19T01:03:40.683169+0000","last_fullsized":"2024-04-19T01:03:40.683169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:39:20.387401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777710+0000","last_change":"2024-04-19T00:45:27.717179+0000","last_active":"2024-04-19T01:03:41.777710+0000","last_peered":"2024-04-19T01:03:41.777710+0000","last_clean":"2024-04-19T01:03:41.777710+0000","last_became_active":"2024-04-19T00:45:27.715591+0000","last_became_peered":"2024-04-19T00:45:27.715591+0000","last_unstale":"2024-04-19T01:03:41.777710+0000","last_undegraded":"2024-04-19T01:03:41.777710+0000","last_fullsized":"2024-04-19T01:03:41.777710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:12:05.981056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404350+0000","last_change":"2024-04-19T00:45:27.745246+0000","last_active":"2024-04-19T01:03:42.404350+0000","last_peered":"2024-04-19T01:03:42.404350+0000","last_clean":"2024-04-19T01:03:42.404350+0000","last_became_active":"2024-04-19T00:45:27.744986+0000","last_became_peered":"2024-04-19T00:45:27.744986+0000","last_unstale":"2024-04-19T01:03:42.404350+0000","last_undegraded":"2024-04-19T01:03:42.404350+0000","last_fullsized":"2024-04-19T01:03:42.404350+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:47:38.164550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683470+0000","last_change":"2024-04-19T00:45:27.767266+0000","last_active":"2024-04-19T01:03:40.683470+0000","last_peered":"2024-04-19T01:03:40.683470+0000","last_clean":"2024-04-19T01:03:40.683470+0000","last_became_active":"2024-04-19T00:45:27.767040+0000","last_became_peered":"2024-04-19T00:45:27.767040+0000","last_unstale":"2024-04-19T01:03:40.683470+0000","last_undegraded":"2024-04-19T01:03:40.683470+0000","last_fullsized":"2024-04-19T01:03:40.683470+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:43:32.043488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777937+0000","last_change":"2024-04-19T00:45:27.717564+0000","last_active":"2024-04-19T01:03:41.777937+0000","last_peered":"2024-04-19T01:03:41.777937+0000","last_clean":"2024-04-19T01:03:41.777937+0000","last_became_active":"2024-04-19T00:45:27.717286+0000","last_became_peered":"2024-04-19T00:45:27.717286+0000","last_unstale":"2024-04-19T01:03:41.777937+0000","last_undegraded":"2024-04-19T01:03:41.777937+0000","last_fullsized":"2024-04-19T01:03:41.777937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:43:17.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683854+0000","last_change":"2024-04-19T00:45:27.799104+0000","last_active":"2024-04-19T01:03:40.683854+0000","last_peered":"2024-04-19T01:03:40.683854+0000","last_clean":"2024-04-19T01:03:40.683854+0000","last_became_active":"2024-04-19T00:45:27.795414+0000","last_became_peered":"2024-04-19T00:45:27.795414+0000","last_unstale":"2024-04-19T01:03:40.683854+0000","last_undegraded":"2024-04-19T01:03:40.683854+0000","last_fullsized":"2024-04-19T01:03:40.683854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:42:29.849497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404539+0000","last_change":"2024-04-19T00:45:27.795299+0000","last_active":"2024-04-19T01:03:42.404539+0000","last_peered":"2024-04-19T01:03:42.404539+0000","last_clean":"2024-04-19T01:03:42.404539+0000","last_became_active":"2024-04-19T00:45:27.790389+0000","last_became_peered":"2024-04-19T00:45:27.790389+0000","last_unstale":"2024-04-19T01:03:42.404539+0000","last_undegraded":"2024-04-19T01:03:42.404539+0000","last_fullsized":"2024-04-19T01:03:42.404539+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:54:10.325529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403468+0000","last_change":"2024-04-19T00:45:27.738033+0000","last_active":"2024-04-19T01:03:42.403468+0000","last_peered":"2024-04-19T01:03:42.403468+0000","last_clean":"2024-04-19T01:03:42.403468+0000","last_became_active":"2024-04-19T00:45:27.735932+0000","last_became_peered":"2024-04-19T00:45:27.735932+0000","last_unstale":"2024-04-19T01:03:42.403468+0000","last_undegraded":"2024-04-19T01:03:42.403468+0000","last_fullsized":"2024-04-19T01:03:42.403468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:35:41.884097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1364,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403517+0000","last_change":"2024-04-19T00:45:27.765213+0000","last_active":"2024-04-19T01:03:42.403517+0000","last_peered":"2024-04-19T01:03:42.403517+0000","last_clean":"2024-04-19T01:03:42.403517+0000","last_became_active":"2024-04-19T00:45:27.752978+0000","last_became_peered":"2024-04-19T00:45:27.752978+0000","last_unstale":"2024-04-19T01:03:42.403517+0000","last_undegraded":"2024-04-19T01:03:42.403517+0000","last_fullsized":"2024-04-19T01:03:42.403517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:55:11.690354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"17'7","reported_seq":1376,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403206+0000","last_change":"2024-04-19T00:45:27.805132+0000","last_active":"2024-04-19T01:03:42.403206+0000","last_peered":"2024-04-19T01:03:42.403206+0000","last_clean":"2024-04-19T01:03:42.403206+0000","last_became_active":"2024-04-19T00:45:27.798161+0000","last_became_peered":"2024-04-19T00:45:27.798161+0000","last_unstale":"2024-04-19T01:03:42.403206+0000","last_undegraded":"2024-04-19T01:03:42.403206+0000","last_fullsized":"2024-04-19T01:03:42.403206+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:51:44.398144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683194+0000","last_change":"2024-04-19T00:45:27.799286+0000","last_active":"2024-04-19T01:03:40.683194+0000","last_peered":"2024-04-19T01:03:40.683194+0000","last_clean":"2024-04-19T01:03:40.683194+0000","last_became_active":"2024-04-19T00:45:27.797033+0000","last_became_peered":"2024-04-19T00:45:27.797033+0000","last_unstale":"2024-04-19T01:03:40.683194+0000","last_undegraded":"2024-04-19T01:03:40.683194+0000","last_fullsized":"2024-04-19T01:03:40.683194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:04:40.412420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403776+0000","last_change":"2024-04-19T00:45:27.760256+0000","last_active":"2024-04-19T01:03:42.403776+0000","last_peered":"2024-04-19T01:03:42.403776+0000","last_clean":"2024-04-19T01:03:42.403776+0000","last_became_active":"2024-04-19T00:45:27.757597+0000","last_became_peered":"2024-04-19T00:45:27.757597+0000","last_unstale":"2024-04-19T01:03:42.403776+0000","last_undegraded":"2024-04-19T01:03:42.403776+0000","last_fullsized":"2024-04-19T01:03:42.403776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:41:10.030457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978226+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T01:03:41.978226+0000","last_peered":"2024-04-19T01:03:41.978226+0000","last_clean":"2024-04-19T01:03:41.978226+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T01:03:41.978226+0000","last_undegraded":"2024-04-19T01:03:41.978226+0000","last_fullsized":"2024-04-19T01:03:41.978226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:18:46.046655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777648+0000","last_change":"2024-04-19T00:45:27.742732+0000","last_active":"2024-04-19T01:03:41.777648+0000","last_peered":"2024-04-19T01:03:41.777648+0000","last_clean":"2024-04-19T01:03:41.777648+0000","last_became_active":"2024-04-19T00:45:27.740409+0000","last_became_peered":"2024-04-19T00:45:27.740409+0000","last_unstale":"2024-04-19T01:03:41.777648+0000","last_undegraded":"2024-04-19T01:03:41.777648+0000","last_fullsized":"2024-04-19T01:03:41.777648+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:13:15.654443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.778149+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T01:03:41.778149+0000","last_peered":"2024-04-19T01:03:41.778149+0000","last_clean":"2024-04-19T01:03:41.778149+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T01:03:41.778149+0000","last_undegraded":"2024-04-19T01:03:41.778149+0000","last_fullsized":"2024-04-19T01:03:41.778149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777602+0000","last_change":"2024-04-19T00:45:27.679388+0000","last_active":"2024-04-19T01:03:41.777602+0000","last_peered":"2024-04-19T01:03:41.777602+0000","last_clean":"2024-04-19T01:03:41.777602+0000","last_became_active":"2024-04-19T00:45:27.677388+0000","last_became_peered":"2024-04-19T00:45:27.677388+0000","last_unstale":"2024-04-19T01:03:41.777602+0000","last_undegraded":"2024-04-19T01:03:41.777602+0000","last_fullsized":"2024-04-19T01:03:41.777602+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:57:14.716992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683573+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T01:03:40.683573+0000","last_peered":"2024-04-19T01:03:40.683573+0000","last_clean":"2024-04-19T01:03:40.683573+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T01:03:40.683573+0000","last_undegraded":"2024-04-19T01:03:40.683573+0000","last_fullsized":"2024-04-19T01:03:40.683573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:48:30.196169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404298+0000","last_change":"2024-04-19T00:45:27.785476+0000","last_active":"2024-04-19T01:03:42.404298+0000","last_peered":"2024-04-19T01:03:42.404298+0000","last_clean":"2024-04-19T01:03:42.404298+0000","last_became_active":"2024-04-19T00:45:27.785226+0000","last_became_peered":"2024-04-19T00:45:27.785226+0000","last_unstale":"2024-04-19T01:03:42.404298+0000","last_undegraded":"2024-04-19T01:03:42.404298+0000","last_fullsized":"2024-04-19T01:03:42.404298+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:32:30.269561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403439+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T01:03:42.403439+0000","last_peered":"2024-04-19T01:03:42.403439+0000","last_clean":"2024-04-19T01:03:42.403439+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T01:03:42.403439+0000","last_undegraded":"2024-04-19T01:03:42.403439+0000","last_fullsized":"2024-04-19T01:03:42.403439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":1537,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977832+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T01:03:41.977832+0000","last_peered":"2024-04-19T01:03:41.977832+0000","last_clean":"2024-04-19T01:03:41.977832+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T01:03:41.977832+0000","last_undegraded":"2024-04-19T01:03:41.977832+0000","last_fullsized":"2024-04-19T01:03:41.977832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:02:45.312062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403239+0000","last_change":"2024-04-19T00:45:27.821135+0000","last_active":"2024-04-19T01:03:42.403239+0000","last_peered":"2024-04-19T01:03:42.403239+0000","last_clean":"2024-04-19T01:03:42.403239+0000","last_became_active":"2024-04-19T00:45:27.813538+0000","last_became_peered":"2024-04-19T00:45:27.813538+0000","last_unstale":"2024-04-19T01:03:42.403239+0000","last_undegraded":"2024-04-19T01:03:42.403239+0000","last_fullsized":"2024-04-19T01:03:42.403239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:20:22.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403483+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T01:03:42.403483+0000","last_peered":"2024-04-19T01:03:42.403483+0000","last_clean":"2024-04-19T01:03:42.403483+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T01:03:42.403483+0000","last_undegraded":"2024-04-19T01:03:42.403483+0000","last_fullsized":"2024-04-19T01:03:42.403483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777775+0000","last_change":"2024-04-19T00:45:27.763529+0000","last_active":"2024-04-19T01:03:41.777775+0000","last_peered":"2024-04-19T01:03:41.777775+0000","last_clean":"2024-04-19T01:03:41.777775+0000","last_became_active":"2024-04-19T00:45:27.763230+0000","last_became_peered":"2024-04-19T00:45:27.763230+0000","last_unstale":"2024-04-19T01:03:41.777775+0000","last_undegraded":"2024-04-19T01:03:41.777775+0000","last_fullsized":"2024-04-19T01:03:41.777775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:22:57.898965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683667+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T01:03:40.683667+0000","last_peered":"2024-04-19T01:03:40.683667+0000","last_clean":"2024-04-19T01:03:40.683667+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T01:03:40.683667+0000","last_undegraded":"2024-04-19T01:03:40.683667+0000","last_fullsized":"2024-04-19T01:03:40.683667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:04:03.225536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404316+0000","last_change":"2024-04-19T00:45:27.647807+0000","last_active":"2024-04-19T01:03:42.404316+0000","last_peered":"2024-04-19T01:03:42.404316+0000","last_clean":"2024-04-19T01:03:42.404316+0000","last_became_active":"2024-04-19T00:45:27.647106+0000","last_became_peered":"2024-04-19T00:45:27.647106+0000","last_unstale":"2024-04-19T01:03:42.404316+0000","last_undegraded":"2024-04-19T01:03:42.404316+0000","last_fullsized":"2024-04-19T01:03:42.404316+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:21:15.104447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978360+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T01:03:41.978360+0000","last_peered":"2024-04-19T01:03:41.978360+0000","last_clean":"2024-04-19T01:03:41.978360+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T01:03:41.978360+0000","last_undegraded":"2024-04-19T01:03:41.978360+0000","last_fullsized":"2024-04-19T01:03:41.978360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:36:04.048348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404715+0000","last_change":"2024-04-19T00:45:27.659480+0000","last_active":"2024-04-19T01:03:42.404715+0000","last_peered":"2024-04-19T01:03:42.404715+0000","last_clean":"2024-04-19T01:03:42.404715+0000","last_became_active":"2024-04-19T00:45:27.659197+0000","last_became_peered":"2024-04-19T00:45:27.659197+0000","last_unstale":"2024-04-19T01:03:42.404715+0000","last_undegraded":"2024-04-19T01:03:42.404715+0000","last_fullsized":"2024-04-19T01:03:42.404715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:55:55.612261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777965+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T01:03:41.777965+0000","last_peered":"2024-04-19T01:03:41.777965+0000","last_clean":"2024-04-19T01:03:41.777965+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T01:03:41.777965+0000","last_undegraded":"2024-04-19T01:03:41.777965+0000","last_fullsized":"2024-04-19T01:03:41.777965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1365,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404412+0000","last_change":"2024-04-19T00:45:27.661883+0000","last_active":"2024-04-19T01:03:42.404412+0000","last_peered":"2024-04-19T01:03:42.404412+0000","last_clean":"2024-04-19T01:03:42.404412+0000","last_became_active":"2024-04-19T00:45:27.661535+0000","last_became_peered":"2024-04-19T00:45:27.661535+0000","last_unstale":"2024-04-19T01:03:42.404412+0000","last_undegraded":"2024-04-19T01:03:42.404412+0000","last_fullsized":"2024-04-19T01:03:42.404412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:46:16.741442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404690+0000","last_change":"2024-04-19T00:45:27.722242+0000","last_active":"2024-04-19T01:03:42.404690+0000","last_peered":"2024-04-19T01:03:42.404690+0000","last_clean":"2024-04-19T01:03:42.404690+0000","last_became_active":"2024-04-19T00:45:27.719717+0000","last_became_peered":"2024-04-19T00:45:27.719717+0000","last_unstale":"2024-04-19T01:03:42.404690+0000","last_undegraded":"2024-04-19T01:03:42.404690+0000","last_fullsized":"2024-04-19T01:03:42.404690+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:28:43.370123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403826+0000","last_change":"2024-04-19T00:45:27.725269+0000","last_active":"2024-04-19T01:03:42.403826+0000","last_peered":"2024-04-19T01:03:42.403826+0000","last_clean":"2024-04-19T01:03:42.403826+0000","last_became_active":"2024-04-19T00:45:27.719793+0000","last_became_peered":"2024-04-19T00:45:27.719793+0000","last_unstale":"2024-04-19T01:03:42.403826+0000","last_undegraded":"2024-04-19T01:03:42.403826+0000","last_fullsized":"2024-04-19T01:03:42.403826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:05:34.645736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404841+0000","last_change":"2024-04-19T00:45:27.770635+0000","last_active":"2024-04-19T01:03:42.404841+0000","last_peered":"2024-04-19T01:03:42.404841+0000","last_clean":"2024-04-19T01:03:42.404841+0000","last_became_active":"2024-04-19T00:45:27.769635+0000","last_became_peered":"2024-04-19T00:45:27.769635+0000","last_unstale":"2024-04-19T01:03:42.404841+0000","last_undegraded":"2024-04-19T01:03:42.404841+0000","last_fullsized":"2024-04-19T01:03:42.404841+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:30:12.981643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777809+0000","last_change":"2024-04-19T00:45:27.752249+0000","last_active":"2024-04-19T01:03:41.777809+0000","last_peered":"2024-04-19T01:03:41.777809+0000","last_clean":"2024-04-19T01:03:41.777809+0000","last_became_active":"2024-04-19T00:45:27.751895+0000","last_became_peered":"2024-04-19T00:45:27.751895+0000","last_unstale":"2024-04-19T01:03:41.777809+0000","last_undegraded":"2024-04-19T01:03:41.777809+0000","last_fullsized":"2024-04-19T01:03:41.777809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:19:10.348532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403676+0000","last_change":"2024-04-19T00:45:27.813207+0000","last_active":"2024-04-19T01:03:42.403676+0000","last_peered":"2024-04-19T01:03:42.403676+0000","last_clean":"2024-04-19T01:03:42.403676+0000","last_became_active":"2024-04-19T00:45:27.812520+0000","last_became_peered":"2024-04-19T00:45:27.812520+0000","last_unstale":"2024-04-19T01:03:42.403676+0000","last_undegraded":"2024-04-19T01:03:42.403676+0000","last_fullsized":"2024-04-19T01:03:42.403676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:21:29.786709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977870+0000","last_change":"2024-04-19T00:45:27.796325+0000","last_active":"2024-04-19T01:03:41.977870+0000","last_peered":"2024-04-19T01:03:41.977870+0000","last_clean":"2024-04-19T01:03:41.977870+0000","last_became_active":"2024-04-19T00:45:27.795698+0000","last_became_peered":"2024-04-19T00:45:27.795698+0000","last_unstale":"2024-04-19T01:03:41.977870+0000","last_undegraded":"2024-04-19T01:03:41.977870+0000","last_fullsized":"2024-04-19T01:03:41.977870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:16:23.414716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978229+0000","last_change":"2024-04-19T00:45:27.750891+0000","last_active":"2024-04-19T01:03:41.978229+0000","last_peered":"2024-04-19T01:03:41.978229+0000","last_clean":"2024-04-19T01:03:41.978229+0000","last_became_active":"2024-04-19T00:45:27.750623+0000","last_became_peered":"2024-04-19T00:45:27.750623+0000","last_unstale":"2024-04-19T01:03:41.978229+0000","last_undegraded":"2024-04-19T01:03:41.978229+0000","last_fullsized":"2024-04-19T01:03:41.978229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:14:11.322418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977779+0000","last_change":"2024-04-19T00:45:27.758851+0000","last_active":"2024-04-19T01:03:41.977779+0000","last_peered":"2024-04-19T01:03:41.977779+0000","last_clean":"2024-04-19T01:03:41.977779+0000","last_became_active":"2024-04-19T00:45:27.758559+0000","last_became_peered":"2024-04-19T00:45:27.758559+0000","last_unstale":"2024-04-19T01:03:41.977779+0000","last_undegraded":"2024-04-19T01:03:41.977779+0000","last_fullsized":"2024-04-19T01:03:41.977779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:04:39.957325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978326+0000","last_change":"2024-04-19T00:45:27.701325+0000","last_active":"2024-04-19T01:03:41.978326+0000","last_peered":"2024-04-19T01:03:41.978326+0000","last_clean":"2024-04-19T01:03:41.978326+0000","last_became_active":"2024-04-19T00:45:27.697796+0000","last_became_peered":"2024-04-19T00:45:27.697796+0000","last_unstale":"2024-04-19T01:03:41.978326+0000","last_undegraded":"2024-04-19T01:03:41.978326+0000","last_fullsized":"2024-04-19T01:03:41.978326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:38:52.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403581+0000","last_change":"2024-04-19T00:45:27.815387+0000","last_active":"2024-04-19T01:03:42.403581+0000","last_peered":"2024-04-19T01:03:42.403581+0000","last_clean":"2024-04-19T01:03:42.403581+0000","last_became_active":"2024-04-19T00:45:27.814761+0000","last_became_peered":"2024-04-19T00:45:27.814761+0000","last_unstale":"2024-04-19T01:03:42.403581+0000","last_undegraded":"2024-04-19T01:03:42.403581+0000","last_fullsized":"2024-04-19T01:03:42.403581+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:29:10.459966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978040+0000","last_change":"2024-04-19T00:45:27.781966+0000","last_active":"2024-04-19T01:03:41.978040+0000","last_peered":"2024-04-19T01:03:41.978040+0000","last_clean":"2024-04-19T01:03:41.978040+0000","last_became_active":"2024-04-19T00:45:27.779682+0000","last_became_peered":"2024-04-19T00:45:27.779682+0000","last_unstale":"2024-04-19T01:03:41.978040+0000","last_undegraded":"2024-04-19T01:03:41.978040+0000","last_fullsized":"2024-04-19T01:03:41.978040+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:45:17.983232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683489+0000","last_change":"2024-04-19T00:45:27.790124+0000","last_active":"2024-04-19T01:03:40.683489+0000","last_peered":"2024-04-19T01:03:40.683489+0000","last_clean":"2024-04-19T01:03:40.683489+0000","last_became_active":"2024-04-19T00:45:27.788325+0000","last_became_peered":"2024-04-19T00:45:27.788325+0000","last_unstale":"2024-04-19T01:03:40.683489+0000","last_undegraded":"2024-04-19T01:03:40.683489+0000","last_fullsized":"2024-04-19T01:03:40.683489+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:16:28.463405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607809,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607808,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607807,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75800,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296040,"statfs":{"total":96636764160,"available":96559144960,"internally_reserved":0,"allocated":4468736,"data_stored":2184837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640512,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79440,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292400,"statfs":{"total":96636764160,"available":96555417600,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640509,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71644,"kb_used_data":4368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300196,"statfs":{"total":96636764160,"available":96563400704,"internally_reserved":0,"allocated":4472832,"data_stored":2184842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640510,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71124,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300716,"statfs":{"total":96636764160,"available":96563933184,"internally_reserved":0,"allocated":3940352,"data_stored":1655914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:30.323 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-19T01:05:30.446 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.446+0000 7f8e98438640 10 monclient: get_monmap_and_config 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.446+0000 7f8e98438640 10 monclient: build_initial_monmap 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.446+0000 7f8e98438640 10 monclient: monmap: 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:30.449628+0000 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:30.449628+0000 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:30.447 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:30.448 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 Processor -- start 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 -- start start 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient: init 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient: _add_conns ranks=[8,9,10,7,2,0,6,5,3,4,1] 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132620 0x7f8e90072f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): picked mon.noname-e con 0x7f8e90132620 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:30.450 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8e90132a00 0x7f8e900a3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): picked mon.noname-a con 0x7f8e90132a00 addr v2:172.21.15.43:3300/0 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8e90132e20 0x7f8e900a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): picked mon.noname-f con 0x7f8e90132e20 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f8e901321d0 con 0x7f8e90132a00 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8e900720d0 con 0x7f8e90132e20 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8e900723b0 con 0x7f8e90132620 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e98438640 10 monclient(hunting): authenticate will time out at 2200.577523s 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f8e90132a00 0x7f8e900a3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132620 0x7f8e90072f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132620 0x7f8e90072f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:58006/0 (socket says 172.21.15.43:58006) 2024-04-19T01:05:30.451 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 1 -- 172.21.15.43:0/3430315254 learned_addr learned my addr 172.21.15.43:0/3430315254 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): get_auth_request con 0x7f8e90132620 auth_method 0 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): get_auth_request con 0x7f8e90132a00 auth_method 0 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e969d5640 1 --2- 172.21.15.43:0/3430315254 >> v2:172.21.15.161:3300/0 conn(0x7f8e90132e20 0x7f8e900a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e961d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e969d5640 10 monclient(hunting): get_auth_request con 0x7f8e90132e20 auth_method 0 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e969d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e969d5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e969d5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient(hunting): handle_auth_done global_id 184194 payload 274 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 -- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132620 msgr2=0x7f8e90072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 --2- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132620 0x7f8e90072f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 -- 172.21.15.43:0/3430315254 >> v2:172.21.15.161:3300/0 conn(0x7f8e90132e20 msgr2=0x7f8e900a38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 --2- 172.21.15.43:0/3430315254 >> v2:172.21.15.161:3300/0 conn(0x7f8e90132e20 0x7f8e900a38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient: _finish_hunting 0 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 monclient: found mon.noname-a 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:30.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.450+0000 7f8e959d3640 1 -- 172.21.15.43:0/3430315254 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e90072690 con 0x7f8e90132a00 2024-04-19T01:05:30.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e959d3640 1 --2- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e90132a00 0x7f8e900a3370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e90073560 tx=0x7f8e8001c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f514ebf88cd02cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:30.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 1 -- 172.21.15.43:0/3430315254 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e800b0070 con 0x7f8e90132a00 2024-04-19T01:05:30.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:30.462 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: dump: 2024-04-19T01:05:30.463 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.464 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.464 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.464 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.464 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.464 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.464 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: _finish_auth 0 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: _check_auth_tickets 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 5 monclient: authenticate success, global_id 184194 2024-04-19T01:05:30.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 1 -- 172.21.15.43:0/3430315254 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e800aa810 con 0x7f8e90132a00 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 1 -- 172.21.15.43:0/3430315254 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e800a9300 con 0x7f8e90132a00 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 4 set_mon_vals no callback set 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e951d2640 10 monclient: dump: 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.466 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: shutdown 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e90132a00 msgr2=0x7f8e900a3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e90132a00 0x7f8e900a3370 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8e90073560 tx=0x7f8e8001c2e0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- 172.21.15.43:0/3430315254 shutdown_connections 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- 172.21.15.43:0/3430315254 >> v2:172.21.15.161:3300/0 conn(0x7f8e90132e20 0x7f8e900a38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f8e90132a00 0x7f8e900a3370 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- 172.21.15.43:0/3430315254 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132620 0x7f8e90072f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- 172.21.15.43:0/3430315254 >> 172.21.15.43:0/3430315254 conn(0x7f8e900a4460 msgr2=0x7f8e900a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- 172.21.15.43:0/3430315254 shutdown_connections 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- 172.21.15.43:0/3430315254 wait complete. 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: build_initial_monmap 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: monmap: 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:30.468 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:30.471 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:30.471 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:30.471 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:30.471 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 librados: starting msgr at 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 librados: starting objecter 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 Processor -- start 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- start start 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 librados: setting wanted keys 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 librados: calling monclient init 2024-04-19T01:05:30.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: init 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient: _add_conns ranks=[2,5,8,0,1,4,7,3,6] 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e90132620 0x7f8e9013bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): picked mon.noname-b con 0x7f8e90132620 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8e90132a00 0x7f8e9013ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): picked mon.noname-h con 0x7f8e90132a00 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 0x7f8e9013cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): picked mon.noname-e con 0x7f8e90132e20 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f8e900705a0 con 0x7f8e90132620 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f8e90070420 con 0x7f8e90132a00 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f8e90070720 con 0x7f8e90132e20 2024-04-19T01:05:30.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e98438640 10 monclient(hunting): authenticate will time out at 2200.583060s 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 0x7f8e9013cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 0x7f8e9013cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:58022/0 (socket says 172.21.15.43:58022) 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e90132620 0x7f8e9013bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 -- 172.21.15.43:0/2996013312 learned_addr learned my addr 172.21.15.43:0/2996013312 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): get_auth_request con 0x7f8e90132e20 auth_method 0 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): get_auth_request con 0x7f8e90132620 auth_method 0 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e959d3640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8e90132a00 0x7f8e9013ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e959d3640 10 monclient(hunting): get_auth_request con 0x7f8e90132a00 auth_method 0 2024-04-19T01:05:30.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e959d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e959d3640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e959d3640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient(hunting): handle_auth_done global_id 144710 payload 824 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 -- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8e90132a00 msgr2=0x7f8e9013ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8e90132a00 0x7f8e9013ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 -- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e90132620 msgr2=0x7f8e9013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e90132620 0x7f8e9013bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient: _finish_hunting 0 2024-04-19T01:05:30.477 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 monclient: found mon.noname-e 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 -- 172.21.15.43:0/2996013312 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e90071810 con 0x7f8e90132e20 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e961d4640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e90132620 0x7f8e9013bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e969d5640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 0x7f8e9013cf80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e88007e60 tx=0x7f8e8800e4b0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e86ffd640 10 client.?.objecter ms_handle_connect 0x7f8e90132e20 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.454+0000 7f8e86ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e8800ee90 con 0x7f8e90132e20 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: dump: 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.478 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.479 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: _finish_auth 0 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 5 monclient: authenticate success, global_id 144710 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e88019070 con 0x7f8e90132e20 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 monclient: _renew_subs 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f8e88014d00 con 0x7f8e90132e20 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e900712c0 con 0x7f8e90132e20 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e949d1640 4 set_mon_vals no callback set 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:30.480 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 monclient: dump: 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:30.481 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 client.144710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 monclient: _renew_subs 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e9012bf30 con 0x7f8e90132e20 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 1 librados: init done 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e900008e0 con 0x7f8e90132e20 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8e88014420 con 0x7f8e90132e20 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e74038fb0 0x7f8e7403b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f8e88049900 con 0x7f8e90132e20 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 client.144710.objecter ms_dispatch 0x7f8e90000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:30.482 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 3 client.144710.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 3 client.144710.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 20 client.144710.objecter dump_active .. 0 homeless 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e961d4640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e74038fb0 0x7f8e7403b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e961d4640 10 monclient: get_auth_request con 0x7f8e74038fb0 auth_method 0 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e961d4640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e74038fb0 0x7f8e7403b470 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009ea0 tx=0x7f8e8c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.458+0000 7f8e86ffd640 10 client.144710.objecter ms_handle_connect 0x7f8e74038fb0 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.462+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e900712c0 con 0x7f8e90132e20 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.462+0000 7f8e86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:30.483 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.462+0000 7f8e86ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:30.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.626+0000 7f8e98438640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:05:30.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.626+0000 7f8e98438640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:05:30.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.626+0000 7f8e98438640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:30.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.626+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e9009d230 con 0x7f8e90132e20 2024-04-19T01:05:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e86ffd640 1 -- 172.21.15.43:0/2996013312 <== mon.8 v2:172.21.15.43:3304/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v707) ==== 74+0+15264 (secure 0 0 0) 0x7f8e90071810 con 0x7f8e90132e20 2024-04-19T01:05:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-19T01:05:30.629 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e86ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:30.629 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T01:05:30.629 INFO:teuthology.orchestra.run.smithi043.stdout:{"epoch":707,"fsid":"1280a06a-2032-4fc0-a20f-543d29f598e9","created":"2024-04-19T00:44:21.810335+0000","modified":"2024-04-19T01:03:40.676536+0000","last_up_change":"2024-04-19T00:44:32.017099+0000","last_in_change":"2024-04-19T00:44:24.286950+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":102,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-04-19T00:44:32.605528+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-04-19T00:44:32.926773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-19T00:45:26.372688+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e366653d-95f2-4573-8eee-ccdd82879a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":702,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6801","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6804","nonce":110940973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6807","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6809","nonce":110940973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6818","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6821","nonce":110940973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6812","nonce":110940973},{"type":"v1","addr":"172.21.15.43:6815","nonce":110940973}]},"public_addr":"172.21.15.43:6804/110940973","cluster_addr":"172.21.15.43:6809/110940973","heartbeat_back_addr":"172.21.15.43:6821/110940973","heartbeat_front_addr":"172.21.15.43:6815/110940973","state":["exists","up"]},{"osd":1,"uuid":"3efcb199-3b20-4ce2-bd9e-9457316d298e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":702,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6800","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6802","nonce":1380723486}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6805","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6808","nonce":1380723486}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6817","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6820","nonce":1380723486}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6811","nonce":1380723486},{"type":"v1","addr":"172.21.15.43:6814","nonce":1380723486}]},"public_addr":"172.21.15.43:6802/1380723486","cluster_addr":"172.21.15.43:6808/1380723486","heartbeat_back_addr":"172.21.15.43:6820/1380723486","heartbeat_front_addr":"172.21.15.43:6814/1380723486","state":["exists","up"]},{"osd":2,"uuid":"08524c86-3cef-40d9-b1b6-89660fd4a0e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":702,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6803","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6806","nonce":1290935720}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6810","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6813","nonce":1290935720}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6822","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6823","nonce":1290935720}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.43:6816","nonce":1290935720},{"type":"v1","addr":"172.21.15.43:6819","nonce":1290935720}]},"public_addr":"172.21.15.43:6806/1290935720","cluster_addr":"172.21.15.43:6813/1290935720","heartbeat_back_addr":"172.21.15.43:6823/1290935720","heartbeat_front_addr":"172.21.15.43:6819/1290935720","state":["exists","up"]},{"osd":3,"uuid":"9b1f698f-5881-4edf-b694-3025af87e7dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":702,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6801","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6804","nonce":2204255832}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6807","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6810","nonce":2204255832}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6821","nonce":2204255832}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2204255832},{"type":"v1","addr":"172.21.15.161:6815","nonce":2204255832}]},"public_addr":"172.21.15.161:6804/2204255832","cluster_addr":"172.21.15.161:6810/2204255832","heartbeat_back_addr":"172.21.15.161:6821/2204255832","heartbeat_front_addr":"172.21.15.161:6815/2204255832","state":["exists","up"]},{"osd":4,"uuid":"85abaf55-3d4c-4d00-a05b-ac3c6d6dd40a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":702,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6803","nonce":3653492594}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6809","nonce":3653492594}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6822","nonce":3653492594}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6813","nonce":3653492594},{"type":"v1","addr":"172.21.15.161:6816","nonce":3653492594}]},"public_addr":"172.21.15.161:6803/3653492594","cluster_addr":"172.21.15.161:6809/3653492594","heartbeat_back_addr":"172.21.15.161:6822/3653492594","heartbeat_front_addr":"172.21.15.161:6816/3653492594","state":["exists","up"]},{"osd":5,"uuid":"3d8e2a0e-809f-460b-a629-1aaa317f38b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":702,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6805","nonce":760479844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6811","nonce":760479844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6823","nonce":760479844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":760479844},{"type":"v1","addr":"172.21.15.161:6817","nonce":760479844}]},"public_addr":"172.21.15.161:6805/760479844","cluster_addr":"172.21.15.161:6811/760479844","heartbeat_back_addr":"172.21.15.161:6823/760479844","heartbeat_front_addr":"172.21.15.161:6817/760479844","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.756279+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.840443+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.708058+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.848897+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.721708+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-19T00:44:29.855016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":14,"snaps":[{"begin":3,"length":2}]},{"pool":15,"snaps":[{"begin":3,"length":4}]},{"pool":34,"snaps":[{"begin":1,"length":1}]},{"pool":68,"snaps":[{"begin":1,"length":1}]},{"pool":94,"snaps":[{"begin":8,"length":2}]},{"pool":108,"snaps":[{"begin":1,"length":1}]},{"pool":125,"snaps":[{"begin":1,"length":2}]},{"pool":140,"snaps":[{"begin":3,"length":1}]},{"pool":146,"snaps":[{"begin":3,"length":2}]},{"pool":149,"snaps":[{"begin":2,"length":1}]},{"pool":154,"snaps":[{"begin":2,"length":2}]},{"pool":163,"snaps":[{"begin":2,"length":3}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":172,"snaps":[{"begin":2,"length":4}]},{"pool":174,"snaps":[{"begin":2,"length":2}]},{"pool":184,"snaps":[{"begin":2,"length":1}]},{"pool":189,"snaps":[{"begin":1,"length":4}]},{"pool":202,"snaps":[{"begin":2,"length":6}]},{"pool":205,"snaps":[{"begin":7,"length":1}]},{"pool":229,"snaps":[{"begin":8,"length":1}]},{"pool":243,"snaps":[{"begin":10,"length":1}]},{"pool":248,"snaps":[{"begin":11,"length":2}]},{"pool":260,"snaps":[{"begin":14,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":323,"snaps":[{"begin":4,"length":1}]},{"pool":326,"snaps":[{"begin":5,"length":1}]},{"pool":330,"snaps":[{"begin":7,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 10 librados: watch_flush enter 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 10 librados: watch_flush exit 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 20 client.144710.objecter shutdown clearing up homeless session... 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 10 client.144710.objecter successfully canceled tick 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e74038fb0 msgr2=0x7f8e7403b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e74038fb0 0x7f8e7403b470 secure :-1 s=READY pgs=3675 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009ea0 tx=0x7f8e8c009240 comp rx=0 tx=0).stop 2024-04-19T01:05:30.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 10 monclient: shutdown 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 msgr2=0x7f8e9013cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 0x7f8e9013cf80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8e88007e60 tx=0x7f8e8800e4b0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 shutdown_connections 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8e74038fb0 0x7f8e7403b470 unknown :-1 s=CLOSED pgs=3675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f8e90132e20 0x7f8e9013cf80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f8e90132a00 0x7f8e9013ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 --2- 172.21.15.43:0/2996013312 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f8e90132620 0x7f8e9013bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 >> 172.21.15.43:0/2996013312 conn(0x7f8e900a4460 msgr2=0x7f8e9012e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 shutdown_connections 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 -- 172.21.15.43:0/2996013312 wait complete. 2024-04-19T01:05:30.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:30.630+0000 7f8e98438640 1 librados: shutdown 2024-04-19T01:05:31.652 INFO:tasks.ceph:Scrubbing osd.0 2024-04-19T01:05:31.652 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:05:31.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient: get_monmap_and_config 2024-04-19T01:05:31.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient: build_initial_monmap 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient: monmap: 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:31.795136+0000 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:31.795136+0000 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:31.807 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 1 Processor -- start 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 1 -- start start 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient: init 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient: _add_conns ranks=[7,1,0,9,3,8,5,4,2,6,10] 2024-04-19T01:05:31.808 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab5c138d60 0x7fab5c0a6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient(hunting): picked mon.noname-i con 0x7fab5c138d60 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7fab5c139140 addr v1:172.21.15.161:6789/0 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab5c13e080 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c09f790 con 0x7fab5c13e080 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab5c09f4e0 con 0x7fab5c139140 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7fab5c0a7d40 con 0x7fab5c138d60 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 10 monclient(hunting): authenticate will time out at 2201.922114s 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7fab5c13e080 0x7fab5c140470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57480/0 (socket says 172.21.15.43:57480) 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 1 -- 172.21.15.43:0/933043224 learned_addr learned my addr 172.21.15.43:0/933043224 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab5c138d60 0x7fab5c0a6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request con 0x7fab5c138d60 auth_method 0 2024-04-19T01:05:31.809 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968592449 0 0) 0x7fab5c09f790 con 0x7fab5c13e080 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): _negotiate 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): my global_id is 184203 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab48003530 con 0x7fab5c13e080 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:31.810 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3279655353 0 0) 0x7fab48003530 con 0x7fab5c13e080 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab5c138d60 msgr2=0x7fab5c0a6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7fab5c138d60 0x7fab5c0a6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 >> v1:172.21.15.161:6789/0 conn(0x7fab5c139140 legacy=0x7fab5c139520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: _finish_hunting 0 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c0a0bd0 con 0x7fab5c13e080 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:31.811 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7fab54001cf0 con 0x7fab5c13e080 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: dump: 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.812 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 >> v1:172.21.15.43:6789/0 conn(0x7fab5c13e080 legacy=0x7fab5c140470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient: _add_conns ranks=[1,2,6,4,0,8,7,5,3] 2024-04-19T01:05:31.813 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab48004240 0x7fab48004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): picked mon.f con 0x7fab48004240 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fab48004da0 0x7fab48005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): picked mon.b con 0x7fab48004da0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48009c60 0x7fab4800c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): picked mon.d con 0x7fab48009c60 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c0a7d40 con 0x7fab48004240 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fab4800c9d0 con 0x7fab48004da0 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fab4800cb80 con 0x7fab48009c60 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:31.814 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5a7fc640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fab48004da0 0x7fab48005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48009c60 0x7fab4800c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): get_auth_request con 0x7fab48004da0 auth_method 0 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): get_auth_request con 0x7fab48009c60 auth_method 0 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab48004240 0x7fab48004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.815 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request con 0x7fab48004240 auth_method 0 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient(hunting): handle_auth_done global_id 184203 payload 293 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 -- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48009c60 msgr2=0x7fab4800c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48009c60 0x7fab4800c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 -- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab48004240 msgr2=0x7fab48004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab48004240 0x7fab48004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.816 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 monclient: found mon.b 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 1 -- 172.21.15.43:0/933043224 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab48003530 con 0x7fab48004da0 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient: _finish_auth 0 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 5 monclient: authenticate success, global_id 184203 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab62133640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48009c60 0x7fab4800c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5b7fe640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab48004240 0x7fab48004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.794+0000 7fab5bfff640 1 -- 172.21.15.43:0/933043224 reap_dead start 2024-04-19T01:05:31.817 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5affd640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fab48004da0 0x7fab48005220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab5c09fdb0 tx=0x7fab4c01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e3d9e9f25fd9468 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab4c0013e0 con 0x7fab48004da0 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: dump: 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.818 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4c003600 con 0x7fab48004da0 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 4 set_mon_vals no callback set 2024-04-19T01:05:31.819 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 1 -- 172.21.15.43:0/933043224 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab4c003c60 con 0x7fab48004da0 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab5a7fc640 10 monclient: dump: 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.820 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.821 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.821 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.821 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.821 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.822 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.822 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.822 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 10 monclient: shutdown 2024-04-19T01:05:31.822 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 -- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fab48004da0 msgr2=0x7fab48005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fab48004da0 0x7fab48005220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab5c09fdb0 tx=0x7fab4c01c320 comp rx=0 tx=0).stop 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 -- 172.21.15.43:0/933043224 shutdown_connections 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 --2- 172.21.15.43:0/933043224 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fab48004da0 0x7fab48005220 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 -- 172.21.15.43:0/933043224 >> 172.21.15.43:0/933043224 conn(0x7fab5c0ab590 msgr2=0x7fab5c0ab9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 -- 172.21.15.43:0/933043224 shutdown_connections 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 1 -- 172.21.15.43:0/933043224 wait complete. 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 10 monclient: build_initial_monmap 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.798+0000 7fab62133640 10 monclient: monmap: 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:31.823 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 librados: starting msgr at 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 librados: starting objecter 2024-04-19T01:05:31.824 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 Processor -- start 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- start start 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 librados: setting wanted keys 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 librados: calling monclient init 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: init 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _add_conns ranks=[6,4,5,2,1,7,0,8,3] 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48004da0 0x7fab5c133a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): picked mon.noname-d con 0x7fab48004da0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fab5c133f80 0x7fab5c0a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): picked mon.noname-c con 0x7fab5c133f80 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab5c0a4650 0x7fab5c0a4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): picked mon.noname-h con 0x7fab5c0a4650 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.825 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fab5c0a6610 con 0x7fab5c133f80 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fab5c0a6490 con 0x7fab5c0a4650 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fab5c136360 con 0x7fab48004da0 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient(hunting): authenticate will time out at 2201.928827s 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fab5c133f80 0x7fab5c0a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48004da0 0x7fab5c133a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fab5c133f80 0x7fab5c0a4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36640/0 (socket says 172.21.15.43:36640) 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 -- 172.21.15.43:0/371058554 learned_addr learned my addr 172.21.15.43:0/371058554 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): get_auth_request con 0x7fab5c133f80 auth_method 0 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.826 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request con 0x7fab48004da0 auth_method 0 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5bfff640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab5c0a4650 0x7fab5c0a4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient(hunting): handle_auth_done global_id 184198 payload 824 2024-04-19T01:05:31.827 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 -- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48004da0 msgr2=0x7fab5c133a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48004da0 0x7fab5c133a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 -- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab5c0a4650 msgr2=0x7fab5c0a4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab5c0a4650 0x7fab5c0a4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 monclient: found mon.noname-c 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 -- 172.21.15.43:0/371058554 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c134360 con 0x7fab5c133f80 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5bfff640 10 monclient: get_auth_request con 0x7fab5c0a4650 auth_method 0 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5bfff640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fab5c0a4650 0x7fab5c0a4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fab48004da0 0x7fab5c133a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5affd640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fab5c133f80 0x7fab5c0a4110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab4c005040 tx=0x7fab48005840 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 client.?.objecter ms_handle_connect 0x7fab5c133f80 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:31.828 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab4c006c50 con 0x7fab5c133f80 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: dump: 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.829 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: _finish_auth 0 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: _check_auth_tickets 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab4c006e10 con 0x7fab5c133f80 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 5 monclient: authenticate success, global_id 184198 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _renew_subs 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c1c3630 con 0x7fab5c133f80 2024-04-19T01:05:31.830 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fab4c003c20 con 0x7fab5c133f80 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: dump: 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.831 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab59ffb640 4 set_mon_vals no callback set 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 client.184198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _renew_subs 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c121ed0 con 0x7fab5c133f80 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 librados: init done 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 client.184198.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:31.832 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 15 client.184198.objecter _session_command_op_assign -1 1 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 20 client.184198.objecter _get_session osd=-1 returning homeless 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 client.184198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fab5c068b10 con 0x7fab5c133f80 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fab4c0067e0 con 0x7fab5c133f80 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab3c038e10 0x7fab3c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fab4c0e3310 con 0x7fab5c133f80 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 client.184198.objecter ms_dispatch 0x7fab5c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 3 client.184198.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 3 client.184198.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:31.833 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 client.184198.objecter checking command 1 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7fab3c0405b0 0x7fab3c042a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 client.184198.objecter _get_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 15 client.184198.objecter _session_command_op_remove -1 1 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 client.184198.objecter _get_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 client.184198.objecter get_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 15 client.184198.objecter _session_command_op_assign 0 1 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 4 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 client.184198.objecter _send_command 1 2024-04-19T01:05:31.834 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 --> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fab3c043120 con 0x7fab3c0405b0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 20 client.184198.objecter dump_active .. 0 homeless 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_get_version_reply(handle=1 version=707) ==== 24+0+0 (secure 0 0 0) 0x7fab4c0ea050 con 0x7fab5c133f80 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab53fff640 10 monclient: handle_get_version_reply finishing 1 version 707 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5bfff640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7fab3c0405b0 0x7fab3c042a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5bfff640 10 monclient: get_auth_request con 0x7fab3c0405b0 auth_method 0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab3c038e10 0x7fab3c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.802+0000 7fab5b7fe640 10 monclient: get_auth_request con 0x7fab3c038e10 auth_method 0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 10 client.184198.objecter ms_handle_connect 0x7fab3c0405b0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab5bfff640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7fab3c0405b0 0x7fab3c042a50 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab5b7fe640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab3c038e10 0x7fab3c03b2d0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7fab44006f90 tx=0x7fab44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 10 client.184198.objecter ms_handle_connect 0x7fab3c038e10 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== osd.0 v2:172.21.15.43:6801/110940973 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fab3c043120 con 0x7fab3c0405b0 2024-04-19T01:05:31.835 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 10 client.184198.objecter ms_dispatch 0x7fab5c000ba0 command_reply(tid 1: 0 ) 2024-04-19T01:05:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 10 client.184198.objecter _finish_command 1 = system:0 2024-04-19T01:05:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 4 2024-04-19T01:05:31.836 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.806+0000 7fab53fff640 15 client.184198.objecter _session_command_op_remove 0 1 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 client.184198.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 15 client.184198.objecter _session_command_op_assign -1 2 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter _get_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter _get_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 15 client.184198.objecter _session_command_op_remove -1 2 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter get_session s=0x7fab3c03fee0 osd=0 3 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 15 client.184198.objecter _session_command_op_assign 0 2 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 4 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 client.184198.objecter _send_command 2 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 --> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fab5c0a0170 con 0x7fab3c0405b0 2024-04-19T01:05:31.837 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab53fff640 1 -- 172.21.15.43:0/371058554 <== osd.0 v2:172.21.15.43:6801/110940973 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fab5c0a0170 con 0x7fab3c0405b0 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab53fff640 10 client.184198.objecter ms_dispatch 0x7fab5c000ba0 command_reply(tid 2: 0 ) 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab53fff640 10 client.184198.objecter _finish_command 2 = system:0 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab53fff640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 4 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab53fff640 15 client.184198.objecter _session_command_op_remove 0 2 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:05:31.838 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 librados: watch_flush enter 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 librados: watch_flush exit 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 client.184198.objecter close_session for osd.0 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7fab3c0405b0 msgr2=0x7fab3c042a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:6801/110940973,v1:172.21.15.43:6804/110940973] conn(0x7fab3c0405b0 0x7fab3c042a50 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter put_session s=0x7fab3c03fee0 osd=0 1 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 client.184198.objecter shutdown clearing up homeless session... 2024-04-19T01:05:31.840 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 client.184198.objecter successfully canceled tick 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab3c038e10 msgr2=0x7fab3c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fab3c038e10 0x7fab3c03b2d0 secure :-1 s=READY pgs=3676 cs=0 l=1 rev1=1 crypto rx=0x7fab44006f90 tx=0x7fab44008040 comp rx=0 tx=0).stop 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 10 monclient: shutdown 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fab5c133f80 msgr2=0x7fab5c0a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 --2- 172.21.15.43:0/371058554 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fab5c133f80 0x7fab5c0a4110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fab4c005040 tx=0x7fab48005840 comp rx=0 tx=0).stop 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab5bfff640 1 -- 172.21.15.43:0/371058554 reap_dead start 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 shutdown_connections 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 >> 172.21.15.43:0/371058554 conn(0x7fab5c0ab590 msgr2=0x7fab5c0a9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 shutdown_connections 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 -- 172.21.15.43:0/371058554 wait complete. 2024-04-19T01:05:31.841 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.818+0000 7fab62133640 1 librados: shutdown 2024-04-19T01:05:31.842 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-19T01:05:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.954+0000 7f9604076640 10 monclient: get_monmap_and_config 2024-04-19T01:05:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.954+0000 7f9604076640 10 monclient: build_initial_monmap 2024-04-19T01:05:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient: monmap: 2024-04-19T01:05:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.959 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:31.958550+0000 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:31.958550+0000 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:31.960 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:31.961 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 Processor -- start 2024-04-19T01:05:31.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 -- start start 2024-04-19T01:05:31.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient: init 2024-04-19T01:05:31.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:31.961 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient: _add_conns ranks=[9,10,4,3,6,7,5,2,8,1,0] 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f95fc06c240 0x7f95fc0a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): picked mon.noname-a con 0x7f95fc06c240 addr v2:172.21.15.43:3300/0 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95fc0a2f90 0x7f95fc0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): picked mon.noname-f con 0x7f95fc0a2f90 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95fc0a4f70 0x7f95fc0a5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): picked mon.noname-c con 0x7f95fc0a4f70 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f95fc06c620 con 0x7f95fc06c240 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f95fc1374f0 con 0x7f95fc0a2f90 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f95fc0722d0 con 0x7f95fc0a4f70 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:31.962 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9604076640 10 monclient(hunting): authenticate will time out at 2202.085742s 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f95fc06c240 0x7f95fc0a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95fc0a4f70 0x7f95fc0a5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f95fc06c240 0x7f95fc0a2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51826/0 (socket says 172.21.15.43:51826) 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 1 -- 172.21.15.43:0/3516935788 learned_addr learned my addr 172.21.15.43:0/3516935788 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): get_auth_request con 0x7f95fc06c240 auth_method 0 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): get_auth_request con 0x7f95fc0a4f70 auth_method 0 2024-04-19T01:05:31.963 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601e12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601611640 1 --2- 172.21.15.43:0/3516935788 >> v2:172.21.15.161:3300/0 conn(0x7f95fc0a2f90 0x7f95fc0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601611640 10 monclient(hunting): get_auth_request con 0x7f95fc0a2f90 auth_method 0 2024-04-19T01:05:31.964 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601611640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9601611640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient(hunting): handle_auth_done global_id 184207 payload 274 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 -- 172.21.15.43:0/3516935788 >> v2:172.21.15.161:3300/0 conn(0x7f95fc0a2f90 msgr2=0x7f95fc0a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 --2- 172.21.15.43:0/3516935788 >> v2:172.21.15.161:3300/0 conn(0x7f95fc0a2f90 0x7f95fc0a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 -- 172.21.15.43:0/3516935788 >> v2:172.21.15.43:3300/0 conn(0x7f95fc06c240 msgr2=0x7f95fc0a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 --2- 172.21.15.43:0/3516935788 >> v2:172.21.15.43:3300/0 conn(0x7f95fc06c240 0x7f95fc0a2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient: _finish_hunting 0 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 monclient: found mon.noname-c 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:31.965 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.958+0000 7f9602613640 1 -- 172.21.15.43:0/3516935788 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc072f50 con 0x7f95fc0a4f70 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9602613640 1 --2- 172.21.15.43:0/3516935788 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95fc0a4f70 0x7f95fc0a5400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95f40097c0 tx=0x7f95f401c320 comp rx=0 tx=0).ready entity=mon.4 client_cookie=1571b039600ab4c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 1 -- 172.21.15.43:0/3516935788 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95f40b0070 con 0x7f95fc0a4f70 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: dump: 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.972 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: _finish_auth 0 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: _check_auth_tickets 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 1 -- 172.21.15.43:0/3516935788 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f40aa810 con 0x7f95fc0a4f70 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 5 monclient: authenticate success, global_id 184207 2024-04-19T01:05:31.973 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:31.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:31.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:31.974 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 4 set_mon_vals no callback set 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 1 -- 172.21.15.43:0/3516935788 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95f40a9300 con 0x7f95fc0a4f70 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9600e10640 10 monclient: dump: 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.975 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: shutdown 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3516935788 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95fc0a4f70 msgr2=0x7f95fc0a5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- 172.21.15.43:0/3516935788 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95fc0a4f70 0x7f95fc0a5400 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95f40097c0 tx=0x7f95f401c320 comp rx=0 tx=0).stop 2024-04-19T01:05:31.976 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3516935788 shutdown_connections 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- 172.21.15.43:0/3516935788 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f95fc0a4f70 0x7f95fc0a5400 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- 172.21.15.43:0/3516935788 >> v2:172.21.15.161:3300/0 conn(0x7f95fc0a2f90 0x7f95fc0a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- 172.21.15.43:0/3516935788 >> v2:172.21.15.43:3300/0 conn(0x7f95fc06c240 0x7f95fc0a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3516935788 >> 172.21.15.43:0/3516935788 conn(0x7f95fc0a62e0 msgr2=0x7f95fc06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3516935788 shutdown_connections 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3516935788 wait complete. 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: build_initial_monmap 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: monmap: 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:31.977 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:31.978 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 librados: starting msgr at 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 librados: starting objecter 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 Processor -- start 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- start start 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 librados: setting wanted keys 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 librados: calling monclient init 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: init 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: _add_conns ranks=[1,0,3,2,4,7,5,6,8] 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95fc06c240 0x7f95fc11fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): picked mon.noname-f con 0x7f95fc06c240 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 0x7f95fc1358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): picked mon.noname-a con 0x7f95fc0a2f90 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:31.979 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95fc0a4f70 0x7f95fc13b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): picked mon.noname-g con 0x7f95fc0a4f70 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc071d00 con 0x7f95fc0a2f90 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f95fc071b80 con 0x7f95fc06c240 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f95fc140510 con 0x7f95fc0a4f70 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient(hunting): authenticate will time out at 2202.090807s 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 0x7f95fc1358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 0x7f95fc1358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51842/0 (socket says 172.21.15.43:51842) 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 -- 172.21.15.43:0/3860491603 learned_addr learned my addr 172.21.15.43:0/3860491603 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): get_auth_request con 0x7f95fc0a2f90 auth_method 0 2024-04-19T01:05:31.980 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601e12640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95fc06c240 0x7f95fc11fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9602613640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95fc0a4f70 0x7f95fc13b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601e12640 10 monclient(hunting): get_auth_request con 0x7f95fc06c240 auth_method 0 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601e12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601e12640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601e12640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9602613640 10 monclient(hunting): get_auth_request con 0x7f95fc0a4f70 auth_method 0 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9602613640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:31.981 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9602613640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9602613640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient(hunting): handle_auth_done global_id 184221 payload 824 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 -- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95fc0a4f70 msgr2=0x7f95fc13b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95fc0a4f70 0x7f95fc13b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 -- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95fc06c240 msgr2=0x7f95fc11fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95fc06c240 0x7f95fc11fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient: _finish_hunting 0 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 monclient: found mon.noname-a 2024-04-19T01:05:31.982 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 -- 172.21.15.43:0/3860491603 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95fc1cf2d0 con 0x7f95fc0a2f90 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9601611640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 0x7f95fc1358a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00d800 tx=0x7f95ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 client.?.objecter ms_handle_connect 0x7f95fc0a2f90 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95ec00bb30 con 0x7f95fc0a2f90 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:31.983 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: dump: 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.984 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ec019070 con 0x7f95fc0a2f90 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 5 monclient: authenticate success, global_id 184221 2024-04-19T01:05:31.985 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: _renew_subs 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95fc120f60 con 0x7f95fc0a2f90 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f95ec00bcf0 con 0x7f95fc0a2f90 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f3fff640 4 set_mon_vals no callback set 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 client.184221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f95f27fc640 10 monclient: dump: 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.986 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: _renew_subs 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:31.987 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95fc1204c0 con 0x7f95fc0a2f90 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.962+0000 7f9604076640 1 librados: init done 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9604076640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9604076640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9604076640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95fc11f830 con 0x7f95fc0a2f90 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f95ec00b6c0 con 0x7f95fc0a2f90 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f95e0038e40 0x7f95e003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f95ec00f040 con 0x7f95fc0a2f90 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 10 client.184221.objecter ms_dispatch 0x7f95fc000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 3 client.184221.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 3 client.184221.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 20 client.184221.objecter dump_active .. 0 homeless 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9601e12640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f95e0038e40 0x7f95e003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:31.988 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9601e12640 10 monclient: get_auth_request con 0x7f95e0038e40 auth_method 0 2024-04-19T01:05:31.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f9601e12640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f95e0038e40 0x7f95e003b300 secure :-1 s=READY pgs=3677 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009ce0 tx=0x7f95f800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:31.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.966+0000 7f95f27fc640 10 client.184221.objecter ms_handle_connect 0x7f95e0038e40 2024-04-19T01:05:31.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.970+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f95ec022020 con 0x7f95fc0a2f90 2024-04-19T01:05:31.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.970+0000 7f95f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:31.989 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:31.970+0000 7f95f27fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:32.133 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.134+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f95fc0724f0 con 0x7f95e0038e40 2024-04-19T01:05:32.134 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.134+0000 7f95f27fc640 1 -- 172.21.15.43:0/3860491603 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f95fc0724f0 con 0x7f95e0038e40 2024-04-19T01:05:32.134 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 0 to deep-scrub 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 10 librados: watch_flush enter 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 10 librados: watch_flush exit 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 20 client.184221.objecter shutdown clearing up homeless session... 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 10 client.184221.objecter successfully canceled tick 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f95e0038e40 msgr2=0x7f95e003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f95e0038e40 0x7f95e003b300 secure :-1 s=READY pgs=3677 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009ce0 tx=0x7f95f800b040 comp rx=0 tx=0).stop 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 10 monclient: shutdown 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 msgr2=0x7f95fc1358a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.137 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 0x7f95fc1358a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00d800 tx=0x7f95ec00dcc0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 shutdown_connections 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f95e0038e40 0x7f95e003b300 unknown :-1 s=CLOSED pgs=3677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f95fc0a4f70 0x7f95fc13b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f95fc0a2f90 0x7f95fc1358a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 --2- 172.21.15.43:0/3860491603 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95fc06c240 0x7f95fc11fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 >> 172.21.15.43:0/3860491603 conn(0x7f95fc0a62e0 msgr2=0x7f95fc0a20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 shutdown_connections 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 -- 172.21.15.43:0/3860491603 wait complete. 2024-04-19T01:05:32.138 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.138+0000 7f9604076640 1 librados: shutdown 2024-04-19T01:05:32.152 INFO:tasks.ceph:Scrubbing osd.1 2024-04-19T01:05:32.153 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient: get_monmap_and_config 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient: monmap: 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:32.283660+0000 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:32.283660+0000 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.281 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.282 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.286 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:32.294 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:32.294 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 Processor -- start 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 -- start start 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient: init 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient: _add_conns ranks=[2,8,10,6,1,4,9,5,3,7,0] 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564135fa0 0x7f55640a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): picked mon.noname-b con 0x7f5564135fa0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 0x7f5564138e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): picked mon.noname-e con 0x7f5564136380 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f55641393b0 0x7f5564139830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.304 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): picked mon.noname-f con 0x7f55641393b0 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f556409f4e0 con 0x7f55641393b0 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f55640a0c40 con 0x7f5564135fa0 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f55640a0f20 con 0x7f5564136380 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.305 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5569c52640 10 monclient(hunting): authenticate will time out at 2202.411332s 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 0x7f5564138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564135fa0 0x7f55640a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 0x7f5564138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:51708/0 (socket says 172.21.15.43:51708) 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 -- 172.21.15.43:0/2709997573 learned_addr learned my addr 172.21.15.43:0/2709997573 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): get_auth_request con 0x7f5564136380 auth_method 0 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): get_auth_request con 0x7f5564135fa0 auth_method 0 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.306 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f55637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5563fff640 1 --2- 172.21.15.43:0/2709997573 >> v2:172.21.15.161:3300/0 conn(0x7f55641393b0 0x7f5564139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5563fff640 10 monclient(hunting): get_auth_request con 0x7f55641393b0 auth_method 0 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5563fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5563fff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5563fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient(hunting): handle_auth_done global_id 144719 payload 274 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 -- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564135fa0 msgr2=0x7f55640a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 --2- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564135fa0 0x7f55640a6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 -- 172.21.15.43:0/2709997573 >> v2:172.21.15.161:3300/0 conn(0x7f55641393b0 msgr2=0x7f5564139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 --2- 172.21.15.43:0/2709997573 >> v2:172.21.15.161:3300/0 conn(0x7f55641393b0 0x7f5564139830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.307 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 monclient: found mon.noname-e 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.282+0000 7f5562ffd640 1 -- 172.21.15.43:0/2709997573 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556409f8b0 con 0x7f5564136380 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5562ffd640 1 --2- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 0x7f5564138e70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5558009870 tx=0x7f555801c420 comp rx=0 tx=0).ready entity=mon.8 client_cookie=30056a48a6f48703 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 1 -- 172.21.15.43:0/2709997573 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55580b0070 con 0x7f5564136380 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: dump: 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.308 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.309 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 1 -- 172.21.15.43:0/2709997573 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55580aa810 con 0x7f5564136380 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 1 -- 172.21.15.43:0/2709997573 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55580a9300 con 0x7f5564136380 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 5 monclient: authenticate success, global_id 144719 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 4 set_mon_vals no callback set 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f55627fc640 10 monclient: dump: 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.310 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 10 monclient: shutdown 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 -- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 msgr2=0x7f5564138e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 --2- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 0x7f5564138e70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5558009870 tx=0x7f555801c420 comp rx=0 tx=0).stop 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 -- 172.21.15.43:0/2709997573 shutdown_connections 2024-04-19T01:05:32.311 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 --2- 172.21.15.43:0/2709997573 >> v2:172.21.15.161:3300/0 conn(0x7f55641393b0 0x7f5564139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 --2- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f5564136380 0x7f5564138e70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 --2- 172.21.15.43:0/2709997573 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564135fa0 0x7f55640a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 -- 172.21.15.43:0/2709997573 >> 172.21.15.43:0/2709997573 conn(0x7f556406bfe0 msgr2=0x7f556406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 -- 172.21.15.43:0/2709997573 shutdown_connections 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 -- 172.21.15.43:0/2709997573 wait complete. 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 10 monclient: monmap: 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:32.312 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 librados: starting msgr at 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.286+0000 7f5569c52640 1 librados: starting objecter 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 Processor -- start 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- start start 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 librados: setting wanted keys 2024-04-19T01:05:32.313 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 librados: calling monclient init 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: init 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _add_conns ranks=[4,2,7,1,3,6,8,5,0] 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5564135fa0 0x7f55641337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): picked mon.noname-c con 0x7f5564135fa0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564136380 0x7f5564133d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): picked mon.noname-b con 0x7f5564136380 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f55641393b0 0x7f5564134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): picked mon.noname-i con 0x7f55641393b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f55641429c0 con 0x7f5564136380 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f55641369b0 con 0x7f5564135fa0 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.314 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f5564142b40 con 0x7f55641393b0 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient(hunting): authenticate will time out at 2202.416591s 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564136380 0x7f5564133d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564136380 0x7f5564133d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:34084/0 (socket says 172.21.15.43:34084) 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 -- 172.21.15.43:0/59285830 learned_addr learned my addr 172.21.15.43:0/59285830 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): get_auth_request con 0x7f5564136380 auth_method 0 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5564135fa0 0x7f55641337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): get_auth_request con 0x7f5564135fa0 auth_method 0 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.315 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient(hunting): handle_auth_done global_id 144677 payload 824 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 -- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f55641393b0 msgr2=0x7f5564134250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f55641393b0 0x7f5564134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 -- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5564135fa0 msgr2=0x7f55641337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5564135fa0 0x7f55641337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 monclient: found mon.noname-b 2024-04-19T01:05:32.316 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 -- 172.21.15.43:0/59285830 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55640a0330 con 0x7f5564136380 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5562ffd640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564136380 0x7f5564133d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f555801c250 tx=0x7f55580191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.?.objecter ms_handle_connect 0x7f5564136380 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55580b9070 con 0x7f5564136380 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: dump: 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.317 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55580044b0 con 0x7f5564136380 2024-04-19T01:05:32.318 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 5 monclient: authenticate success, global_id 144677 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _renew_subs 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556409fe50 con 0x7f5564136380 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f55580b0040 con 0x7f5564136380 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: dump: 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.319 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55617fa640 4 set_mon_vals no callback set 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 client.144677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _renew_subs 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:32.320 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55640a36f0 con 0x7f5564136380 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 librados: init done 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 client.144677.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 15 client.144677.objecter _session_command_op_assign -1 1 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 20 client.144677.objecter _get_session osd=-1 returning homeless 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 client.144677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5564068ce0 con 0x7f5564136380 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f55580bc060 con 0x7f5564136380 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5550038e10 0x7f555003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.321 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f55580e2170 con 0x7f5564136380 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.144677.objecter ms_dispatch 0x7f5564000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 3 client.144677.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 3 client.144677.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.144677.objecter checking command 1 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f55500405a0 0x7f5550042a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 client.144677.objecter _get_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 15 client.144677.objecter _session_command_op_remove -1 1 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 client.144677.objecter _get_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 client.144677.objecter get_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 15 client.144677.objecter _session_command_op_assign 1 1 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 4 2024-04-19T01:05:32.322 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.144677.objecter _send_command 1 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55580b1e50 con 0x7f55500405a0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 20 client.144677.objecter dump_active .. 0 homeless 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_get_version_reply(handle=1 version=707) ==== 24+0+0 (secure 0 0 0) 0x7f55580e9050 con 0x7f5564136380 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 monclient: handle_get_version_reply finishing 1 version 707 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5563fff640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f55500405a0 0x7f5550042a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5550038e10 0x7f555003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5563fff640 10 monclient: get_auth_request con 0x7f55500405a0 auth_method 0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 10 monclient: get_auth_request con 0x7f5550038e10 auth_method 0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.144677.objecter ms_handle_connect 0x7f55500405a0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f5563fff640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f55500405a0 0x7f5550042a40 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.323 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f554bfff640 10 client.144677.objecter ms_handle_connect 0x7f5550038e10 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.290+0000 7f55637fe640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5550038e10 0x7f555003b2d0 secure :-1 s=READY pgs=3678 cs=0 l=1 rev1=1 crypto rx=0x7f5554009b60 tx=0x7f5554009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.294+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== osd.1 v2:172.21.15.43:6800/1380723486 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55580b1e50 con 0x7f55500405a0 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.294+0000 7f554bfff640 10 client.144677.objecter ms_dispatch 0x7f5564000ba0 command_reply(tid 1: 0 ) 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.294+0000 7f554bfff640 10 client.144677.objecter _finish_command 1 = system:0 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.294+0000 7f554bfff640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 4 2024-04-19T01:05:32.324 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.294+0000 7f554bfff640 15 client.144677.objecter _session_command_op_remove 1 1 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 client.144677.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 15 client.144677.objecter _session_command_op_assign -1 2 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter _get_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.326 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter _get_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 15 client.144677.objecter _session_command_op_remove -1 2 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter get_session s=0x7f555003fed0 osd=1 3 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 15 client.144677.objecter _session_command_op_assign 1 2 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 4 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 client.144677.objecter _send_command 2 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 --> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f55640a5530 con 0x7f55500405a0 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f554bfff640 1 -- 172.21.15.43:0/59285830 <== osd.1 v2:172.21.15.43:6800/1380723486 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f55640a5530 con 0x7f55500405a0 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f554bfff640 10 client.144677.objecter ms_dispatch 0x7f5564000ba0 command_reply(tid 2: 0 ) 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f554bfff640 10 client.144677.objecter _finish_command 2 = system:0 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f554bfff640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 4 2024-04-19T01:05:32.327 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f554bfff640 15 client.144677.objecter _session_command_op_remove 1 2 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:05:32.328 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 librados: watch_flush enter 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 librados: watch_flush exit 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 client.144677.objecter close_session for osd.1 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f55500405a0 msgr2=0x7f5550042a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:6800/1380723486,v1:172.21.15.43:6802/1380723486] conn(0x7f55500405a0 0x7f5550042a40 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter put_session s=0x7f555003fed0 osd=1 1 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 client.144677.objecter shutdown clearing up homeless session... 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 client.144677.objecter successfully canceled tick 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5550038e10 msgr2=0x7f555003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5550038e10 0x7f555003b2d0 secure :-1 s=READY pgs=3678 cs=0 l=1 rev1=1 crypto rx=0x7f5554009b60 tx=0x7f5554009190 comp rx=0 tx=0).stop 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 10 monclient: shutdown 2024-04-19T01:05:32.330 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564136380 msgr2=0x7f5564133d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 --2- 172.21.15.43:0/59285830 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f5564136380 0x7f5564133d10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f555801c250 tx=0x7f55580191c0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5563fff640 1 -- 172.21.15.43:0/59285830 reap_dead start 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 shutdown_connections 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 >> 172.21.15.43:0/59285830 conn(0x7f556406bfe0 msgr2=0x7f55640a92c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 shutdown_connections 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 -- 172.21.15.43:0/59285830 wait complete. 2024-04-19T01:05:32.331 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.306+0000 7f5569c52640 1 librados: shutdown 2024-04-19T01:05:32.331 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-19T01:05:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.446+0000 7f87a3a6e640 10 monclient: get_monmap_and_config 2024-04-19T01:05:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.446+0000 7f87a3a6e640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.446+0000 7f87a3a6e640 10 monclient: monmap: 2024-04-19T01:05:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.447 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:32.450301+0000 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:32.450301+0000 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:32.448 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:32.449 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 Processor -- start 2024-04-19T01:05:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 -- start start 2024-04-19T01:05:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient: init 2024-04-19T01:05:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.452 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient: _add_conns ranks=[8,0,6,9,2,5,7,4,3,10,1] 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c136100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): picked mon.noname-e con 0x7f879c135d20 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f879c135860 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 0x7f879c13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): picked mon.noname-d con 0x7f879c0a4050 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f879c0a2400 con 0x7f879c135860 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f879c0a1780 con 0x7f879c0a4050 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f879c0a1130 con 0x7f879c135d20 2024-04-19T01:05:32.453 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a3a6e640 10 monclient(hunting): authenticate will time out at 2202.577391s 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c136100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 0x7f879c13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c136100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:51710/0 (socket says 172.21.15.43:51710) 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 0x7f879c13dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39664/0 (socket says 172.21.15.43:39664) 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 1 -- 172.21.15.43:0/661294627 learned_addr learned my addr 172.21.15.43:0/661294627 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): get_auth_request con 0x7f879c135d20 auth_method 0 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): get_auth_request con 0x7f879c0a4050 auth_method 0 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.454 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a180a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient(hunting): handle_auth_done global_id 184236 payload 274 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 -- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 msgr2=0x7f879c136100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 --2- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c136100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 -- 172.21.15.43:0/661294627 >> v1:172.21.15.43:6789/0 conn(0x7f879c135860 legacy=0x7f879c0a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 monclient: found mon.noname-d 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a0808640 1 -- 172.21.15.43:0/661294627 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2155539975 0 0) 0x7f879c0a2400 con 0x7f879c135860 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:32.455 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a200b640 1 -- 172.21.15.43:0/661294627 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c0a1410 con 0x7f879c0a4050 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.450+0000 7f87a0808640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 1 --2- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 0x7f879c13dba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8794009dc0 tx=0x7f879401c8e0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ce6b5e5c9c560076 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 1 -- 172.21.15.43:0/661294627 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f87940b2070 con 0x7f879c0a4050 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: dump: 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.456 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 1 -- 172.21.15.43:0/661294627 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87940a59f0 con 0x7f879c0a4050 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 1 -- 172.21.15.43:0/661294627 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f87940a5be0 con 0x7f879c0a4050 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.457 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a0808640 10 monclient: dump: 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.458 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 5 monclient: authenticate success, global_id 184236 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 4 set_mon_vals no callback set 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: shutdown 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 msgr2=0x7f879c13dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 0x7f879c13dba0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8794009dc0 tx=0x7f879401c8e0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- 172.21.15.43:0/661294627 shutdown_connections 2024-04-19T01:05:32.459 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f879c0a4050 0x7f879c13dba0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/661294627 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c136100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- 172.21.15.43:0/661294627 >> 172.21.15.43:0/661294627 conn(0x7f879c0a85c0 msgr2=0x7f879c0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- 172.21.15.43:0/661294627 shutdown_connections 2024-04-19T01:05:32.460 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- 172.21.15.43:0/661294627 wait complete. 2024-04-19T01:05:32.465 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: monmap: 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.466 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 librados: starting msgr at 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 librados: starting objecter 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 Processor -- start 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- start start 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 librados: setting wanted keys 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 librados: calling monclient init 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: init 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient: _add_conns ranks=[0,2,8,3,5,6,7,1,4] 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 0x7f879c1cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): picked mon.noname-a con 0x7f879c0a4050 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:32.467 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f879c135860 0x7f879c13d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): picked mon.noname-b con 0x7f879c135860 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c1c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): picked mon.noname-e con 0x7f879c135d20 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f879c06c1f0 con 0x7f879c0a4050 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f879c073460 con 0x7f879c135860 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f879c06c370 con 0x7f879c135d20 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a3a6e640 10 monclient(hunting): authenticate will time out at 2202.583127s 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 0x7f879c1cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 0x7f879c1cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51852/0 (socket says 172.21.15.43:51852) 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 -- 172.21.15.43:0/309508439 learned_addr learned my addr 172.21.15.43:0/309508439 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): get_auth_request con 0x7f879c0a4050 auth_method 0 2024-04-19T01:05:32.468 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f879c135860 0x7f879c13d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): get_auth_request con 0x7f879c135860 auth_method 0 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c1c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): get_auth_request con 0x7f879c135d20 auth_method 0 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.469 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a200b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient(hunting): handle_auth_done global_id 184239 payload 824 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 -- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 msgr2=0x7f879c1c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c1c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 -- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f879c135860 msgr2=0x7f879c13d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f879c135860 0x7f879c13d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 monclient: found mon.noname-a 2024-04-19T01:05:32.470 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 -- 172.21.15.43:0/309508439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f879c0736a0 con 0x7f879c0a4050 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a1009640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f879c135860 0x7f879c13d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87a180a640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 0x7f879c1cc630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8798010dc0 tx=0x7f879800e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87927fc640 10 client.?.objecter ms_handle_connect 0x7f879c0a4050 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.454+0000 7f87927fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f879801e070 con 0x7f879c0a4050 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: dump: 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.471 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.472 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879800ee40 con 0x7f879c0a4050 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 5 monclient: authenticate success, global_id 184239 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 monclient: _renew_subs 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f879c1382d0 con 0x7f879c0a4050 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f879800d350 con 0x7f879c0a4050 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 client.184239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:32.473 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f8793fff640 4 set_mon_vals no callback set 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 monclient: dump: 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.474 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 monclient: _renew_subs 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f879c1373e0 con 0x7f879c0a4050 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 1 librados: init done 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f8798011660 con 0x7f879c0a4050 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f879c135010 con 0x7f879c0a4050 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8780038e40 0x7f878003b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.475 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f879804b9d0 con 0x7f879c0a4050 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 client.184239.objecter ms_dispatch 0x7f879c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 3 client.184239.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 3 client.184239.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 20 client.184239.objecter dump_active .. 0 homeless 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a1009640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8780038e40 0x7f878003b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a1009640 10 monclient: get_auth_request con 0x7f8780038e40 auth_method 0 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87927fc640 10 client.184239.objecter ms_handle_connect 0x7f8780038e40 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.458+0000 7f87a1009640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8780038e40 0x7f878003b300 secure :-1 s=READY pgs=3679 cs=0 l=1 rev1=1 crypto rx=0x7f879c135210 tx=0x7f878c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.462+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8798020e10 con 0x7f879c0a4050 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.462+0000 7f87927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.476 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.462+0000 7f87927fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:32.626 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.626+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f879c138a00 con 0x7f8780038e40 2024-04-19T01:05:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 1 to deep-scrub 2024-04-19T01:05:32.627 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.626+0000 7f87927fc640 1 -- 172.21.15.43:0/309508439 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f879c138a00 con 0x7f8780038e40 2024-04-19T01:05:32.630 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 10 librados: watch_flush enter 2024-04-19T01:05:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 10 librados: watch_flush exit 2024-04-19T01:05:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 20 client.184239.objecter shutdown clearing up homeless session... 2024-04-19T01:05:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 10 client.184239.objecter successfully canceled tick 2024-04-19T01:05:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8780038e40 msgr2=0x7f878003b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.631 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8780038e40 0x7f878003b300 secure :-1 s=READY pgs=3679 cs=0 l=1 rev1=1 crypto rx=0x7f879c135210 tx=0x7f878c009040 comp rx=0 tx=0).stop 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 10 monclient: shutdown 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 msgr2=0x7f879c1cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 0x7f879c1cc630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8798010dc0 tx=0x7f879800e730 comp rx=0 tx=0).stop 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 shutdown_connections 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f8780038e40 0x7f878003b300 unknown :-1 s=CLOSED pgs=3679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.632 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f879c135d20 0x7f879c1c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f879c135860 0x7f879c13d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 --2- 172.21.15.43:0/309508439 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f879c0a4050 0x7f879c1cc630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 >> 172.21.15.43:0/309508439 conn(0x7f879c0a85c0 msgr2=0x7f879c0a66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 shutdown_connections 2024-04-19T01:05:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 -- 172.21.15.43:0/309508439 wait complete. 2024-04-19T01:05:32.633 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.630+0000 7f87a3a6e640 1 librados: shutdown 2024-04-19T01:05:32.647 INFO:tasks.ceph:Scrubbing osd.2 2024-04-19T01:05:32.647 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:05:32.768 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient: get_monmap_and_config 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient: monmap: 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:32.771593+0000 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:32.771593+0000 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.769 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.770 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.770 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.770 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.770 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:32.770 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:32.770 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 Processor -- start 2024-04-19T01:05:32.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 -- start start 2024-04-19T01:05:32.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient: init 2024-04-19T01:05:32.772 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient: _add_conns ranks=[6,10,1,9,5,3,7,4,8,0,2] 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 0x7ff6d00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): picked mon.noname-d con 0x7ff6d0135fa0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff6d0136380 0x7ff6d0138e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): picked mon.noname-f con 0x7ff6d0136380 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7ff6d01393b0 addr v1:172.21.15.161:6789/0 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d009f4e0 con 0x7ff6d01393b0 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d00a0c40 con 0x7ff6d0136380 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ff6d00a0f20 con 0x7ff6d0135fa0 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d6c91640 10 monclient(hunting): authenticate will time out at 2202.899157s 2024-04-19T01:05:32.773 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 0x7ff6d00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 0x7ff6d00a6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39670/0 (socket says 172.21.15.43:39670) 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/912950730 learned_addr learned my addr 172.21.15.43:0/912950730 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): get_auth_request con 0x7ff6d0135fa0 auth_method 0 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6cffff640 1 --2- 172.21.15.43:0/912950730 >> v2:172.21.15.161:3300/0 conn(0x7ff6d0136380 0x7ff6d0138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6cffff640 10 monclient(hunting): get_auth_request con 0x7ff6d0136380 auth_method 0 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.774 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_done global_id 184245 payload 274 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/912950730 >> v2:172.21.15.161:3300/0 conn(0x7ff6d0136380 msgr2=0x7ff6d0138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 --2- 172.21.15.43:0/912950730 >> v2:172.21.15.161:3300/0 conn(0x7ff6d0136380 0x7ff6d0138e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/912950730 >> v1:172.21.15.161:6789/0 conn(0x7ff6d01393b0 legacy=0x7ff6d01401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 monclient: found mon.noname-d 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:32.775 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.770+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/912950730 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d009f8b0 con 0x7ff6d0135fa0 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d4a2d640 1 --2- 172.21.15.43:0/912950730 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 0x7ff6d00a6180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009870 tx=0x7ff6c001c420 comp rx=0 tx=0).ready entity=mon.6 client_cookie=ce37d84a2fd5c768 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 1 -- 172.21.15.43:0/912950730 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6c00b1070 con 0x7ff6d0135fa0 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: dump: 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.783 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.784 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 1 -- 172.21.15.43:0/912950730 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c00ab830 con 0x7ff6d0135fa0 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 5 monclient: authenticate success, global_id 184245 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 1 -- 172.21.15.43:0/912950730 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6c00aa320 con 0x7ff6d0135fa0 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 4 set_mon_vals no callback set 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6cf7fe640 10 monclient: dump: 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.785 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.786 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 10 monclient: shutdown 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 -- 172.21.15.43:0/912950730 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 msgr2=0x7ff6d00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 --2- 172.21.15.43:0/912950730 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 0x7ff6d00a6180 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0009870 tx=0x7ff6c001c420 comp rx=0 tx=0).stop 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 -- 172.21.15.43:0/912950730 shutdown_connections 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 --2- 172.21.15.43:0/912950730 >> v2:172.21.15.161:3300/0 conn(0x7ff6d0136380 0x7ff6d0138e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 --2- 172.21.15.43:0/912950730 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0135fa0 0x7ff6d00a6180 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 -- 172.21.15.43:0/912950730 >> 172.21.15.43:0/912950730 conn(0x7ff6d006bfe0 msgr2=0x7ff6d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 -- 172.21.15.43:0/912950730 shutdown_connections 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 -- 172.21.15.43:0/912950730 wait complete. 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 10 monclient: monmap: 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.787 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.788 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 librados: starting msgr at 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.774+0000 7ff6d6c91640 1 librados: starting objecter 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 Processor -- start 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- start start 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 librados: setting wanted keys 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 librados: calling monclient init 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: init 2024-04-19T01:05:32.789 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _add_conns ranks=[4,6,7,8,2,1,0,5,3] 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff6d0135fa0 0x7ff6d00a52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d0135fa0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0136380 0x7ff6d01d1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): picked mon.noname-d con 0x7ff6d0136380 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff6d01393b0 0x7ff6d01d19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): picked mon.noname-i con 0x7ff6d01393b0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7ff6d01368e0 con 0x7ff6d0135fa0 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7ff6d0136760 con 0x7ff6d0136380 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.790 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff6d01440e0 con 0x7ff6d01393b0 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient(hunting): authenticate will time out at 2202.904700s 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff6d0135fa0 0x7ff6d00a52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff6d0135fa0 0x7ff6d00a52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36680/0 (socket says 172.21.15.43:36680) 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/4152777180 learned_addr learned my addr 172.21.15.43:0/4152777180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): get_auth_request con 0x7ff6d0135fa0 auth_method 0 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0136380 0x7ff6d01d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): get_auth_request con 0x7ff6d0136380 auth_method 0 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.791 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d522e640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff6d01393b0 0x7ff6d01d19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient(hunting): handle_auth_done global_id 184225 payload 824 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff6d01393b0 msgr2=0x7ff6d01d19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff6d01393b0 0x7ff6d01d19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0136380 msgr2=0x7ff6d01d1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0136380 0x7ff6d01d1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.792 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 monclient: found mon.noname-c 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 -- 172.21.15.43:0/4152777180 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d009d9b0 con 0x7ff6d0135fa0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d522e640 10 monclient: get_auth_request con 0x7ff6d01393b0 auth_method 0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d522e640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff6d01393b0 0x7ff6d01d19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7ff6d0136380 0x7ff6d01d1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d4a2d640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff6d0135fa0 0x7ff6d00a52b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00af040 tx=0x7ff6c00191c0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff6d0135fa0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6c00ba070 con 0x7ff6d0135fa0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: dump: 2024-04-19T01:05:32.793 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.794 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c00044b0 con 0x7ff6d0135fa0 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff6c00b1040 con 0x7ff6d0135fa0 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.795 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: dump: 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.796 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6ceffd640 4 set_mon_vals no callback set 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 5 monclient: authenticate success, global_id 184225 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _renew_subs 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d00a0330 con 0x7ff6d0135fa0 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 client.184225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _renew_subs 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d00a3120 con 0x7ff6d0135fa0 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 librados: init done 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff6c0004040 con 0x7ff6d0135fa0 2024-04-19T01:05:32.797 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 client.184225.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 15 client.184225.objecter _session_command_op_assign -1 1 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 20 client.184225.objecter _get_session osd=-1 returning homeless 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 client.184225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6d009fe50 con 0x7ff6d0135fa0 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff6bc038e10 0x7ff6bc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7ff6c00e2450 con 0x7ff6d0135fa0 2024-04-19T01:05:32.798 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.184225.objecter ms_dispatch 0x7ff6d0000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 3 client.184225.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 3 client.184225.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.184225.objecter checking command 1 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7ff6bc0405a0 0x7ff6bc042a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 client.184225.objecter _get_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 15 client.184225.objecter _session_command_op_remove -1 1 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 client.184225.objecter _get_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 client.184225.objecter get_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 15 client.184225.objecter _session_command_op_assign 2 1 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 4 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.184225.objecter _send_command 1 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 --> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6c0008e50 con 0x7ff6bc0405a0 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 20 client.184225.objecter dump_active .. 0 homeless 2024-04-19T01:05:32.799 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_get_version_reply(handle=1 version=707) ==== 24+0+0 (secure 0 0 0) 0x7ff6c00e9050 con 0x7ff6d0135fa0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 monclient: handle_get_version_reply finishing 1 version 707 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d522e640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7ff6bc0405a0 0x7ff6bc042a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d522e640 10 monclient: get_auth_request con 0x7ff6bc0405a0 auth_method 0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff6bc038e10 0x7ff6bc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 10 monclient: get_auth_request con 0x7ff6bc038e10 auth_method 0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.184225.objecter ms_handle_connect 0x7ff6bc0405a0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6d522e640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7ff6bc0405a0 0x7ff6bc042a40 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cffff640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff6bc038e10 0x7ff6bc03b2d0 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 crypto rx=0x7ff6d009d510 tx=0x7ff6c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.778+0000 7ff6cd7fa640 10 client.184225.objecter ms_handle_connect 0x7ff6bc038e10 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.782+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== osd.2 v2:172.21.15.43:6803/1290935720 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6c0008e50 con 0x7ff6bc0405a0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.782+0000 7ff6cd7fa640 10 client.184225.objecter ms_dispatch 0x7ff6d0000ba0 command_reply(tid 1: 0 ) 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.782+0000 7ff6cd7fa640 10 client.184225.objecter _finish_command 1 = system:0 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.782+0000 7ff6cd7fa640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 4 2024-04-19T01:05:32.800 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.782+0000 7ff6cd7fa640 15 client.184225.objecter _session_command_op_remove 2 1 2024-04-19T01:05:32.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 client.184225.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-19T01:05:32.801 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 15 client.184225.objecter _session_command_op_assign -1 2 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter _get_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter _get_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 15 client.184225.objecter _session_command_op_remove -1 2 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter get_session s=0x7ff6bc03fed0 osd=2 3 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 15 client.184225.objecter _session_command_op_assign 2 2 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 4 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 client.184225.objecter _send_command 2 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 --> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff6d00a3590 con 0x7ff6bc0405a0 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6cd7fa640 1 -- 172.21.15.43:0/4152777180 <== osd.2 v2:172.21.15.43:6803/1290935720 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff6c800f3f0 con 0x7ff6bc0405a0 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6cd7fa640 10 client.184225.objecter ms_dispatch 0x7ff6d0000ba0 command_reply(tid 2: 0 ) 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6cd7fa640 10 client.184225.objecter _finish_command 2 = system:0 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6cd7fa640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 4 2024-04-19T01:05:32.802 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6cd7fa640 15 client.184225.objecter _session_command_op_remove 2 2 2024-04-19T01:05:32.803 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:32.803 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:05:32.803 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:32.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 librados: watch_flush enter 2024-04-19T01:05:32.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 librados: watch_flush exit 2024-04-19T01:05:32.804 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 client.184225.objecter close_session for osd.2 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7ff6bc0405a0 msgr2=0x7ff6bc042a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:6803/1290935720,v1:172.21.15.43:6806/1290935720] conn(0x7ff6bc0405a0 0x7ff6bc042a40 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter put_session s=0x7ff6bc03fed0 osd=2 1 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 client.184225.objecter shutdown clearing up homeless session... 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 client.184225.objecter successfully canceled tick 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff6bc038e10 msgr2=0x7ff6bc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff6bc038e10 0x7ff6bc03b2d0 secure :-1 s=READY pgs=3680 cs=0 l=1 rev1=1 crypto rx=0x7ff6d009d510 tx=0x7ff6c4008040 comp rx=0 tx=0).stop 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 10 monclient: shutdown 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff6d0135fa0 msgr2=0x7ff6d00a52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 --2- 172.21.15.43:0/4152777180 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7ff6d0135fa0 0x7ff6d00a52b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00af040 tx=0x7ff6c00191c0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d522e640 1 -- 172.21.15.43:0/4152777180 reap_dead start 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 shutdown_connections 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 >> 172.21.15.43:0/4152777180 conn(0x7ff6d006bfe0 msgr2=0x7ff6d01382a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.805 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 shutdown_connections 2024-04-19T01:05:32.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 -- 172.21.15.43:0/4152777180 wait complete. 2024-04-19T01:05:32.806 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.794+0000 7ff6d6c91640 1 librados: shutdown 2024-04-19T01:05:32.808 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient: get_monmap_and_config 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient: monmap: 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.924 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:32.927281+0000 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:32.927281+0000 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.925 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:32.926 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:32.926 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 1 Processor -- start 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 1 -- start start 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient: init 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient: _add_conns ranks=[10,1,0,5,6,8,7,4,9,2,3] 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f341c138cd0 0x7f341c1390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.936 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient(hunting): picked mon.noname-f con 0x7f341c138cd0 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient(hunting): picked mon.noname-f-legacy con 0x7f341c0a5b80 addr v1:172.21.15.161:6789/0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f341c1395f0 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341c1366d0 con 0x7f341c1395f0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341c0a3dc0 con 0x7f341c0a5b80 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f341c0a4410 con 0x7f341c138cd0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3424181640 10 monclient(hunting): authenticate will time out at 2203.054385s 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f341c1395f0 0x7f341c135fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57500/0 (socket says 172.21.15.43:57500) 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 1 -- 172.21.15.43:0/1053965765 learned_addr learned my addr 172.21.15.43:0/1053965765 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 1 --2- 172.21.15.43:0/1053965765 >> v2:172.21.15.161:3300/0 conn(0x7f341c138cd0 0x7f341c1390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): get_auth_request con 0x7f341c138cd0 auth_method 0 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.937 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703480437 0 0) 0x7f341c1366d0 con 0x7f341c1395f0 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _negotiate 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): my global_id is 184248 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3408003530 con 0x7f341c1395f0 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1817485740 0 0) 0x7f341c0a3dc0 con 0x7f341c0a5b80 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _negotiate 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.938 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): my global_id is 144721 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f341c1366d0 con 0x7f341c0a5b80 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.0 v1:172.21.15.43:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3557531929 0 0) 0x7f3408003530 con 0x7f341c1395f0 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 >> v2:172.21.15.161:3300/0 conn(0x7f341c138cd0 msgr2=0x7f341c1390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 --2- 172.21.15.43:0/1053965765 >> v2:172.21.15.161:3300/0 conn(0x7f341c138cd0 0x7f341c1390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 >> v1:172.21.15.161:6789/0 conn(0x7f341c0a5b80 legacy=0x7f341c1358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 monclient: found mon.noname-a-legacy 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.43:6789/0 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 --> v1:172.21.15.43:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c0a40a0 con 0x7f341c1395f0 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.0 v1:172.21.15.43:6789/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (unknown 2662180337 0 0) 0x7f3414001cf0 con 0x7f341c1395f0 2024-04-19T01:05:32.939 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: dump: 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.940 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] but i'm connected to v1:172.21.15.43:6789/0, reconnecting 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 >> v1:172.21.15.43:6789/0 conn(0x7f341c1395f0 legacy=0x7f341c135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient: _add_conns ranks=[8,7,2,3,1,0,4,5,6] 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3408004810 0x7f3408004bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): picked mon.e con 0x7f3408004810 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3408005130 0x7f34080099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): picked mon.i con 0x7f3408005130 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3408009f00 0x7f340800c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): picked mon.b con 0x7f3408009f00 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:32.941 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f341c0a4410 con 0x7f3408009f00 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f340800c9f0 con 0x7f3408005130 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f340800cba0 con 0x7f3408004810 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3420f1b640 10 monclient(hunting): _finish_auth 0 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3408009f00 0x7f340800c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): get_auth_request con 0x7f3408009f00 auth_method 0 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3408004810 0x7f3408004bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.942 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): get_auth_request con 0x7f3408004810 auth_method 0 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f3421f1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342171c640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3408005130 0x7f34080099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342271e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342171c640 10 monclient(hunting): get_auth_request con 0x7f3408005130 auth_method 0 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.926+0000 7f342171c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342171c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342171c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.943 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 10 monclient(hunting): handle_auth_done global_id 184248 payload 293 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 -- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3408004810 msgr2=0x7f3408004bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f3408004810 0x7f3408004bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 -- 172.21.15.43:0/1053965765 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3408005130 msgr2=0x7f34080099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3408005130 0x7f34080099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 monclient: found mon.b 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:32.944 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 -- 172.21.15.43:0/1053965765 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3408003530 con 0x7f3408009f00 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 5 monclient: authenticate success, global_id 184248 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342171c640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f3408005130 0x7f34080099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 -- 172.21.15.43:0/1053965765 reap_dead start 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f342271e640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3408009f00 0x7f340800c2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f341400c620 tx=0x7f341401cfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d65d5efbbea1798 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f341400c240 con 0x7f3408009f00 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: dump: 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.945 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.946 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34140a9070 con 0x7f3408009f00 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 1 -- 172.21.15.43:0/1053965765 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34140079b0 con 0x7f3408009f00 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 4 set_mon_vals no callback set 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3420f1b640 10 monclient: dump: 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.947 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 10 monclient: shutdown 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 -- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3408009f00 msgr2=0x7f340800c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3408009f00 0x7f340800c2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f341400c620 tx=0x7f341401cfc0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 -- 172.21.15.43:0/1053965765 shutdown_connections 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 --2- 172.21.15.43:0/1053965765 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f3408009f00 0x7f340800c2f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 -- 172.21.15.43:0/1053965765 >> 172.21.15.43:0/1053965765 conn(0x7f341c0ab2f0 msgr2=0x7f341c0ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 -- 172.21.15.43:0/1053965765 shutdown_connections 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 1 -- 172.21.15.43:0/1053965765 wait complete. 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 10 monclient: build_initial_monmap 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.930+0000 7f3424181640 10 monclient: monmap: 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:32.949 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 librados: starting msgr at 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 librados: starting objecter 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 Processor -- start 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- start start 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 librados: setting wanted keys 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 librados: calling monclient init 2024-04-19T01:05:32.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: init 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _add_conns ranks=[1,7,6,4,2,5,3,0,8] 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3408009f00 0x7f341c1d0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): picked mon.noname-f con 0x7f3408009f00 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f341c1d0d80 0x7f341c1d1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): picked mon.noname-i con 0x7f341c1d0d80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 0x7f341c133700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): picked mon.noname-d con 0x7f341c1332c0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f341c0a2010 con 0x7f3408009f00 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f341c0a1e90 con 0x7f341c1332c0 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f341c1390b0 con 0x7f341c1d0d80 2024-04-19T01:05:32.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient(hunting): authenticate will time out at 2203.060872s 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 0x7f341c133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 0x7f341c133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39700/0 (socket says 172.21.15.43:39700) 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 -- 172.21.15.43:0/287121000 learned_addr learned my addr 172.21.15.43:0/287121000 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): get_auth_request con 0x7f341c1332c0 auth_method 0 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f341c1d0d80 0x7f341c1d1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): get_auth_request con 0x7f341c1d0d80 auth_method 0 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3408009f00 0x7f341c1d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): get_auth_request con 0x7f3408009f00 auth_method 0 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342171c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:32.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient(hunting): handle_auth_done global_id 184263 payload 824 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 -- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f341c1d0d80 msgr2=0x7f341c1d1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f341c1d0d80 0x7f341c1d1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 -- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3408009f00 msgr2=0x7f341c1d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3408009f00 0x7f341c1d0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient: _finish_hunting 0 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 monclient: found mon.noname-d 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 -- 172.21.15.43:0/287121000 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341c09d180 con 0x7f341c1332c0 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f342271e640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 0x7f341c133700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f341401b720 tx=0x7f3408005840 comp rx=0 tx=0).ready entity=mon.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 client.?.objecter ms_handle_connect 0x7f341c1332c0 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f3414001cf0 con 0x7f341c1332c0 2024-04-19T01:05:32.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: dump: 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.955 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34140013b0 con 0x7f341c1332c0 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f34140aed30 con 0x7f341c1332c0 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 5 monclient: authenticate success, global_id 184263 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _renew_subs 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f341c09cb50 con 0x7f341c1332c0 2024-04-19T01:05:32.956 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340ffff640 4 set_mon_vals no callback set 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 monclient: dump: 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:32.957 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 client.184263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _renew_subs 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({osdmap=0}) -- 0x7f341c09c870 con 0x7f341c1332c0 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 librados: init done 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 10 monclient: _send_mon_message to mon.d at v2:172.21.15.43:3303/0 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mon.6 v2:172.21.15.43:3303/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f34140b5030 con 0x7f341c1332c0 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f341c068240 con 0x7f341c1332c0 2024-04-19T01:05:32.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3400038e10 0x7f340003b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mon.6 v2:172.21.15.43:3303/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f34140b3020 con 0x7f341c1332c0 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 10 client.184263.objecter ms_dispatch 0x7f341c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 3 client.184263.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 3 client.184263.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f340e7fc640 20 client.184263.objecter dump_active .. 0 homeless 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3400038e10 0x7f340003b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.934+0000 7f3421f1d640 10 monclient: get_auth_request con 0x7f3400038e10 auth_method 0 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.938+0000 7f340e7fc640 10 client.184263.objecter ms_handle_connect 0x7f3400038e10 2024-04-19T01:05:32.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.938+0000 7f3421f1d640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3400038e10 0x7f340003b2d0 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7f3410009b60 tx=0x7f3410009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.938+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mon.6 v2:172.21.15.43:3303/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f34140bc060 con 0x7f341c1332c0 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.938+0000 7f340e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:32.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:32.938+0000 7f340e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f341c0693b0 con 0x7f3400038e10 2024-04-19T01:05:33.103 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f340e7fc640 1 -- 172.21.15.43:0/287121000 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f341c0693b0 con 0x7f3400038e10 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 2 to deep-scrub 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 10 librados: watch_flush enter 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 10 librados: watch_flush exit 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 20 client.184263.objecter shutdown clearing up homeless session... 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 10 client.184263.objecter successfully canceled tick 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3400038e10 msgr2=0x7f340003b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3400038e10 0x7f340003b2d0 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7f3410009b60 tx=0x7f3410009190 comp rx=0 tx=0).stop 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 10 monclient: shutdown 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 msgr2=0x7f341c133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 0x7f341c133700 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f341401b720 tx=0x7f3408005840 comp rx=0 tx=0).stop 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 shutdown_connections 2024-04-19T01:05:33.104 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3400038e10 0x7f340003b2d0 unknown :-1 s=CLOSED pgs=3681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f341c1332c0 0x7f341c133700 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f341c1d0d80 0x7f341c1d1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 --2- 172.21.15.43:0/287121000 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3408009f00 0x7f341c1d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 >> 172.21.15.43:0/287121000 conn(0x7f341c0ab2f0 msgr2=0x7f341c0a6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 shutdown_connections 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 -- 172.21.15.43:0/287121000 wait complete. 2024-04-19T01:05:33.105 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.102+0000 7f3424181640 1 librados: shutdown 2024-04-19T01:05:33.119 INFO:tasks.ceph:Scrubbing osd.3 2024-04-19T01:05:33.119 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:05:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient: get_monmap_and_config 2024-04-19T01:05:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient: monmap: 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:33.239392+0000 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:33.239392+0000 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.249 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 Processor -- start 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 -- start start 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient: init 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient: _add_conns ranks=[2,6,4,7,0,3,9,8,5,10,1] 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ec135fa0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 0x7fc3ec138e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): picked mon.noname-d con 0x7fc3ec136380 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:33.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3ec1393b0 0x7fc3ec139830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ec1393b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec09f4e0 con 0x7fc3ec135fa0 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fc3ec0a0c40 con 0x7fc3ec1393b0 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fc3ec0a0f20 con 0x7fc3ec136380 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f3024640 10 monclient(hunting): authenticate will time out at 2203.366547s 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 0x7fc3ec138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 0x7fc3ec138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39708/0 (socket says 172.21.15.43:39708) 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 -- 172.21.15.43:0/1056244206 learned_addr learned my addr 172.21.15.43:0/1056244206 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:33.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request con 0x7fc3ec136380 auth_method 0 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3ec1393b0 0x7fc3ec139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): get_auth_request con 0x7fc3ec135fa0 auth_method 0 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): get_auth_request con 0x7fc3ec1393b0 auth_method 0 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f15c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_done global_id 184272 payload 274 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 -- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3ec1393b0 msgr2=0x7fc3ec139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3ec1393b0 0x7fc3ec139830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 -- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 msgr2=0x7fc3ec0a6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:33.253 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec0a6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 monclient: found mon.noname-d 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3ebfff640 1 -- 172.21.15.43:0/1056244206 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec09f8b0 con 0x7fc3ec136380 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.238+0000 7fc3f0dc0640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec0a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 0x7fc3ec138e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0098a0 tx=0x7fc3dc01c440 comp rx=0 tx=0).ready entity=mon.6 client_cookie=4c4d513a1eeb7da1 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 1 -- 172.21.15.43:0/1056244206 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3dc0b0070 con 0x7fc3ec136380 2024-04-19T01:05:33.254 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: dump: 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.255 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 1 -- 172.21.15.43:0/1056244206 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3dc0aa810 con 0x7fc3ec136380 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 5 monclient: authenticate success, global_id 184272 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 4 set_mon_vals no callback set 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 1 -- 172.21.15.43:0/1056244206 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3dc0a9300 con 0x7fc3ec136380 2024-04-19T01:05:33.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3eb7fe640 10 monclient: dump: 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.257 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: shutdown 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 msgr2=0x7fc3ec138e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 0x7fc3ec138e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0098a0 tx=0x7fc3dc01c440 comp rx=0 tx=0).stop 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- 172.21.15.43:0/1056244206 shutdown_connections 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fc3ec1393b0 0x7fc3ec139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fc3ec136380 0x7fc3ec138e70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- 172.21.15.43:0/1056244206 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec0a6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- 172.21.15.43:0/1056244206 >> 172.21.15.43:0/1056244206 conn(0x7fc3ec06bfe0 msgr2=0x7fc3ec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- 172.21.15.43:0/1056244206 shutdown_connections 2024-04-19T01:05:33.258 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- 172.21.15.43:0/1056244206 wait complete. 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: monmap: 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.259 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 librados: starting msgr at 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 librados: starting objecter 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 Processor -- start 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- start start 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 librados: setting wanted keys 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 librados: calling monclient init 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: init 2024-04-19T01:05:33.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient: _add_conns ranks=[2,8,5,1,7,4,0,3,6] 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec13f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ec135fa0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec136380 0x7fc3ec134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): picked mon.noname-e con 0x7fc3ec136380 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec1393b0 0x7fc3ec135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): picked mon.noname-h con 0x7fc3ec1393b0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec1429c0 con 0x7fc3ec135fa0 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fc3ec1369b0 con 0x7fc3ec1393b0 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fc3ec142b40 con 0x7fc3ec136380 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f3024640 10 monclient(hunting): authenticate will time out at 2203.371267s 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec13f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec136380 0x7fc3ec134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec13f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:34098/0 (socket says 172.21.15.43:34098) 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 -- 172.21.15.43:0/2286916462 learned_addr learned my addr 172.21.15.43:0/2286916462 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): get_auth_request con 0x7fc3ec135fa0 auth_method 0 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request con 0x7fc3ec136380 auth_method 0 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient(hunting): handle_auth_done global_id 144704 payload 824 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 -- 172.21.15.43:0/2286916462 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec136380 msgr2=0x7fc3ec134bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fc3ec136380 0x7fc3ec134bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 -- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec1393b0 msgr2=0x7fc3ec135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fc3ec1393b0 0x7fc3ec135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 monclient: found mon.noname-b 2024-04-19T01:05:33.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 -- 172.21.15.43:0/2286916462 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec13e800 con 0x7fc3ec135fa0 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3f0dc0640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec13f260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000ebe0 tx=0x7fc3e000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3e97fa640 10 client.?.objecter ms_handle_connect 0x7fc3ec135fa0 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.242+0000 7fc3e97fa640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3e0017070 con 0x7fc3ec135fa0 2024-04-19T01:05:33.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: dump: 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.265 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e0013470 con 0x7fc3ec135fa0 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fc3e0013a90 con 0x7fc3ec135fa0 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: dump: 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.267 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3eaffd640 4 set_mon_vals no callback set 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 5 monclient: authenticate success, global_id 144704 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 monclient: _renew_subs 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ec0a0330 con 0x7fc3ec135fa0 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 client.144704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:33.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 monclient: _renew_subs 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ec13e520 con 0x7fc3ec135fa0 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 1 librados: init done 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 client.144704.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 15 client.144704.objecter _session_command_op_assign -1 1 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 20 client.144704.objecter _get_session osd=-1 returning homeless 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 client.144704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3ec13d350 con 0x7fc3ec135fa0 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fc3e001a070 con 0x7fc3ec135fa0 2024-04-19T01:05:33.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3d8038e10 0x7fc3d803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fc3e000f040 con 0x7fc3ec135fa0 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 client.144704.objecter ms_dispatch 0x7fc3ec000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 3 client.144704.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 3 client.144704.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 client.144704.objecter checking command 1 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fc3d80405b0 0x7fc3d8042a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 client.144704.objecter _get_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 15 client.144704.objecter _session_command_op_remove -1 1 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 client.144704.objecter _get_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 client.144704.objecter get_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 15 client.144704.objecter _session_command_op_assign 3 1 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 4 2024-04-19T01:05:33.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 client.144704.objecter _send_command 1 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 --> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3e0022e50 con 0x7fc3d80405b0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 20 client.144704.objecter dump_active .. 0 homeless 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_get_version_reply(handle=1 version=707) ==== 24+0+0 (secure 0 0 0) 0x7fc3e0050050 con 0x7fc3ec135fa0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 monclient: handle_get_version_reply finishing 1 version 707 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3ebfff640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3d8038e10 0x7fc3d803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3ebfff640 10 monclient: get_auth_request con 0x7fc3d8038e10 auth_method 0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f15c1640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fc3d80405b0 0x7fc3d8042a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f15c1640 10 monclient: get_auth_request con 0x7fc3d80405b0 auth_method 0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3f15c1640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fc3d80405b0 0x7fc3d8042a50 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 client.144704.objecter ms_handle_connect 0x7fc3d80405b0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3e97fa640 10 client.144704.objecter ms_handle_connect 0x7fc3d8038e10 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.246+0000 7fc3ebfff640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3d8038e10 0x7fc3d803b2d0 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ad000 tx=0x7fc3dc0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.250+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== osd.3 v2:172.21.15.161:6801/2204255832 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3e0022e50 con 0x7fc3d80405b0 2024-04-19T01:05:33.271 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.250+0000 7fc3e97fa640 10 client.144704.objecter ms_dispatch 0x7fc3ec000ba0 command_reply(tid 1: 0 ) 2024-04-19T01:05:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.250+0000 7fc3e97fa640 10 client.144704.objecter _finish_command 1 = system:0 2024-04-19T01:05:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.250+0000 7fc3e97fa640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 4 2024-04-19T01:05:33.272 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.250+0000 7fc3e97fa640 15 client.144704.objecter _session_command_op_remove 3 1 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 client.144704.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 15 client.144704.objecter _session_command_op_assign -1 2 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter _get_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter _get_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 15 client.144704.objecter _session_command_op_remove -1 2 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter get_session s=0x7fc3d803fee0 osd=3 3 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 15 client.144704.objecter _session_command_op_assign 3 2 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 4 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 client.144704.objecter _send_command 2 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 --> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc3ec068ce0 con 0x7fc3d80405b0 2024-04-19T01:05:33.273 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3e97fa640 1 -- 172.21.15.43:0/2286916462 <== osd.3 v2:172.21.15.161:6801/2204255832 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc3ec068ce0 con 0x7fc3d80405b0 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3e97fa640 10 client.144704.objecter ms_dispatch 0x7fc3ec000ba0 command_reply(tid 2: 0 ) 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3e97fa640 10 client.144704.objecter _finish_command 2 = system:0 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3e97fa640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 4 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3e97fa640 15 client.144704.objecter _session_command_op_remove 3 2 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:05:33.274 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 librados: watch_flush enter 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 librados: watch_flush exit 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 client.144704.objecter close_session for osd.3 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fc3d80405b0 msgr2=0x7fc3d8042a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6801/2204255832,v1:172.21.15.161:6804/2204255832] conn(0x7fc3d80405b0 0x7fc3d8042a50 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter put_session s=0x7fc3d803fee0 osd=3 1 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 client.144704.objecter shutdown clearing up homeless session... 2024-04-19T01:05:33.277 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 client.144704.objecter successfully canceled tick 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3d8038e10 msgr2=0x7fc3d803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fc3d8038e10 0x7fc3d803b2d0 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0ad000 tx=0x7fc3dc0a6000 comp rx=0 tx=0).stop 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 10 monclient: shutdown 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 msgr2=0x7fc3ec13f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 --2- 172.21.15.43:0/2286916462 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fc3ec135fa0 0x7fc3ec13f260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000ebe0 tx=0x7fc3e000c520 comp rx=0 tx=0).stop 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f15c1640 1 -- 172.21.15.43:0/2286916462 reap_dead start 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 shutdown_connections 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 >> 172.21.15.43:0/2286916462 conn(0x7fc3ec06bfe0 msgr2=0x7fc3ec136b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 shutdown_connections 2024-04-19T01:05:33.278 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 -- 172.21.15.43:0/2286916462 wait complete. 2024-04-19T01:05:33.279 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.262+0000 7fc3f3024640 1 librados: shutdown 2024-04-19T01:05:33.279 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-19T01:05:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.394+0000 7f43cd1a6640 10 monclient: get_monmap_and_config 2024-04-19T01:05:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.394+0000 7f43cd1a6640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.394+0000 7f43cd1a6640 10 monclient: monmap: 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:33.398338+0000 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:33.398338+0000 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.396 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:33.397 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:33.397 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 Processor -- start 2024-04-19T01:05:33.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 -- start start 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient: init 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient: _add_conns ranks=[8,3,0,4,5,7,6,9,2,1,10] 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f43c80a3050 0x7f43c8136e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): picked mon.noname-e con 0x7f43c80a3050 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3430 0x7f43c80a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): picked mon.noname-g con 0x7f43c80a3430 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43c809a0d0 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c8076730 con 0x7f43c809a0d0 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f43c8076480 con 0x7f43c80a3430 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f43c809dd20 con 0x7f43c80a3050 2024-04-19T01:05:33.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43cd1a6640 10 monclient(hunting): authenticate will time out at 2203.525489s 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f43c80a3050 0x7f43c8136e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c759e640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f43c809a0d0 0x7f43c809a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57510/0 (socket says 172.21.15.43:57510) 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c759e640 1 -- 172.21.15.43:0/2586880713 learned_addr learned my addr 172.21.15.43:0/2586880713 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): get_auth_request con 0x7f43c80a3050 auth_method 0 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c659c640 1 --2- 172.21.15.43:0/2586880713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3430 0x7f43c80a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c659c640 10 monclient(hunting): get_auth_request con 0x7f43c80a3430 auth_method 0 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c659c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_done global_id 144755 payload 274 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2586880713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3430 msgr2=0x7f43c80a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 1 --2- 172.21.15.43:0/2586880713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3430 0x7f43c80a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2586880713 >> v1:172.21.15.43:6789/0 conn(0x7f43c809a0d0 legacy=0x7f43c809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c5d9b640 1 -- 172.21.15.43:0/2586880713 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 346735029 0 0) 0x7f43c8076730 con 0x7f43c809a0d0 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 1 monclient: found mon.noname-e 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2586880713 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c809ec70 con 0x7f43c80a3050 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.398+0000 7f43c5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 --2- 172.21.15.43:0/2586880713 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f43c80a3050 0x7f43c8136e40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097f0 tx=0x7f43b801c430 comp rx=0 tx=0).ready entity=mon.8 client_cookie=a699d832621ccc88 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 1 -- 172.21.15.43:0/2586880713 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43b80b0070 con 0x7f43c80a3050 2024-04-19T01:05:33.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: dump: 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.409 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 1 -- 172.21.15.43:0/2586880713 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b80aa810 con 0x7f43c80a3050 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 5 monclient: authenticate success, global_id 144755 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:33.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 1 -- 172.21.15.43:0/2586880713 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43b80a9300 con 0x7f43c80a3050 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 4 set_mon_vals no callback set 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c5d9b640 10 monclient: dump: 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.411 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.412 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: shutdown 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2586880713 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f43c80a3050 msgr2=0x7f43c8136e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2586880713 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f43c80a3050 0x7f43c8136e40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f43b80097f0 tx=0x7f43b801c430 comp rx=0 tx=0).stop 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2586880713 shutdown_connections 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2586880713 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3430 0x7f43c80a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2586880713 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f43c80a3050 0x7f43c8136e40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2586880713 >> 172.21.15.43:0/2586880713 conn(0x7f43c80a4140 msgr2=0x7f43c80a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2586880713 shutdown_connections 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2586880713 wait complete. 2024-04-19T01:05:33.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: monmap: 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.414 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 librados: starting msgr at 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 librados: starting objecter 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 Processor -- start 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- start start 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 librados: setting wanted keys 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 librados: calling monclient init 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: init 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _add_conns ranks=[0,3,5,4,1,6,2,8,7] 2024-04-19T01:05:33.415 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 0x7f43c81d0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): picked mon.noname-a con 0x7f43c809a0d0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3050 0x7f43c813dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): picked mon.noname-g con 0x7f43c80a3050 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43c80a3430 0x7f43c813e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): picked mon.noname-h con 0x7f43c80a3430 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f43c80a3d50 con 0x7f43c809a0d0 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f43c81416f0 con 0x7f43c80a3050 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f43c809df40 con 0x7f43c80a3430 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient(hunting): authenticate will time out at 2203.530235s 2024-04-19T01:05:33.416 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 0x7f43c81d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 0x7f43c81d0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51858/0 (socket says 172.21.15.43:51858) 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2919861180 learned_addr learned my addr 172.21.15.43:0/2919861180 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): get_auth_request con 0x7f43c809a0d0 auth_method 0 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient(hunting): handle_auth_done global_id 184266 payload 824 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43c80a3430 msgr2=0x7f43c813e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43c80a3430 0x7f43c813e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.417 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c659c640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3050 0x7f43c813dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3050 msgr2=0x7f43c813dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3050 0x7f43c813dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 monclient: found mon.noname-a 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 -- 172.21.15.43:0/2919861180 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c80a1ab0 con 0x7f43c809a0d0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c6d9d640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 0x7f43c81d0800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009710 tx=0x7f43b80191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 client.?.objecter ms_handle_connect 0x7f43c809a0d0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43b80b0040 con 0x7f43c809a0d0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:33.418 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: dump: 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.419 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 5 monclient: authenticate success, global_id 184266 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b80044b0 con 0x7f43c809a0d0 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _renew_subs 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c808fe20 con 0x7f43c809a0d0 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f43b80aacf0 con 0x7f43c809a0d0 2024-04-19T01:05:33.420 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 client.184266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 10 monclient: dump: 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.421 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43c4d99640 4 set_mon_vals no callback set 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _renew_subs 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c809d1f0 con 0x7f43c809a0d0 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 1 librados: init done 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mon.0 v2:172.21.15.43:3300/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f43b8004040 con 0x7f43c809a0d0 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.422 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.402+0000 7f43cd1a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.43:3300/0 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c80008e0 con 0x7f43c809a0d0 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43b4038e10 0x7f43b403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mon.0 v2:172.21.15.43:3300/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f43b80e2cf0 con 0x7f43c809a0d0 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 10 client.184266.objecter ms_dispatch 0x7f43c8000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 3 client.184266.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 3 client.184266.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 20 client.184266.objecter dump_active .. 0 homeless 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43c659c640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43b4038e10 0x7f43b403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43c659c640 10 monclient: get_auth_request con 0x7f43b4038e10 auth_method 0 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43af7fe640 10 client.184266.objecter ms_handle_connect 0x7f43b4038e10 2024-04-19T01:05:33.423 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.406+0000 7f43c659c640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43b4038e10 0x7f43b403b2d0 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7f43c809e650 tx=0x7f43bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.410+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mon.0 v2:172.21.15.43:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f43b80160b0 con 0x7f43c809a0d0 2024-04-19T01:05:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.410+0000 7f43af7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.424 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.410+0000 7f43af7fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:33.570 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.570+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f43c809bd40 con 0x7f43b4038e10 2024-04-19T01:05:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43af7fe640 1 -- 172.21.15.43:0/2919861180 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f43c809bd40 con 0x7f43b4038e10 2024-04-19T01:05:33.571 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 3 to deep-scrub 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 10 librados: watch_flush enter 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 10 librados: watch_flush exit 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 20 client.184266.objecter shutdown clearing up homeless session... 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 10 client.184266.objecter successfully canceled tick 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43b4038e10 msgr2=0x7f43b403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43b4038e10 0x7f43b403b2d0 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7f43c809e650 tx=0x7f43bc008040 comp rx=0 tx=0).stop 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 10 monclient: shutdown 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 msgr2=0x7f43c81d0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.575 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 0x7f43c81d0800 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009710 tx=0x7f43b80191d0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 shutdown_connections 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f43b4038e10 0x7f43b403b2d0 unknown :-1 s=CLOSED pgs=3683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f43c80a3430 0x7f43c813e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f43c80a3050 0x7f43c813dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 --2- 172.21.15.43:0/2919861180 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f43c809a0d0 0x7f43c81d0800 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 >> 172.21.15.43:0/2919861180 conn(0x7f43c80a4140 msgr2=0x7f43c8095900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 shutdown_connections 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 -- 172.21.15.43:0/2919861180 wait complete. 2024-04-19T01:05:33.576 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.574+0000 7f43cd1a6640 1 librados: shutdown 2024-04-19T01:05:33.589 INFO:tasks.ceph:Scrubbing osd.4 2024-04-19T01:05:33.589 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.718+0000 7fd17f743640 10 monclient: get_monmap_and_config 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.718+0000 7fd17f743640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.718+0000 7fd17f743640 10 monclient: monmap: 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:33.720925+0000 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:33.720925+0000 2024-04-19T01:05:33.719 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:33.720 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 Processor -- start 2024-04-19T01:05:33.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 -- start start 2024-04-19T01:05:33.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient: init 2024-04-19T01:05:33.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient: _add_conns ranks=[6,4,2,1,3,7,8,10,5,9,0] 2024-04-19T01:05:33.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd1780a29a0 0x7fd178135380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): picked mon.noname-d con 0x7fd1780a29a0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd1780730a0 0x7fd1780734e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): picked mon.noname-c con 0x7fd1780730a0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd17809ba70 0x7fd17809bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): picked mon.noname-b con 0x7fd17809ba70 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7fd1780a2d80 con 0x7fd17809ba70 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7fd17809cdb0 con 0x7fd1780730a0 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fd17809dbd0 con 0x7fd1780a29a0 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17f743640 10 monclient(hunting): authenticate will time out at 2203.854715s 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd17809ba70 0x7fd17809bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd17809ba70 0x7fd17809bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:34104/0 (socket says 172.21.15.43:34104) 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 1 -- 172.21.15.43:0/283394621 learned_addr learned my addr 172.21.15.43:0/283394621 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): get_auth_request con 0x7fd17809ba70 auth_method 0 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd1780730a0 0x7fd1780734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): get_auth_request con 0x7fd1780730a0 auth_method 0 2024-04-19T01:05:33.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd1780a29a0 0x7fd178135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): get_auth_request con 0x7fd1780a29a0 auth_method 0 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.731 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17d4df640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17dce0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient(hunting): handle_auth_done global_id 184243 payload 274 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 -- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd1780a29a0 msgr2=0x7fd178135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd1780a29a0 0x7fd178135380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 -- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd17809ba70 msgr2=0x7fd17809bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd17809ba70 0x7fd17809bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.732 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 monclient: found mon.noname-c 2024-04-19T01:05:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.726+0000 7fd17ccde640 1 -- 172.21.15.43:0/283394621 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17809e530 con 0x7fd1780730a0 2024-04-19T01:05:33.733 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.730+0000 7fd17ccde640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd1780730a0 0x7fd1780734e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd16800a9c0 tx=0x7fd168020a30 comp rx=0 tx=0).ready entity=mon.4 client_cookie=b504b36c3b2f29bc server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 1 -- 172.21.15.43:0/283394621 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1680b6070 con 0x7fd1780730a0 2024-04-19T01:05:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.739 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: dump: 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.740 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 1 -- 172.21.15.43:0/283394621 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1680a9c30 con 0x7fd1780730a0 2024-04-19T01:05:33.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 1 -- 172.21.15.43:0/283394621 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd16800b040 con 0x7fd1780730a0 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd16ffff640 10 monclient: dump: 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.742 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd17f743640 5 monclient: authenticate success, global_id 184243 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd17f743640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.734+0000 7fd17f743640 4 set_mon_vals no callback set 2024-04-19T01:05:33.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.738+0000 7fd17f743640 10 monclient: shutdown 2024-04-19T01:05:33.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.738+0000 7fd17f743640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.738+0000 7fd17f743640 1 -- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd1780730a0 msgr2=0x7fd1780734e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.744 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.738+0000 7fd17f743640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd1780730a0 0x7fd1780734e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd16800a9c0 tx=0x7fd168020a30 comp rx=0 tx=0).stop 2024-04-19T01:05:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/283394621 shutdown_connections 2024-04-19T01:05:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7fd17809ba70 0x7fd17809bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7fd1780730a0 0x7fd1780734e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 --2- 172.21.15.43:0/283394621 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fd1780a29a0 0x7fd178135380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.745 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/283394621 >> 172.21.15.43:0/283394621 conn(0x7fd1780a75f0 msgr2=0x7fd1780a3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/283394621 shutdown_connections 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/283394621 wait complete. 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient: monmap: 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.746 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 librados: starting msgr at 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 librados: starting objecter 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 Processor -- start 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- start start 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 librados: setting wanted keys 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 librados: calling monclient init 2024-04-19T01:05:33.747 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient: init 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient: _add_conns ranks=[3,0,8,4,6,2,1,7,5] 2024-04-19T01:05:33.752 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd1780730a0 0x7fd178072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): picked mon.noname-g con 0x7fd1780730a0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd17809ba70 0x7fd1781d5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): picked mon.noname-a con 0x7fd17809ba70 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd1780a29a0 0x7fd1781d5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): picked mon.noname-e con 0x7fd1780a29a0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd1780a29a0 0x7fd1781d5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd1780a29a0 0x7fd1781d5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:51754/0 (socket says 172.21.15.43:51754) 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd17809ba70 0x7fd1781d5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.753 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 1 -- 172.21.15.43:0/2953675631 learned_addr learned my addr 172.21.15.43:0/2953675631 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7fd178135a40 con 0x7fd17809ba70 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fd1781358c0 con 0x7fd1780730a0 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fd17813e8e0 con 0x7fd1780a29a0 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17f743640 10 monclient(hunting): authenticate will time out at 2203.875605s 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): get_auth_request con 0x7fd1780a29a0 auth_method 0 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): get_auth_request con 0x7fd17809ba70 auth_method 0 2024-04-19T01:05:33.754 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17dce0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.746+0000 7fd17ccde640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 10 monclient(hunting): handle_auth_done global_id 144764 payload 824 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 -- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd1780730a0 msgr2=0x7fd178072130 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fd1780730a0 0x7fd178072130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 -- 172.21.15.43:0/2953675631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd17809ba70 msgr2=0x7fd1781d5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7fd17809ba70 0x7fd1781d5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.755 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 monclient: found mon.noname-e 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1781d63b0 con 0x7fd1780a29a0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17dce0640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd1780a29a0 0x7fd1781d5e70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd170009d60 tx=0x7fd17000c490 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 client.?.objecter ms_handle_connect 0x7fd1780a29a0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd1700039a0 con 0x7fd1780a29a0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: dump: 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.756 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.757 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd170003b80 con 0x7fd1780a29a0 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 5 monclient: authenticate success, global_id 144764 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 monclient: _renew_subs 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd178072cd0 con 0x7fd1780a29a0 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fd170003d30 con 0x7fd1780a29a0 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:33.758 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 monclient: dump: 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.759 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16f7fe640 4 set_mon_vals no callback set 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 client.144764.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 monclient: _renew_subs 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fd178137670 con 0x7fd1780a29a0 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 1 librados: init done 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fd170012050 con 0x7fd1780a29a0 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd15c038e10 0x7fd15c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fd17004d170 con 0x7fd1780a29a0 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 10 client.144764.objecter ms_dispatch 0x7fd178000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 3 client.144764.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 3 client.144764.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:33.760 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd16dffb640 20 client.144764.objecter dump_active .. 0 homeless 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17d4df640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd15c038e10 0x7fd15c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17d4df640 10 monclient: get_auth_request con 0x7fd15c038e10 auth_method 0 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 client.144764.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 15 client.144764.objecter _session_command_op_assign -1 1 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7fd17813bcb0 0x7fd178136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 20 client.144764.objecter _get_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 20 client.144764.objecter _get_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 15 client.144764.objecter _session_command_op_remove -1 1 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 20 client.144764.objecter get_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 15 client.144764.objecter _session_command_op_assign 4 1 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 4 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 10 client.144764.objecter _send_command 1 2024-04-19T01:05:33.761 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd178135410 con 0x7fd17813bcb0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17ccde640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7fd17813bcb0 0x7fd178136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.750+0000 7fd17ccde640 10 monclient: get_auth_request con 0x7fd17813bcb0 auth_method 0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd17d4df640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd15c038e10 0x7fd15c03b2d0 secure :-1 s=READY pgs=3684 cs=0 l=1 rev1=1 crypto rx=0x7fd1740098d0 tx=0x7fd174009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 10 client.144764.objecter ms_handle_connect 0x7fd15c038e10 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 10 client.144764.objecter ms_handle_connect 0x7fd17813bcb0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd17ccde640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7fd17813bcb0 0x7fd178136390 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== osd.4 v2:172.21.15.161:6800/3653492594 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd178135410 con 0x7fd17813bcb0 2024-04-19T01:05:33.762 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 10 client.144764.objecter ms_dispatch 0x7fd178000ba0 command_reply(tid 1: 0 ) 2024-04-19T01:05:33.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 10 client.144764.objecter _finish_command 1 = system:0 2024-04-19T01:05:33.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 4 2024-04-19T01:05:33.763 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.754+0000 7fd16dffb640 15 client.144764.objecter _session_command_op_remove 4 1 2024-04-19T01:05:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 client.144764.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-19T01:05:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 15 client.144764.objecter _session_command_op_assign -1 2 2024-04-19T01:05:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter _get_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter _get_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.782 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 15 client.144764.objecter _session_command_op_remove -1 2 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter get_session s=0x7fd17809d6d0 osd=4 3 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 15 client.144764.objecter _session_command_op_assign 4 2 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 4 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 client.144764.objecter _send_command 2 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 --> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd178068b10 con 0x7fd17813bcb0 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd16dffb640 1 -- 172.21.15.43:0/2953675631 <== osd.4 v2:172.21.15.161:6800/3653492594 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd178068b10 con 0x7fd17813bcb0 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd16dffb640 10 client.144764.objecter ms_dispatch 0x7fd178000ba0 command_reply(tid 2: 0 ) 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd16dffb640 10 client.144764.objecter _finish_command 2 = system:0 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd16dffb640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 4 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd16dffb640 15 client.144764.objecter _session_command_op_remove 4 2 2024-04-19T01:05:33.783 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:33.784 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:05:33.784 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 librados: watch_flush enter 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 librados: watch_flush exit 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 client.144764.objecter close_session for osd.4 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7fd17813bcb0 msgr2=0x7fd178136390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6800/3653492594,v1:172.21.15.161:6803/3653492594] conn(0x7fd17813bcb0 0x7fd178136390 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter put_session s=0x7fd17809d6d0 osd=4 1 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 client.144764.objecter shutdown clearing up homeless session... 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 client.144764.objecter successfully canceled tick 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd15c038e10 msgr2=0x7fd15c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.785 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fd15c038e10 0x7fd15c03b2d0 secure :-1 s=READY pgs=3684 cs=0 l=1 rev1=1 crypto rx=0x7fd1740098d0 tx=0x7fd174009190 comp rx=0 tx=0).stop 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 10 monclient: shutdown 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd1780a29a0 msgr2=0x7fd1781d5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 --2- 172.21.15.43:0/2953675631 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fd1780a29a0 0x7fd1781d5e70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd170009d60 tx=0x7fd17000c490 comp rx=0 tx=0).stop 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17dce0640 1 -- 172.21.15.43:0/2953675631 reap_dead start 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 shutdown_connections 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 >> 172.21.15.43:0/2953675631 conn(0x7fd1780a75f0 msgr2=0x7fd178133920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 shutdown_connections 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 -- 172.21.15.43:0/2953675631 wait complete. 2024-04-19T01:05:33.786 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.782+0000 7fd17f743640 1 librados: shutdown 2024-04-19T01:05:33.803 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-19T01:05:33.920 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 10 monclient: get_monmap_and_config 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 10 monclient: monmap: 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:33.921046+0000 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:33.921046+0000 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.921 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 1 Processor -- start 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 1 -- start start 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 10 monclient: init 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.918+0000 7f9e20e5b640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.922 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient: _add_conns ranks=[0,4,7,6,9,3,2,10,1,5,8] 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e1c09e4c0 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9e1c136e60 0x7f9e1c13bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): picked mon.noname-c con 0x7f9e1c136e60 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9e1c13c1e0 0x7f9e1c13e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): picked mon.noname-i con 0x7f9e1c13c1e0 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e1c137260 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f9e1c0a1c50 con 0x7f9e1c136e60 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f9e1c0a1e20 con 0x7f9e1c13c1e0 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient(hunting): authenticate will time out at 2204.048131s 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1a59c640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f9e1c09e4c0 0x7f9e1c09e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57514/0 (socket says 172.21.15.43:57514) 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1a59c640 1 -- 172.21.15.43:0/1776335537 learned_addr learned my addr 172.21.15.43:0/1776335537 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:33.923 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9e1c136e60 0x7f9e1c13bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): get_auth_request con 0x7f9e1c136e60 auth_method 0 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1ad9d640 1 --2- 172.21.15.43:0/1776335537 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9e1c13c1e0 0x7f9e1c13e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1ad9d640 10 monclient(hunting): get_auth_request con 0x7f9e1c13c1e0 auth_method 0 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.924 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 1 -- 172.21.15.43:0/1776335537 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724737773 0 0) 0x7f9e0c001120 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient(hunting): _negotiate 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient(hunting): my global_id is 184284 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 1 -- 172.21.15.43:0/1776335537 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e04003530 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient(hunting): handle_auth_done global_id 184252 payload 274 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 -- 172.21.15.43:0/1776335537 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9e1c13c1e0 msgr2=0x7f9e1c13e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 --2- 172.21.15.43:0/1776335537 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9e1c13c1e0 0x7f9e1c13e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 -- 172.21.15.43:0/1776335537 >> v1:172.21.15.43:6789/0 conn(0x7f9e1c09e4c0 legacy=0x7f9e1c09e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 monclient: found mon.noname-c 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:33.925 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 -- 172.21.15.43:0/1776335537 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c06c760 con 0x7f9e1c136e60 2024-04-19T01:05:33.931 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e19d9b640 1 --2- 172.21.15.43:0/1776335537 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9e1c136e60 0x7f9e1c13bca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9e100097f0 tx=0x7f9e1001c3a0 comp rx=0 tx=0).ready entity=mon.4 client_cookie=560fde7d7f71e7a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 1 -- 172.21.15.43:0/1776335537 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9e100b0070 con 0x7f9e1c136e60 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: dump: 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.932 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.933 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 1 -- 172.21.15.43:0/1776335537 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e100aa810 con 0x7f9e1c136e60 2024-04-19T01:05:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 1 -- 172.21.15.43:0/1776335537 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9e100a9300 con 0x7f9e1c136e60 2024-04-19T01:05:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 5 monclient: authenticate success, global_id 184252 2024-04-19T01:05:33.934 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:33.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 4 set_mon_vals no callback set 2024-04-19T01:05:33.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:33.940 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e1959a640 10 monclient: dump: 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.947 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 10 monclient: shutdown 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 -- 172.21.15.43:0/1776335537 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9e1c136e60 msgr2=0x7f9e1c13bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/1776335537 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9e1c136e60 0x7f9e1c13bca0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9e100097f0 tx=0x7f9e1001c3a0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 -- 172.21.15.43:0/1776335537 shutdown_connections 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/1776335537 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f9e1c13c1e0 0x7f9e1c13e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/1776335537 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f9e1c136e60 0x7f9e1c13bca0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.922+0000 7f9e20e5b640 1 -- 172.21.15.43:0/1776335537 >> 172.21.15.43:0/1776335537 conn(0x7f9e1c0a85c0 msgr2=0x7f9e1c0a4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:33.948 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- 172.21.15.43:0/1776335537 shutdown_connections 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- 172.21.15.43:0/1776335537 wait complete. 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: build_initial_monmap 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: monmap: 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:33.949 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 librados: starting msgr at 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 librados: starting objecter 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 Processor -- start 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- start start 2024-04-19T01:05:33.950 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 librados: setting wanted keys 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 librados: calling monclient init 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: init 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _add_conns ranks=[8,5,0,4,7,1,2,6,3] 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 0x7f9e1c1ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): picked mon.noname-e con 0x7f9e1c09e4c0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9e1c136e60 0x7f9e1c137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): picked mon.noname-h con 0x7f9e1c136e60 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 0x7f9e1c137fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1c13c1e0 addr [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- --> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] -- mon_getmap magic: 0 -- 0x7f9e1c1416c0 con 0x7f9e1c13c1e0 2024-04-19T01:05:33.951 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f9e1c141540 con 0x7f9e1c136e60 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f9e1c141840 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient(hunting): authenticate will time out at 2204.053399s 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 0x7f9e1c1ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 0x7f9e1c137fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 0x7f9e1c1ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:51764/0 (socket says 172.21.15.43:51764) 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 -- 172.21.15.43:0/440919519 learned_addr learned my addr 172.21.15.43:0/440919519 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:33.952 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): get_auth_request con 0x7f9e1c09e4c0 auth_method 0 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 1 --2- >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 0x7f9e1c137fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51878/0 (socket says 172.21.15.43:51878) 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): get_auth_request con 0x7f9e1c13c1e0 auth_method 0 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.953 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9e1c136e60 0x7f9e1c137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 10 monclient(hunting): get_auth_request con 0x7f9e1c136e60 auth_method 0 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient(hunting): handle_auth_done global_id 144773 payload 824 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 -- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9e1c136e60 msgr2=0x7f9e1c137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9e1c136e60 0x7f9e1c137a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.954 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 -- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 msgr2=0x7f9e1c137fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 0x7f9e1c137fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient: _finish_hunting 0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 monclient: found mon.noname-e 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 -- 172.21.15.43:0/440919519 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e1c13b140 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1ad9d640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 0x7f9e1c137fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e1a59c640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 0x7f9e1c1ced30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00c730 tx=0x7f9e0c00cbf0 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 client.?.objecter ms_handle_connect 0x7f9e1c09e4c0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9e0c017070 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:33.955 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: dump: 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.956 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: _finish_auth 0 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 5 monclient: authenticate success, global_id 144773 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e0c005d30 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _renew_subs 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e1c1395d0 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:33.957 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f9e0c0074e0 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 monclient: dump: 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:33.958 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 client.144773.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _renew_subs 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e1c13b420 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e18d99640 4 set_mon_vals no callback set 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 librados: init done 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.959 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e1c1399e0 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f9e0c007680 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9dfc038e10 0x7f9dfc03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f9e0c0483b0 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 10 client.144773.objecter ms_dispatch 0x7f9e1c000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 3 client.144773.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 3 client.144773.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e0affd640 20 client.144773.objecter dump_active .. 0 homeless 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9dfc038e10 0x7f9dfc03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.926+0000 7f9e19d9b640 10 monclient: get_auth_request con 0x7f9dfc038e10 auth_method 0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.930+0000 7f9e19d9b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9dfc038e10 0x7f9dfc03b2d0 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7f9e100ad000 tx=0x7f9e100a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.930+0000 7f9e0affd640 10 client.144773.objecter ms_handle_connect 0x7f9dfc038e10 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.930+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9e1c137260 con 0x7f9e1c09e4c0 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.930+0000 7f9e0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:33.960 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:33.930+0000 7f9e0affd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:34.093 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.094+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f9e1c133cf0 con 0x7f9dfc038e10 2024-04-19T01:05:34.094 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.094+0000 7f9e0affd640 1 -- 172.21.15.43:0/440919519 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9e1c133cf0 con 0x7f9dfc038e10 2024-04-19T01:05:34.095 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 4 to deep-scrub 2024-04-19T01:05:34.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 10 librados: watch_flush enter 2024-04-19T01:05:34.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 10 librados: watch_flush exit 2024-04-19T01:05:34.097 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 20 client.144773.objecter shutdown clearing up homeless session... 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 10 client.144773.objecter successfully canceled tick 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9dfc038e10 msgr2=0x7f9dfc03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9dfc038e10 0x7f9dfc03b2d0 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7f9e100ad000 tx=0x7f9e100a6000 comp rx=0 tx=0).stop 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 10 monclient: shutdown 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 msgr2=0x7f9e1c1ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 0x7f9e1c1ced30 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00c730 tx=0x7f9e0c00cbf0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 shutdown_connections 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f9e1c13c1e0 0x7f9e1c137fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f9e1c136e60 0x7f9e1c137a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f9dfc038e10 0x7f9dfc03b2d0 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 --2- 172.21.15.43:0/440919519 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f9e1c09e4c0 0x7f9e1c1ced30 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.098 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 >> 172.21.15.43:0/440919519 conn(0x7f9e1c0a85c0 msgr2=0x7f9e1c0a5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 shutdown_connections 2024-04-19T01:05:34.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 -- 172.21.15.43:0/440919519 wait complete. 2024-04-19T01:05:34.099 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.098+0000 7f9e20e5b640 1 librados: shutdown 2024-04-19T01:05:34.112 INFO:tasks.ceph:Scrubbing osd.5 2024-04-19T01:05:34.112 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient: get_monmap_and_config 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient: build_initial_monmap 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient: monmap: 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:34.232042+0000 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:34.232042+0000 2024-04-19T01:05:34.229 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:34.230 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 Processor -- start 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 -- start start 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient: init 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient: _add_conns ranks=[2,0,10,8,6,9,3,5,1,7,4] 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440135fa0 0x7ff4400a6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff440135fa0 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff440136380 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff440139580 0x7ff4401411b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): picked mon.noname-f con 0x7ff440139580 addr v2:172.21.15.161:3300/0 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff44009f4e0 con 0x7ff440136380 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4400a0c40 con 0x7ff440139580 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.233 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff4400a0f20 con 0x7ff440135fa0 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.230+0000 7ff4484c9640 10 monclient(hunting): authenticate will time out at 2204.359900s 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440135fa0 0x7ff4400a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445a64640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7ff440136380 0x7ff440138e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57528/0 (socket says 172.21.15.43:57528) 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445a64640 1 -- 172.21.15.43:0/4093027232 learned_addr learned my addr 172.21.15.43:0/4093027232 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): get_auth_request con 0x7ff440135fa0 auth_method 0 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.234 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient(hunting): handle_auth_done global_id 144722 payload 274 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446a66640 1 --2- 172.21.15.43:0/4093027232 >> v2:172.21.15.161:3300/0 conn(0x7ff440139580 0x7ff4401411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 -- 172.21.15.43:0/4093027232 >> v2:172.21.15.161:3300/0 conn(0x7ff440139580 msgr2=0x7ff4401411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 --2- 172.21.15.43:0/4093027232 >> v2:172.21.15.161:3300/0 conn(0x7ff440139580 0x7ff4401411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 1 -- 172.21.15.43:0/4093027232 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698127789 0 0) 0x7ff44009f4e0 con 0x7ff440136380 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 -- 172.21.15.43:0/4093027232 >> v1:172.21.15.43:6789/0 conn(0x7ff440136380 legacy=0x7ff440138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient: _finish_hunting 0 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 monclient: found mon.noname-b 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 -- 172.21.15.43:0/4093027232 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44009f8b0 con 0x7ff440135fa0 2024-04-19T01:05:34.235 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff446265640 1 --2- 172.21.15.43:0/4093027232 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440135fa0 0x7ff4400a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0097c0 tx=0x7ff43c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f471f1d4f7b3b372 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 1 -- 172.21.15.43:0/4093027232 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff43c0b2070 con 0x7ff440135fa0 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: dump: 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.243 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: _finish_auth 0 2024-04-19T01:05:34.244 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: _check_auth_tickets 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 5 monclient: authenticate success, global_id 144722 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 1 -- 172.21.15.43:0/4093027232 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff43c0ab7f0 con 0x7ff440135fa0 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 1 -- 172.21.15.43:0/4093027232 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff43c0abe10 con 0x7ff440135fa0 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 4 set_mon_vals no callback set 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff445263640 10 monclient: dump: 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.245 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 10 monclient: shutdown 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.246 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 -- 172.21.15.43:0/4093027232 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440135fa0 msgr2=0x7ff4400a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 --2- 172.21.15.43:0/4093027232 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440135fa0 0x7ff4400a6180 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0097c0 tx=0x7ff43c01c360 comp rx=0 tx=0).stop 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 -- 172.21.15.43:0/4093027232 shutdown_connections 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 --2- 172.21.15.43:0/4093027232 >> v2:172.21.15.161:3300/0 conn(0x7ff440139580 0x7ff4401411b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 --2- 172.21.15.43:0/4093027232 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440135fa0 0x7ff4400a6180 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 -- 172.21.15.43:0/4093027232 >> 172.21.15.43:0/4093027232 conn(0x7ff44006bfe0 msgr2=0x7ff44006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 -- 172.21.15.43:0/4093027232 shutdown_connections 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.234+0000 7ff4484c9640 1 -- 172.21.15.43:0/4093027232 wait complete. 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: build_initial_monmap 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: monmap: 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:34.247 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 librados: starting msgr at 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 librados: starting objecter 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 Processor -- start 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- start start 2024-04-19T01:05:34.248 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 librados: setting wanted keys 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 librados: calling monclient init 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: init 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _add_conns ranks=[8,2,7,1,4,5,6,0,3] 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff440135fa0 0x7ff44009be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): picked mon.noname-e con 0x7ff440135fa0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440136380 0x7ff44009c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff440136380 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff440139580 0x7ff44009c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): picked mon.noname-i con 0x7ff440139580 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7ff440143f10 con 0x7ff440136380 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7ff440144090 con 0x7ff440139580 2024-04-19T01:05:34.249 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7ff440144210 con 0x7ff440135fa0 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient(hunting): authenticate will time out at 2204.365086s 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440136380 0x7ff44009c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff440135fa0 0x7ff44009be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440136380 0x7ff44009c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:34122/0 (socket says 172.21.15.43:34122) 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 -- 172.21.15.43:0/2826189779 learned_addr learned my addr 172.21.15.43:0/2826189779 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): get_auth_request con 0x7ff440135fa0 auth_method 0 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): get_auth_request con 0x7ff440136380 auth_method 0 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.250 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446a66640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff440139580 0x7ff44009c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient(hunting): handle_auth_done global_id 144731 payload 824 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 -- 172.21.15.43:0/2826189779 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff440135fa0 msgr2=0x7ff44009be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff440135fa0 0x7ff44009be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 -- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff440139580 msgr2=0x7ff44009c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff440139580 0x7ff44009c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient: _finish_hunting 0 2024-04-19T01:05:34.251 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 monclient: found mon.noname-b 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 -- 172.21.15.43:0/2826189779 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff44009fe50 con 0x7ff440136380 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446a66640 10 monclient: get_auth_request con 0x7ff440139580 auth_method 0 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7ff440135fa0 0x7ff44009be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446a66640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7ff440139580 0x7ff44009c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff445a64640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440136380 0x7ff44009c3a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c420 tx=0x7ff43000c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 client.?.objecter ms_handle_connect 0x7ff440136380 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4300047f0 con 0x7ff440136380 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: dump: 2024-04-19T01:05:34.252 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.253 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.254 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.255 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: _finish_auth 0 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff430004990 con 0x7ff440136380 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 5 monclient: authenticate success, global_id 144731 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _renew_subs 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff44009d8e0 con 0x7ff440136380 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:34.256 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7ff4300133c0 con 0x7ff440136380 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff444a62640 4 set_mon_vals no callback set 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 client.144731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: dump: 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.257 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.258 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.259 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _renew_subs 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4400a3600 con 0x7ff440136380 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 librados: init done 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 client.144731.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 15 client.144731.objecter _session_command_op_assign -1 1 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 20 client.144731.objecter _get_session osd=-1 returning homeless 2024-04-19T01:05:34.260 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 client.144731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.43:3301/0 2024-04-19T01:05:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff440068ce0 con 0x7ff440136380 2024-04-19T01:05:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== mon.2 v2:172.21.15.43:3301/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7ff430011070 con 0x7ff440136380 2024-04-19T01:05:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff424038e10 0x7ff42403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== mon.2 v2:172.21.15.43:3301/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7ff430048d00 con 0x7ff440136380 2024-04-19T01:05:34.261 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 client.144731.objecter ms_dispatch 0x7ff440000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 3 client.144731.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 3 client.144731.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 client.144731.objecter checking command 1 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7ff4240405a0 0x7ff424042a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 client.144731.objecter _get_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 15 client.144731.objecter _session_command_op_remove -1 1 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 client.144731.objecter _get_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 client.144731.objecter get_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 15 client.144731.objecter _session_command_op_assign 5 1 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 4 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 client.144731.objecter _send_command 1 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 --> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff43001be70 con 0x7ff4240405a0 2024-04-19T01:05:34.262 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 20 client.144731.objecter dump_active .. 0 homeless 2024-04-19T01:05:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== mon.2 v2:172.21.15.43:3301/0 6 ==== mon_get_version_reply(handle=1 version=707) ==== 24+0+0 (secure 0 0 0) 0x7ff43004b3d0 con 0x7ff440136380 2024-04-19T01:05:34.263 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff436ffd640 10 monclient: handle_get_version_reply finishing 1 version 707 2024-04-19T01:05:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff424038e10 0x7ff42403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446265640 10 monclient: get_auth_request con 0x7ff424038e10 auth_method 0 2024-04-19T01:05:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446a66640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7ff4240405a0 0x7ff424042a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.238+0000 7ff446a66640 10 monclient: get_auth_request con 0x7ff4240405a0 auth_method 0 2024-04-19T01:05:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 10 client.144731.objecter ms_handle_connect 0x7ff4240405a0 2024-04-19T01:05:34.264 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff446a66640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7ff4240405a0 0x7ff424042a40 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 10 client.144731.objecter ms_handle_connect 0x7ff424038e10 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff446265640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff424038e10 0x7ff42403b2d0 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0af000 tx=0x7ff43c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== osd.5 v2:172.21.15.161:6802/760479844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff43001be70 con 0x7ff4240405a0 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 10 client.144731.objecter ms_dispatch 0x7ff440000ba0 command_reply(tid 1: 0 ) 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 10 client.144731.objecter _finish_command 1 = system:0 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 4 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.242+0000 7ff436ffd640 15 client.144731.objecter _session_command_op_remove 5 1 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 client.144731.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 15 client.144731.objecter _session_command_op_assign -1 2 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter _get_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter _get_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 15 client.144731.objecter _session_command_op_remove -1 2 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter get_session s=0x7ff42403fed0 osd=5 3 2024-04-19T01:05:34.265 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 15 client.144731.objecter _session_command_op_assign 5 2 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 4 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 client.144731.objecter _send_command 2 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 --> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff44009e1a0 con 0x7ff4240405a0 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff436ffd640 1 -- 172.21.15.43:0/2826189779 <== osd.5 v2:172.21.15.161:6802/760479844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff44009e1a0 con 0x7ff4240405a0 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff436ffd640 10 client.144731.objecter ms_dispatch 0x7ff440000ba0 command_reply(tid 2: 0 ) 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff436ffd640 10 client.144731.objecter _finish_command 2 = system:0 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff436ffd640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 4 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff436ffd640 15 client.144731.objecter _session_command_op_remove 5 2 2024-04-19T01:05:34.266 INFO:teuthology.orchestra.run.smithi043.stdout:{ 2024-04-19T01:05:34.267 INFO:teuthology.orchestra.run.smithi043.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-19T01:05:34.267 INFO:teuthology.orchestra.run.smithi043.stdout:} 2024-04-19T01:05:34.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 librados: watch_flush enter 2024-04-19T01:05:34.268 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 librados: watch_flush exit 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 client.144731.objecter close_session for osd.5 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7ff4240405a0 msgr2=0x7ff424042a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6802/760479844,v1:172.21.15.161:6805/760479844] conn(0x7ff4240405a0 0x7ff424042a40 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter put_session s=0x7ff42403fed0 osd=5 1 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 client.144731.objecter shutdown clearing up homeless session... 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 client.144731.objecter successfully canceled tick 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff424038e10 msgr2=0x7ff42403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7ff424038e10 0x7ff42403b2d0 secure :-1 s=READY pgs=3686 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0af000 tx=0x7ff43c008040 comp rx=0 tx=0).stop 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 10 monclient: shutdown 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.269 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440136380 msgr2=0x7ff44009c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 --2- 172.21.15.43:0/2826189779 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7ff440136380 0x7ff44009c3a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c420 tx=0x7ff43000c8e0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff446a66640 1 -- 172.21.15.43:0/2826189779 reap_dead start 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 shutdown_connections 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 >> 172.21.15.43:0/2826189779 conn(0x7ff44006bfe0 msgr2=0x7ff440137fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 shutdown_connections 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.254+0000 7ff4484c9640 1 -- 172.21.15.43:0/2826189779 wait complete. 2024-04-19T01:05:34.270 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.258+0000 7ff4484c9640 1 librados: shutdown 2024-04-19T01:05:34.271 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.382+0000 7f5dd160f640 10 monclient: get_monmap_and_config 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.382+0000 7f5dd160f640 10 monclient: build_initial_monmap 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.382+0000 7f5dd160f640 10 monclient: monmap: 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:34.385254+0000 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:34.385254+0000 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.385 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.382+0000 7f5dd160f640 1 Processor -- start 2024-04-19T01:05:34.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.382+0000 7f5dd160f640 1 -- start start 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.382+0000 7f5dd160f640 10 monclient: init 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient: _add_conns ranks=[9,5,7,0,6,2,1,10,4,3,8] 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5dcc138a50 0x7f5dcc138e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): picked mon.noname-a con 0x7f5dcc138a50 addr v2:172.21.15.43:3300/0 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc139370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): picked mon.noname-h con 0x7f5dcc0a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5dcc137960 0x7f5dcc137de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): picked mon.noname-i con 0x7f5dcc137960 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7f5dcc0a1020 con 0x7f5dcc138a50 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5dcc1398b0 con 0x7f5dcc0a5bb0 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f5dcc0a0780 con 0x7f5dcc137960 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient(hunting): authenticate will time out at 2204.512246s 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5dcc138a50 0x7f5dcc138e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7f5dcc138a50 0x7f5dcc138e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3300/0 says I am v2:172.21.15.43:51882/0 (socket says 172.21.15.43:51882) 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 -- 172.21.15.43:0/3848641641 learned_addr learned my addr 172.21.15.43:0/3848641641 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): get_auth_request con 0x7f5dcc138a50 auth_method 0 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.388 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcb7fe640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5dcc137960 0x7f5dcc137de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dca7fc640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc139370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient(hunting): handle_auth_done global_id 184311 payload 274 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 -- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5dcc137960 msgr2=0x7f5dcc137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5dcc137960 0x7f5dcc137de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 -- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 msgr2=0x7f5dcc139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc139370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient: _finish_hunting 0 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 monclient: found mon.noname-a 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.43:3300/0 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 -- 172.21.15.43:0/3848641641 --> v2:172.21.15.43:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc0a0d40 con 0x7f5dcc138a50 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcb7fe640 10 monclient: get_auth_request con 0x7f5dcc137960 auth_method 0 2024-04-19T01:05:34.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dca7fc640 10 monclient: get_auth_request con 0x7f5dcc0a5bb0 auth_method 0 2024-04-19T01:05:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcb7fe640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5dcc137960 0x7f5dcc137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:34.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dca7fc640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:34.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dcaffd640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc138a50 0x7f5dcc138e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009870 tx=0x7f5dbc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e9baa0c811b029 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 1 -- 172.21.15.43:0/3848641641 <== mon.0 v2:172.21.15.43:3300/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dbc0b0070 con 0x7f5dcc138a50 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: dump: 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.398 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: _finish_auth 0 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: _check_auth_tickets 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 5 monclient: authenticate success, global_id 184311 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 1 -- 172.21.15.43:0/3848641641 <== mon.0 v2:172.21.15.43:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc0aa3a0 con 0x7f5dcc138a50 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:34.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 1 -- 172.21.15.43:0/3848641641 <== mon.0 v2:172.21.15.43:3300/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dbc0aaa40 con 0x7f5dcc138a50 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dc9ffb640 10 monclient: dump: 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.400 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 4 set_mon_vals no callback set 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 10 monclient: shutdown 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 -- 172.21.15.43:0/3848641641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc138a50 msgr2=0x7f5dcc138e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.386+0000 7f5dd160f640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc138a50 0x7f5dcc138e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009870 tx=0x7f5dbc01c3f0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/3848641641 shutdown_connections 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f5dcc137960 0x7f5dcc137de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc139370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 --2- 172.21.15.43:0/3848641641 >> [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] conn(0x7f5dcc138a50 0x7f5dcc138e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/3848641641 >> 172.21.15.43:0/3848641641 conn(0x7f5dcc0a7cb0 msgr2=0x7f5dcc0a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/3848641641 shutdown_connections 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/3848641641 wait complete. 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: build_initial_monmap 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: monmap: 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:34.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 librados: starting msgr at 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 librados: starting objecter 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 Processor -- start 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- start start 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 librados: setting wanted keys 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 librados: calling monclient init 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: init 2024-04-19T01:05:34.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _add_conns ranks=[5,6,4,8,2,0,7,3,1] 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc09dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): picked mon.noname-h con 0x7f5dcc0a5bb0 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 0x7f5dcc09e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): picked mon.noname-d con 0x7f5dcc137960 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 0x7f5dcc1355e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): picked mon.noname-c con 0x7f5dcc138a50 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f5dcc0a2100 con 0x7f5dcc138a50 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f5dcc0a1f80 con 0x7f5dcc0a5bb0 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f5dcc142810 con 0x7f5dcc137960 2024-04-19T01:05:34.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient(hunting): authenticate will time out at 2204.517652s 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 0x7f5dcc09e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 0x7f5dcc1355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 0x7f5dcc09e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3303/0 says I am v2:172.21.15.43:39726/0 (socket says 172.21.15.43:39726) 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 1 -- 172.21.15.43:0/2445135863 learned_addr learned my addr 172.21.15.43:0/2445135863 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): get_auth_request con 0x7f5dcc137960 auth_method 0 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 0x7f5dcc1355e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36712/0 (socket says 172.21.15.43:36712) 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): get_auth_request con 0x7f5dcc138a50 auth_method 0 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc09dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 10 monclient(hunting): get_auth_request con 0x7f5dcc0a5bb0 auth_method 0 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient(hunting): handle_auth_done global_id 184261 payload 824 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 -- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 msgr2=0x7f5dcc09e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 0x7f5dcc09e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 -- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 msgr2=0x7f5dcc09dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc09dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient: _finish_hunting 0 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 monclient: found mon.noname-c 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 -- 172.21.15.43:0/2445135863 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dcc09fc90 con 0x7f5dcc138a50 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dca7fc640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 0x7f5dcc09e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcb7fe640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 0x7f5dcc1355e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400cdf0 tx=0x7f5dc400a710 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 client.?.objecter ms_handle_connect 0x7f5dcc138a50 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dc401a070 con 0x7f5dcc138a50 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:34.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: dump: 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.408 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: _finish_auth 0 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: _check_auth_tickets 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 5 monclient: authenticate success, global_id 184261 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _renew_subs 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc400ae20 con 0x7f5dcc138a50 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dcc09ee60 con 0x7f5dcc138a50 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f5dc4013340 con 0x7f5dcc138a50 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.409 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dc97fa640 4 set_mon_vals no callback set 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 client.184261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 monclient: dump: 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.410 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _renew_subs 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dcc09c6f0 con 0x7f5dcc138a50 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 librados: init done 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f5dc400d640 con 0x7f5dcc138a50 2024-04-19T01:05:34.411 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dcc137020 con 0x7f5dcc138a50 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5db8038e10 0x7f5db803b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f5dc40497b0 con 0x7f5dcc138a50 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 10 client.184261.objecter ms_dispatch 0x7f5dcc000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 3 client.184261.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 3 client.184261.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5db37fe640 20 client.184261.objecter dump_active .. 0 homeless 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5db8038e10 0x7f5db803b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.390+0000 7f5dcaffd640 10 monclient: get_auth_request con 0x7f5db8038e10 auth_method 0 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.394+0000 7f5dcaffd640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5db8038e10 0x7f5db803b2d0 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0ad000 tx=0x7f5dbc0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.412 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.394+0000 7f5db37fe640 10 client.184261.objecter ms_handle_connect 0x7f5db8038e10 2024-04-19T01:05:34.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.398+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5dcc09fc90 con 0x7f5dcc138a50 2024-04-19T01:05:34.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.398+0000 7f5db37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.413 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.398+0000 7f5db37fe640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:34.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.558+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f5dcc0a5290 con 0x7f5db8038e10 2024-04-19T01:05:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.558+0000 7f5db37fe640 1 -- 172.21.15.43:0/2445135863 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5dcc0a5290 con 0x7f5db8038e10 2024-04-19T01:05:34.558 INFO:teuthology.orchestra.run.smithi043.stderr:instructed osd(s) 5 to deep-scrub 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 10 librados: watch_flush enter 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 10 librados: watch_flush exit 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 20 client.184261.objecter shutdown clearing up homeless session... 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 10 client.184261.objecter successfully canceled tick 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5db8038e10 msgr2=0x7f5db803b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5db8038e10 0x7f5db803b2d0 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0ad000 tx=0x7f5dbc0a6000 comp rx=0 tx=0).stop 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 10 monclient: shutdown 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 msgr2=0x7f5dcc1355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 0x7f5dcc1355e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400cdf0 tx=0x7f5dc400a710 comp rx=0 tx=0).stop 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 shutdown_connections 2024-04-19T01:05:34.561 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f5db8038e10 0x7f5db803b2d0 unknown :-1 s=CLOSED pgs=3687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f5dcc138a50 0x7f5dcc1355e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f5dcc137960 0x7f5dcc09e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 --2- 172.21.15.43:0/2445135863 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f5dcc0a5bb0 0x7f5dcc09dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 >> 172.21.15.43:0/2445135863 conn(0x7f5dcc0a7cb0 msgr2=0x7f5dcc0a60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 shutdown_connections 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 -- 172.21.15.43:0/2445135863 wait complete. 2024-04-19T01:05:34.562 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.562+0000 7f5dd160f640 1 librados: shutdown 2024-04-19T01:05:34.576 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient: get_monmap_and_config 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient: build_initial_monmap 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient: monmap: 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:34.704284+0000 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:34.704284+0000 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:34.705 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 Processor -- start 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 -- start start 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient: init 2024-04-19T01:05:34.706 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient: _add_conns ranks=[8,7,0,3,10,1,6,2,9,4,5] 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f36a0138cd0 0x7f36a01390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): picked mon.noname-e con 0x7f36a0138cd0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 --2- >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f36a00a5b80 0x7f36a01395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): picked mon.noname-i con 0x7f36a00a5b80 addr [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36a01358b0 addr v1:172.21.15.43:6789/0 2024-04-19T01:05:34.707 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 -- --> v1:172.21.15.43:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36a00a3dc0 con 0x7f36a01358b0 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 -- --> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] -- mon_getmap magic: 0 -- 0x7f36a00a4410 con 0x7f36a00a5b80 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f36a0139b30 con 0x7f36a0138cd0 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a7ac9640 10 monclient(hunting): authenticate will time out at 2204.831418s 2024-04-19T01:05:34.710 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a6066640 1 --1- >> v1:172.21.15.43:6789/0 conn(0x7f36a01358b0 0x7f36a0135d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.43:6789/0 says I am v1:172.21.15.43:57544/0 (socket says 172.21.15.43:57544) 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a6066640 1 -- 172.21.15.43:0/1904708666 learned_addr learned my addr 172.21.15.43:0/1904708666 (peer_addr_for_me v1:172.21.15.43:0/0) 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f36a0138cd0 0x7f36a01390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): get_auth_request con 0x7f36a0138cd0 auth_method 0 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5865640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5064640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f36a00a5b80 0x7f36a01395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5064640 10 monclient(hunting): get_auth_request con 0x7f36a00a5b80 auth_method 0 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5064640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.711 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a5064640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a4863640 1 -- 172.21.15.43:0/1904708666 <== mon.0 v1:172.21.15.43:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2883794735 0 0) 0x7f36a00a3dc0 con 0x7f36a01358b0 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a4863640 10 monclient(hunting): _negotiate 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a4863640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a4863640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.702+0000 7f36a4863640 10 monclient(hunting): my global_id is 184320 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 1 -- 172.21.15.43:0/1904708666 --> v1:172.21.15.43:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f368c003530 con 0x7f36a01358b0 2024-04-19T01:05:34.712 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 10 monclient(hunting): handle_auth_done global_id 144791 payload 274 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 1 -- 172.21.15.43:0/1904708666 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f36a00a5b80 msgr2=0x7f36a01395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f36a00a5b80 0x7f36a01395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 1 -- 172.21.15.43:0/1904708666 >> v1:172.21.15.43:6789/0 conn(0x7f36a01358b0 legacy=0x7f36a0135d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 10 monclient: _finish_hunting 0 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 1 monclient: found mon.noname-e 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:34.713 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 1 -- 172.21.15.43:0/1904708666 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a00a40a0 con 0x7f36a0138cd0 2024-04-19T01:05:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a5865640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f36a0138cd0 0x7f36a01390b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f369c0097f0 tx=0x7f369c01c340 comp rx=0 tx=0).ready entity=mon.8 client_cookie=77733215f9ef7f21 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 1 -- 172.21.15.43:0/1904708666 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f369c0b0070 con 0x7f36a0138cd0 2024-04-19T01:05:34.720 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: dump: 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.721 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: _finish_auth 0 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: _check_auth_tickets 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 5 monclient: authenticate success, global_id 144791 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 1 -- 172.21.15.43:0/1904708666 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369c0aa810 con 0x7f36a0138cd0 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 1 -- 172.21.15.43:0/1904708666 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f369c0a9300 con 0x7f36a0138cd0 2024-04-19T01:05:34.722 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 4 set_mon_vals no callback set 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a4863640 10 monclient: dump: 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.723 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 10 monclient: shutdown 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 -- 172.21.15.43:0/1904708666 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f36a0138cd0 msgr2=0x7f36a01390b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.724 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f36a0138cd0 0x7f36a01390b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f369c0097f0 tx=0x7f369c01c340 comp rx=0 tx=0).stop 2024-04-19T01:05:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 -- 172.21.15.43:0/1904708666 shutdown_connections 2024-04-19T01:05:34.725 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] conn(0x7f36a00a5b80 0x7f36a01395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/1904708666 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f36a0138cd0 0x7f36a01390b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 -- 172.21.15.43:0/1904708666 >> 172.21.15.43:0/1904708666 conn(0x7f36a00ab2f0 msgr2=0x7f36a00ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 -- 172.21.15.43:0/1904708666 shutdown_connections 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.706+0000 7f36a7ac9640 1 -- 172.21.15.43:0/1904708666 wait complete. 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: build_initial_monmap 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: monmap: 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:34.726 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.727 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 librados: starting msgr at 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 librados: starting objecter 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 Processor -- start 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- start start 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 librados: setting wanted keys 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 librados: calling monclient init 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: init 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _add_conns ranks=[5,4,6,0,3,1,8,2,7] 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f36a00a5b80 0x7f36a00a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): picked mon.noname-h con 0x7f36a00a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 0x7f36a00a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): picked mon.noname-c con 0x7f36a01358b0 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f36a0138cd0 0x7f36a01326e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): picked mon.noname-d con 0x7f36a0138cd0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.728 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f36a0143ab0 con 0x7f36a01358b0 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7f36a00a20a0 con 0x7f36a00a5b80 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7f36a0143c30 con 0x7f36a0138cd0 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient(hunting): authenticate will time out at 2204.837364s 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 0x7f36a00a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 0x7f36a00a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:36720/0 (socket says 172.21.15.43:36720) 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 -- 172.21.15.43:0/3708639842 learned_addr learned my addr 172.21.15.43:0/3708639842 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): get_auth_request con 0x7f36a01358b0 auth_method 0 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f36a0138cd0 0x7f36a01326e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.729 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): get_auth_request con 0x7f36a0138cd0 auth_method 0 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.730 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:34.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:34.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a6066640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:34.734 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient(hunting): handle_auth_done global_id 184270 payload 824 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5865640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f36a00a5b80 0x7f36a00a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 -- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f36a0138cd0 msgr2=0x7f36a01326e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f36a0138cd0 0x7f36a01326e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 -- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f36a00a5b80 msgr2=0x7f36a00a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f36a00a5b80 0x7f36a00a0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient: _finish_hunting 0 2024-04-19T01:05:34.735 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 monclient: found mon.noname-c 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 -- 172.21.15.43:0/3708639842 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36a009c920 con 0x7f36a01358b0 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5064640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 0x7f36a00a1070 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f369000d7d0 tx=0x7f369000dc90 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 client.?.objecter ms_handle_connect 0x7f36a01358b0 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f36900044a0 con 0x7f36a01358b0 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: dump: 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.736 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: _finish_auth 0 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: _check_auth_tickets 2024-04-19T01:05:34.737 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369000b6c0 con 0x7f36a01358b0 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f369000bce0 con 0x7f36a01358b0 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 5 monclient: authenticate success, global_id 184270 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _renew_subs 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36a009c5d0 con 0x7f36a01358b0 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f3697fff640 4 set_mon_vals no callback set 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:34.738 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 monclient: dump: 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:34.740 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 client.184270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _renew_subs 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f36a009c010 con 0x7f36a01358b0 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 librados: init done 2024-04-19T01:05:34.741 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f369000b870 con 0x7f36a01358b0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36a00693b0 con 0x7f36a01358b0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3684038e40 0x7f368403b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f369000f040 con 0x7f36a01358b0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 10 client.184270.objecter ms_dispatch 0x7f36a0000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 3 client.184270.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 3 client.184270.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36967fc640 20 client.184270.objecter dump_active .. 0 homeless 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5865640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3684038e40 0x7f368403b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.710+0000 7f36a5865640 10 monclient: get_auth_request con 0x7f3684038e40 auth_method 0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.714+0000 7f36a5865640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3684038e40 0x7f368403b300 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7f369c0ad000 tx=0x7f369c0a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:34.742 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.714+0000 7f36967fc640 10 client.184270.objecter ms_handle_connect 0x7f3684038e40 2024-04-19T01:05:34.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.718+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f36900042c0 con 0x7f36a01358b0 2024-04-19T01:05:34.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.718+0000 7f36967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:34.743 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.718+0000 7f36967fc640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:34.873 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.874+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f36a01d1af0 con 0x7f3684038e40 2024-04-19T01:05:34.882 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.882+0000 7f36967fc640 1 -- 172.21.15.43:0/3708639842 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113606 (secure 0 0 0) 0x7f36a01d1af0 con 0x7f3684038e40 2024-04-19T01:05:34.883 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T01:05:34.883 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 10 librados: watch_flush enter 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 10 librados: watch_flush exit 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 20 client.184270.objecter shutdown clearing up homeless session... 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 10 client.184270.objecter successfully canceled tick 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3684038e40 msgr2=0x7f368403b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3684038e40 0x7f368403b300 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7f369c0ad000 tx=0x7f369c0a6000 comp rx=0 tx=0).stop 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 10 monclient: shutdown 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 msgr2=0x7f36a00a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 0x7f36a00a1070 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f369000d7d0 tx=0x7f369000dc90 comp rx=0 tx=0).stop 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 shutdown_connections 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f3684038e40 0x7f368403b300 unknown :-1 s=CLOSED pgs=3688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.885 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7f36a0138cd0 0x7f36a01326e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f36a01358b0 0x7f36a00a1070 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 --2- 172.21.15.43:0/3708639842 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7f36a00a5b80 0x7f36a00a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 >> 172.21.15.43:0/3708639842 conn(0x7f36a00ab2f0 msgr2=0x7f36a00a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 shutdown_connections 2024-04-19T01:05:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 -- 172.21.15.43:0/3708639842 wait complete. 2024-04-19T01:05:34.886 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:34.886+0000 7f36a7ac9640 1 librados: shutdown 2024-04-19T01:05:34.899 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1330,"stamp":"2024-04-19T01:05:33.333701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124,"ondisk_log_size":124,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":448576,"kb_used_data":24112,"kb_used_omap":9,"kb_used_meta":424374,"kb_avail":565782464,"statfs":{"total":579820584960,"available":579361243136,"internally_reserved":0,"allocated":24690688,"data_stored":10993320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":434559649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004111"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683169+0000","last_change":"2024-04-19T00:45:27.797265+0000","last_active":"2024-04-19T01:03:40.683169+0000","last_peered":"2024-04-19T01:03:40.683169+0000","last_clean":"2024-04-19T01:03:40.683169+0000","last_became_active":"2024-04-19T00:45:27.796802+0000","last_became_peered":"2024-04-19T00:45:27.796802+0000","last_unstale":"2024-04-19T01:03:40.683169+0000","last_undegraded":"2024-04-19T01:03:40.683169+0000","last_fullsized":"2024-04-19T01:03:40.683169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:39:20.387401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777710+0000","last_change":"2024-04-19T00:45:27.717179+0000","last_active":"2024-04-19T01:03:41.777710+0000","last_peered":"2024-04-19T01:03:41.777710+0000","last_clean":"2024-04-19T01:03:41.777710+0000","last_became_active":"2024-04-19T00:45:27.715591+0000","last_became_peered":"2024-04-19T00:45:27.715591+0000","last_unstale":"2024-04-19T01:03:41.777710+0000","last_undegraded":"2024-04-19T01:03:41.777710+0000","last_fullsized":"2024-04-19T01:03:41.777710+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:12:05.981056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1367,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:27.189222+0000","last_change":"2024-04-19T00:45:27.745246+0000","last_active":"2024-04-19T01:05:27.189222+0000","last_peered":"2024-04-19T01:05:27.189222+0000","last_clean":"2024-04-19T01:05:27.189222+0000","last_became_active":"2024-04-19T00:45:27.744986+0000","last_became_peered":"2024-04-19T00:45:27.744986+0000","last_unstale":"2024-04-19T01:05:27.189222+0000","last_undegraded":"2024-04-19T01:05:27.189222+0000","last_fullsized":"2024-04-19T01:05:27.189222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683470+0000","last_change":"2024-04-19T00:45:27.767266+0000","last_active":"2024-04-19T01:03:40.683470+0000","last_peered":"2024-04-19T01:03:40.683470+0000","last_clean":"2024-04-19T01:03:40.683470+0000","last_became_active":"2024-04-19T00:45:27.767040+0000","last_became_peered":"2024-04-19T00:45:27.767040+0000","last_unstale":"2024-04-19T01:03:40.683470+0000","last_undegraded":"2024-04-19T01:03:40.683470+0000","last_fullsized":"2024-04-19T01:03:40.683470+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:43:32.043488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777937+0000","last_change":"2024-04-19T00:45:27.717564+0000","last_active":"2024-04-19T01:03:41.777937+0000","last_peered":"2024-04-19T01:03:41.777937+0000","last_clean":"2024-04-19T01:03:41.777937+0000","last_became_active":"2024-04-19T00:45:27.717286+0000","last_became_peered":"2024-04-19T00:45:27.717286+0000","last_unstale":"2024-04-19T01:03:41.777937+0000","last_undegraded":"2024-04-19T01:03:41.777937+0000","last_fullsized":"2024-04-19T01:03:41.777937+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:43:17.618945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683854+0000","last_change":"2024-04-19T00:45:27.799104+0000","last_active":"2024-04-19T01:03:40.683854+0000","last_peered":"2024-04-19T01:03:40.683854+0000","last_clean":"2024-04-19T01:03:40.683854+0000","last_became_active":"2024-04-19T00:45:27.795414+0000","last_became_peered":"2024-04-19T00:45:27.795414+0000","last_unstale":"2024-04-19T01:03:40.683854+0000","last_undegraded":"2024-04-19T01:03:40.683854+0000","last_fullsized":"2024-04-19T01:03:40.683854+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:42:29.849497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1367,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:04:57.185198+0000","last_change":"2024-04-19T00:45:27.795299+0000","last_active":"2024-04-19T01:04:57.185198+0000","last_peered":"2024-04-19T01:04:57.185198+0000","last_clean":"2024-04-19T01:04:57.185198+0000","last_became_active":"2024-04-19T00:45:27.790389+0000","last_became_peered":"2024-04-19T00:45:27.790389+0000","last_unstale":"2024-04-19T01:04:57.185198+0000","last_undegraded":"2024-04-19T01:04:57.185198+0000","last_fullsized":"2024-04-19T01:04:57.185198+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403468+0000","last_change":"2024-04-19T00:45:27.738033+0000","last_active":"2024-04-19T01:03:42.403468+0000","last_peered":"2024-04-19T01:03:42.403468+0000","last_clean":"2024-04-19T01:03:42.403468+0000","last_became_active":"2024-04-19T00:45:27.735932+0000","last_became_peered":"2024-04-19T00:45:27.735932+0000","last_unstale":"2024-04-19T01:03:42.403468+0000","last_undegraded":"2024-04-19T01:03:42.403468+0000","last_fullsized":"2024-04-19T01:03:42.403468+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:35:41.884097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1364,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403517+0000","last_change":"2024-04-19T00:45:27.765213+0000","last_active":"2024-04-19T01:03:42.403517+0000","last_peered":"2024-04-19T01:03:42.403517+0000","last_clean":"2024-04-19T01:03:42.403517+0000","last_became_active":"2024-04-19T00:45:27.752978+0000","last_became_peered":"2024-04-19T00:45:27.752978+0000","last_unstale":"2024-04-19T01:03:42.403517+0000","last_undegraded":"2024-04-19T01:03:42.403517+0000","last_fullsized":"2024-04-19T01:03:42.403517+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:55:11.690354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"17'7","reported_seq":1376,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403206+0000","last_change":"2024-04-19T00:45:27.805132+0000","last_active":"2024-04-19T01:03:42.403206+0000","last_peered":"2024-04-19T01:03:42.403206+0000","last_clean":"2024-04-19T01:03:42.403206+0000","last_became_active":"2024-04-19T00:45:27.798161+0000","last_became_peered":"2024-04-19T00:45:27.798161+0000","last_unstale":"2024-04-19T01:03:42.403206+0000","last_undegraded":"2024-04-19T01:03:42.403206+0000","last_fullsized":"2024-04-19T01:03:42.403206+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:51:44.398144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683194+0000","last_change":"2024-04-19T00:45:27.799286+0000","last_active":"2024-04-19T01:03:40.683194+0000","last_peered":"2024-04-19T01:03:40.683194+0000","last_clean":"2024-04-19T01:03:40.683194+0000","last_became_active":"2024-04-19T00:45:27.797033+0000","last_became_peered":"2024-04-19T00:45:27.797033+0000","last_unstale":"2024-04-19T01:03:40.683194+0000","last_undegraded":"2024-04-19T01:03:40.683194+0000","last_fullsized":"2024-04-19T01:03:40.683194+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:04:40.412420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403776+0000","last_change":"2024-04-19T00:45:27.760256+0000","last_active":"2024-04-19T01:03:42.403776+0000","last_peered":"2024-04-19T01:03:42.403776+0000","last_clean":"2024-04-19T01:03:42.403776+0000","last_became_active":"2024-04-19T00:45:27.757597+0000","last_became_peered":"2024-04-19T00:45:27.757597+0000","last_unstale":"2024-04-19T01:03:42.403776+0000","last_undegraded":"2024-04-19T01:03:42.403776+0000","last_fullsized":"2024-04-19T01:03:42.403776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:41:10.030457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978226+0000","last_change":"2024-04-19T00:44:34.474693+0000","last_active":"2024-04-19T01:03:41.978226+0000","last_peered":"2024-04-19T01:03:41.978226+0000","last_clean":"2024-04-19T01:03:41.978226+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T01:03:41.978226+0000","last_undegraded":"2024-04-19T01:03:41.978226+0000","last_fullsized":"2024-04-19T01:03:41.978226+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:18:46.046655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777648+0000","last_change":"2024-04-19T00:45:27.742732+0000","last_active":"2024-04-19T01:03:41.777648+0000","last_peered":"2024-04-19T01:03:41.777648+0000","last_clean":"2024-04-19T01:03:41.777648+0000","last_became_active":"2024-04-19T00:45:27.740409+0000","last_became_peered":"2024-04-19T00:45:27.740409+0000","last_unstale":"2024-04-19T01:03:41.777648+0000","last_undegraded":"2024-04-19T01:03:41.777648+0000","last_fullsized":"2024-04-19T01:03:41.777648+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:13:15.654443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1387,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.778149+0000","last_change":"2024-04-19T00:44:34.473317+0000","last_active":"2024-04-19T01:03:41.778149+0000","last_peered":"2024-04-19T01:03:41.778149+0000","last_clean":"2024-04-19T01:03:41.778149+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T01:03:41.778149+0000","last_undegraded":"2024-04-19T01:03:41.778149+0000","last_fullsized":"2024-04-19T01:03:41.778149+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:03:29.112656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777602+0000","last_change":"2024-04-19T00:45:27.679388+0000","last_active":"2024-04-19T01:03:41.777602+0000","last_peered":"2024-04-19T01:03:41.777602+0000","last_clean":"2024-04-19T01:03:41.777602+0000","last_became_active":"2024-04-19T00:45:27.677388+0000","last_became_peered":"2024-04-19T00:45:27.677388+0000","last_unstale":"2024-04-19T01:03:41.777602+0000","last_undegraded":"2024-04-19T01:03:41.777602+0000","last_fullsized":"2024-04-19T01:03:41.777602+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:57:14.716992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683573+0000","last_change":"2024-04-19T00:44:34.474353+0000","last_active":"2024-04-19T01:03:40.683573+0000","last_peered":"2024-04-19T01:03:40.683573+0000","last_clean":"2024-04-19T01:03:40.683573+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T01:03:40.683573+0000","last_undegraded":"2024-04-19T01:03:40.683573+0000","last_fullsized":"2024-04-19T01:03:40.683573+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T00:48:30.196169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1367,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404413+0000","last_change":"2024-04-19T00:45:27.785476+0000","last_active":"2024-04-19T01:03:42.404413+0000","last_peered":"2024-04-19T01:03:42.404413+0000","last_clean":"2024-04-19T01:03:42.404413+0000","last_became_active":"2024-04-19T00:45:27.785226+0000","last_became_peered":"2024-04-19T00:45:27.785226+0000","last_unstale":"2024-04-19T01:03:42.404413+0000","last_undegraded":"2024-04-19T01:03:42.404413+0000","last_fullsized":"2024-04-19T01:03:42.404413+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403439+0000","last_change":"2024-04-19T00:44:34.070310+0000","last_active":"2024-04-19T01:03:42.403439+0000","last_peered":"2024-04-19T01:03:42.403439+0000","last_clean":"2024-04-19T01:03:42.403439+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T01:03:42.403439+0000","last_undegraded":"2024-04-19T01:03:42.403439+0000","last_fullsized":"2024-04-19T01:03:42.403439+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:12:38.878926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'117","reported_seq":1537,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977832+0000","last_change":"2024-04-19T00:44:34.073067+0000","last_active":"2024-04-19T01:03:41.977832+0000","last_peered":"2024-04-19T01:03:41.977832+0000","last_clean":"2024-04-19T01:03:41.977832+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T01:03:41.977832+0000","last_undegraded":"2024-04-19T01:03:41.977832+0000","last_fullsized":"2024-04-19T01:03:41.977832+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:02:45.312062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403239+0000","last_change":"2024-04-19T00:45:27.821135+0000","last_active":"2024-04-19T01:03:42.403239+0000","last_peered":"2024-04-19T01:03:42.403239+0000","last_clean":"2024-04-19T01:03:42.403239+0000","last_became_active":"2024-04-19T00:45:27.813538+0000","last_became_peered":"2024-04-19T00:45:27.813538+0000","last_unstale":"2024-04-19T01:03:42.403239+0000","last_undegraded":"2024-04-19T01:03:42.403239+0000","last_fullsized":"2024-04-19T01:03:42.403239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:20:22.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1370,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403483+0000","last_change":"2024-04-19T00:44:34.071797+0000","last_active":"2024-04-19T01:03:42.403483+0000","last_peered":"2024-04-19T01:03:42.403483+0000","last_clean":"2024-04-19T01:03:42.403483+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T01:03:42.403483+0000","last_undegraded":"2024-04-19T01:03:42.403483+0000","last_fullsized":"2024-04-19T01:03:42.403483+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:46:25.146282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777775+0000","last_change":"2024-04-19T00:45:27.763529+0000","last_active":"2024-04-19T01:03:41.777775+0000","last_peered":"2024-04-19T01:03:41.777775+0000","last_clean":"2024-04-19T01:03:41.777775+0000","last_became_active":"2024-04-19T00:45:27.763230+0000","last_became_peered":"2024-04-19T00:45:27.763230+0000","last_unstale":"2024-04-19T01:03:41.777775+0000","last_undegraded":"2024-04-19T01:03:41.777775+0000","last_fullsized":"2024-04-19T01:03:41.777775+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:22:57.898965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1380,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683667+0000","last_change":"2024-04-19T00:44:34.064735+0000","last_active":"2024-04-19T01:03:40.683667+0000","last_peered":"2024-04-19T01:03:40.683667+0000","last_clean":"2024-04-19T01:03:40.683667+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T01:03:40.683667+0000","last_undegraded":"2024-04-19T01:03:40.683667+0000","last_fullsized":"2024-04-19T01:03:40.683667+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:04:03.225536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1367,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404404+0000","last_change":"2024-04-19T00:45:27.647807+0000","last_active":"2024-04-19T01:03:42.404404+0000","last_peered":"2024-04-19T01:03:42.404404+0000","last_clean":"2024-04-19T01:03:42.404404+0000","last_became_active":"2024-04-19T00:45:27.647106+0000","last_became_peered":"2024-04-19T00:45:27.647106+0000","last_unstale":"2024-04-19T01:03:42.404404+0000","last_undegraded":"2024-04-19T01:03:42.404404+0000","last_fullsized":"2024-04-19T01:03:42.404404+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978360+0000","last_change":"2024-04-19T00:44:34.473535+0000","last_active":"2024-04-19T01:03:41.978360+0000","last_peered":"2024-04-19T01:03:41.978360+0000","last_clean":"2024-04-19T01:03:41.978360+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T01:03:41.978360+0000","last_undegraded":"2024-04-19T01:03:41.978360+0000","last_fullsized":"2024-04-19T01:03:41.978360+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:36:04.048348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404860+0000","last_change":"2024-04-19T00:45:27.659480+0000","last_active":"2024-04-19T01:03:42.404860+0000","last_peered":"2024-04-19T01:03:42.404860+0000","last_clean":"2024-04-19T01:03:42.404860+0000","last_became_active":"2024-04-19T00:45:27.659197+0000","last_became_peered":"2024-04-19T00:45:27.659197+0000","last_unstale":"2024-04-19T01:03:42.404860+0000","last_undegraded":"2024-04-19T01:03:42.404860+0000","last_fullsized":"2024-04-19T01:03:42.404860+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777965+0000","last_change":"2024-04-19T00:44:34.066542+0000","last_active":"2024-04-19T01:03:41.777965+0000","last_peered":"2024-04-19T01:03:41.777965+0000","last_clean":"2024-04-19T01:03:41.777965+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T01:03:41.777965+0000","last_undegraded":"2024-04-19T01:03:41.777965+0000","last_fullsized":"2024-04-19T01:03:41.777965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:44:33.030587+0000","last_clean_scrub_stamp":"2024-04-19T00:44:33.030587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:12:09.032801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1366,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.404568+0000","last_change":"2024-04-19T00:45:27.661883+0000","last_active":"2024-04-19T01:03:42.404568+0000","last_peered":"2024-04-19T01:03:42.404568+0000","last_clean":"2024-04-19T01:03:42.404568+0000","last_became_active":"2024-04-19T00:45:27.661535+0000","last_became_peered":"2024-04-19T00:45:27.661535+0000","last_unstale":"2024-04-19T01:03:42.404568+0000","last_undegraded":"2024-04-19T01:03:42.404568+0000","last_fullsized":"2024-04-19T01:03:42.404568+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1367,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:04:12.178564+0000","last_change":"2024-04-19T00:45:27.722242+0000","last_active":"2024-04-19T01:04:12.178564+0000","last_peered":"2024-04-19T01:04:12.178564+0000","last_clean":"2024-04-19T01:04:12.178564+0000","last_became_active":"2024-04-19T00:45:27.719717+0000","last_became_peered":"2024-04-19T00:45:27.719717+0000","last_unstale":"2024-04-19T01:04:12.178564+0000","last_undegraded":"2024-04-19T01:04:12.178564+0000","last_fullsized":"2024-04-19T01:04:12.178564+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1363,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403826+0000","last_change":"2024-04-19T00:45:27.725269+0000","last_active":"2024-04-19T01:03:42.403826+0000","last_peered":"2024-04-19T01:03:42.403826+0000","last_clean":"2024-04-19T01:03:42.403826+0000","last_became_active":"2024-04-19T00:45:27.719793+0000","last_became_peered":"2024-04-19T00:45:27.719793+0000","last_unstale":"2024-04-19T01:03:42.403826+0000","last_undegraded":"2024-04-19T01:03:42.403826+0000","last_fullsized":"2024-04-19T01:03:42.403826+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:05:34.645736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1367,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:27.189247+0000","last_change":"2024-04-19T00:45:27.770635+0000","last_active":"2024-04-19T01:05:27.189247+0000","last_peered":"2024-04-19T01:05:27.189247+0000","last_clean":"2024-04-19T01:05:27.189247+0000","last_became_active":"2024-04-19T00:45:27.769635+0000","last_became_peered":"2024-04-19T00:45:27.769635+0000","last_unstale":"2024-04-19T01:05:27.189247+0000","last_undegraded":"2024-04-19T01:05:27.189247+0000","last_fullsized":"2024-04-19T01:05:27.189247+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.777809+0000","last_change":"2024-04-19T00:45:27.752249+0000","last_active":"2024-04-19T01:03:41.777809+0000","last_peered":"2024-04-19T01:03:41.777809+0000","last_clean":"2024-04-19T01:03:41.777809+0000","last_became_active":"2024-04-19T00:45:27.751895+0000","last_became_peered":"2024-04-19T00:45:27.751895+0000","last_unstale":"2024-04-19T01:03:41.777809+0000","last_undegraded":"2024-04-19T01:03:41.777809+0000","last_fullsized":"2024-04-19T01:03:41.777809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:19:10.348532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403676+0000","last_change":"2024-04-19T00:45:27.813207+0000","last_active":"2024-04-19T01:03:42.403676+0000","last_peered":"2024-04-19T01:03:42.403676+0000","last_clean":"2024-04-19T01:03:42.403676+0000","last_became_active":"2024-04-19T00:45:27.812520+0000","last_became_peered":"2024-04-19T00:45:27.812520+0000","last_unstale":"2024-04-19T01:03:42.403676+0000","last_undegraded":"2024-04-19T01:03:42.403676+0000","last_fullsized":"2024-04-19T01:03:42.403676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:21:29.786709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977870+0000","last_change":"2024-04-19T00:45:27.796325+0000","last_active":"2024-04-19T01:03:41.977870+0000","last_peered":"2024-04-19T01:03:41.977870+0000","last_clean":"2024-04-19T01:03:41.977870+0000","last_became_active":"2024-04-19T00:45:27.795698+0000","last_became_peered":"2024-04-19T00:45:27.795698+0000","last_unstale":"2024-04-19T01:03:41.977870+0000","last_undegraded":"2024-04-19T01:03:41.977870+0000","last_fullsized":"2024-04-19T01:03:41.977870+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:16:23.414716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978229+0000","last_change":"2024-04-19T00:45:27.750891+0000","last_active":"2024-04-19T01:03:41.978229+0000","last_peered":"2024-04-19T01:03:41.978229+0000","last_clean":"2024-04-19T01:03:41.978229+0000","last_became_active":"2024-04-19T00:45:27.750623+0000","last_became_peered":"2024-04-19T00:45:27.750623+0000","last_unstale":"2024-04-19T01:03:41.978229+0000","last_undegraded":"2024-04-19T01:03:41.978229+0000","last_fullsized":"2024-04-19T01:03:41.978229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:14:11.322418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.977779+0000","last_change":"2024-04-19T00:45:27.758851+0000","last_active":"2024-04-19T01:03:41.977779+0000","last_peered":"2024-04-19T01:03:41.977779+0000","last_clean":"2024-04-19T01:03:41.977779+0000","last_became_active":"2024-04-19T00:45:27.758559+0000","last_became_peered":"2024-04-19T00:45:27.758559+0000","last_unstale":"2024-04-19T01:03:41.977779+0000","last_undegraded":"2024-04-19T01:03:41.977779+0000","last_fullsized":"2024-04-19T01:03:41.977779+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:04:39.957325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978326+0000","last_change":"2024-04-19T00:45:27.701325+0000","last_active":"2024-04-19T01:03:41.978326+0000","last_peered":"2024-04-19T01:03:41.978326+0000","last_clean":"2024-04-19T01:03:41.978326+0000","last_became_active":"2024-04-19T00:45:27.697796+0000","last_became_peered":"2024-04-19T00:45:27.697796+0000","last_unstale":"2024-04-19T01:03:41.978326+0000","last_undegraded":"2024-04-19T01:03:41.978326+0000","last_fullsized":"2024-04-19T01:03:41.978326+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:38:52.173417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1362,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:42.403581+0000","last_change":"2024-04-19T00:45:27.815387+0000","last_active":"2024-04-19T01:03:42.403581+0000","last_peered":"2024-04-19T01:03:42.403581+0000","last_clean":"2024-04-19T01:03:42.403581+0000","last_became_active":"2024-04-19T00:45:27.814761+0000","last_became_peered":"2024-04-19T00:45:27.814761+0000","last_unstale":"2024-04-19T01:03:42.403581+0000","last_undegraded":"2024-04-19T01:03:42.403581+0000","last_fullsized":"2024-04-19T01:03:42.403581+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:29:10.459966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:41.978040+0000","last_change":"2024-04-19T00:45:27.781966+0000","last_active":"2024-04-19T01:03:41.978040+0000","last_peered":"2024-04-19T01:03:41.978040+0000","last_clean":"2024-04-19T01:03:41.978040+0000","last_became_active":"2024-04-19T00:45:27.779682+0000","last_became_peered":"2024-04-19T00:45:27.779682+0000","last_unstale":"2024-04-19T01:03:41.978040+0000","last_undegraded":"2024-04-19T01:03:41.978040+0000","last_fullsized":"2024-04-19T01:03:41.978040+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:45:17.983232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:03:40.683489+0000","last_change":"2024-04-19T00:45:27.790124+0000","last_active":"2024-04-19T01:03:40.683489+0000","last_peered":"2024-04-19T01:03:40.683489+0000","last_clean":"2024-04-19T01:03:40.683489+0000","last_became_active":"2024-04-19T00:45:27.788325+0000","last_became_peered":"2024-04-19T00:45:27.788325+0000","last_unstale":"2024-04-19T01:03:40.683489+0000","last_undegraded":"2024-04-19T01:03:40.683489+0000","last_fullsized":"2024-04-19T01:03:40.683489+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T00:45:26.542604+0000","last_clean_scrub_stamp":"2024-04-19T00:45:26.542604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:16:28.463405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607810,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607809,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75284,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296556,"statfs":{"total":96636764160,"available":96559673344,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607808,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75800,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296040,"statfs":{"total":96636764160,"available":96559144960,"internally_reserved":0,"allocated":4468736,"data_stored":2184837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640513,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79440,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292400,"statfs":{"total":96636764160,"available":96555417600,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640510,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71644,"kb_used_data":4368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300196,"statfs":{"total":96636764160,"available":96563400704,"internally_reserved":0,"allocated":4472832,"data_stored":2184842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640511,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71124,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300716,"statfs":{"total":96636764160,"available":96563933184,"internally_reserved":0,"allocated":3940352,"data_stored":1655914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.18 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.19 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.1a last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.1b last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.4 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.5 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.6 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.905 INFO:tasks.ceph:pgid 8.7 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 8.0 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 8.1 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 8.2 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 8.3 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 8.9 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 8.8 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.906 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 8.b last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 8.a last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 8.d last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 8.c last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.907 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.908 INFO:tasks.ceph:pgid 8.f last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.908 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-04-19T00:44:33.030587+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=44, tm_sec=33, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.908 INFO:tasks.ceph:pgid 8.e last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.908 INFO:tasks.ceph:pgid 8.1f last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.908 INFO:tasks.ceph:pgid 8.1e last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.909 INFO:tasks.ceph:pgid 8.1d last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.909 INFO:tasks.ceph:pgid 8.1c last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.909 INFO:tasks.ceph:pgid 8.13 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.909 INFO:tasks.ceph:pgid 8.12 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.909 INFO:tasks.ceph:pgid 8.11 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.910 INFO:tasks.ceph:pgid 8.10 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.910 INFO:tasks.ceph:pgid 8.17 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.911 INFO:tasks.ceph:pgid 8.16 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.911 INFO:tasks.ceph:pgid 8.15 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.911 INFO:tasks.ceph:pgid 8.14 last_scrub_stamp 2024-04-19T00:45:26.542604+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=0, tm_min=45, tm_sec=26, tm_wday=4, tm_yday=110, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=19, tm_hour=1, tm_min=5, tm_sec=30, tm_wday=4, tm_yday=110, tm_isdst=0) 2024-04-19T01:05:34.911 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-19T01:05:54.913 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-19T01:05:55.047 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.045+0000 7f6ab0676640 10 monclient: get_monmap_and_config 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.045+0000 7f6ab0676640 10 monclient: build_initial_monmap 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient: monmap: 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:55.050481+0000 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:55.050481+0000 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:55.048 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:55.049 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.053 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 Processor -- start 2024-04-19T01:05:55.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 -- start start 2024-04-19T01:05:55.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient: init 2024-04-19T01:05:55.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:55.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient: _add_conns ranks=[4,3,2,1,8,5,7,9,10,0,6] 2024-04-19T01:05:55.054 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa806c240 0x7f6aa80a2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa806c240 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aa80a2f90 0x7f6aa80a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): picked mon.noname-g con 0x7f6aa80a2f90 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 0x7f6aa80a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): picked mon.noname-b con 0x7f6aa80a4f70 addr [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 -- --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa806c620 con 0x7f6aa80a4f70 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa81374f0 con 0x7f6aa80a2f90 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.055 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f6aa80722d0 con 0x7f6aa806c240 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6ab0676640 10 monclient(hunting): authenticate will time out at 2225.177855s 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 0x7f6aa80a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa806c240 0x7f6aa80a2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 --2- >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 0x7f6aa80a53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3301/0 says I am v2:172.21.15.43:35450/0 (socket says 172.21.15.43:35450) 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 -- 172.21.15.43:0/1343616452 learned_addr learned my addr 172.21.15.43:0/1343616452 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): get_auth_request con 0x7f6aa80a4f70 auth_method 0 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): get_auth_request con 0x7f6aa806c240 auth_method 0 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.056 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.057 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aae412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient(hunting): handle_auth_done global_id 144740 payload 274 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 -- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa806c240 msgr2=0x7f6aa80a2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aadc11640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aa80a2f90 0x7f6aa80a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa806c240 0x7f6aa80a2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 -- 172.21.15.43:0/1343616452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aa80a2f90 msgr2=0x7f6aa80a4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aa80a2f90 0x7f6aa80a4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient: _finish_hunting 0 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 monclient: found mon.noname-b 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.43:3301/0 2024-04-19T01:05:55.058 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aaec13640 1 -- 172.21.15.43:0/1343616452 --> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa8072f50 con 0x7f6aa80a4f70 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aadc11640 10 monclient: get_auth_request con 0x7f6aa80a2f90 auth_method 0 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.049+0000 7f6aadc11640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aa80a2f90 0x7f6aa80a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 0x7f6aa80a53d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00098a0 tx=0x7f6aa001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64771394f972975e server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 1 -- 172.21.15.43:0/1343616452 <== mon.2 v2:172.21.15.43:3301/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6aa00b0070 con 0x7f6aa80a4f70 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: dump: 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.059 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: _finish_auth 0 2024-04-19T01:05:55.060 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: _check_auth_tickets 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 1 -- 172.21.15.43:0/1343616452 <== mon.2 v2:172.21.15.43:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa00aa810 con 0x7f6aa80a4f70 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 5 monclient: authenticate success, global_id 144740 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 1 -- 172.21.15.43:0/1343616452 <== mon.2 v2:172.21.15.43:3301/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6aa00a4e60 con 0x7f6aa80a4f70 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 4 set_mon_vals no callback set 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aad410640 10 monclient: dump: 2024-04-19T01:05:55.061 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.062 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: shutdown 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 msgr2=0x7f6aa80a53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 0x7f6aa80a53d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6aa00098a0 tx=0x7f6aa001c4b0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- 172.21.15.43:0/1343616452 shutdown_connections 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] conn(0x7f6aa80a4f70 0x7f6aa80a53d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7f6aa80a2f90 0x7f6aa80a4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- 172.21.15.43:0/1343616452 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa806c240 0x7f6aa80a2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- 172.21.15.43:0/1343616452 >> 172.21.15.43:0/1343616452 conn(0x7f6aa80a62e0 msgr2=0x7f6aa806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:55.063 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- 172.21.15.43:0/1343616452 shutdown_connections 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- 172.21.15.43:0/1343616452 wait complete. 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: build_initial_monmap 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: monmap: 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:55.064 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 librados: starting msgr at 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 librados: starting objecter 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 Processor -- start 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- start start 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 librados: setting wanted keys 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 librados: calling monclient init 2024-04-19T01:05:55.065 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: init 2024-04-19T01:05:55.070 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient: _add_conns ranks=[8,1,4,2,7,6,0,5,3] 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6aa806c240 0x7f6aa806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): picked mon.noname-e con 0x7f6aa806c240 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aa80a2f90 0x7f6aa8135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): picked mon.noname-f con 0x7f6aa80a2f90 addr [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 0x7f6aa8135970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): picked mon.noname-c con 0x7f6aa80a4f70 addr [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa80a34f0 con 0x7f6aa80a2f90 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.071 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_getmap magic: 0 -- 0x7f6aa80a3370 con 0x7f6aa80a4f70 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7f6aa8140510 con 0x7f6aa806c240 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6ab0676640 10 monclient(hunting): authenticate will time out at 2225.183047s 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6aa806c240 0x7f6aa806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 0x7f6aa8135970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 --2- >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 0x7f6aa8135970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3302/0 says I am v2:172.21.15.43:39862/0 (socket says 172.21.15.43:39862) 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6aa806c240 0x7f6aa806eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:47728/0 (socket says 172.21.15.43:47728) 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 -- 172.21.15.43:0/3454864729 learned_addr learned my addr 172.21.15.43:0/3454864729 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): get_auth_request con 0x7f6aa80a4f70 auth_method 0 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.072 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): get_auth_request con 0x7f6aa806c240 auth_method 0 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aae412640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aadc11640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aa80a2f90 0x7f6aa8135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.073 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient(hunting): handle_auth_done global_id 184288 payload 824 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 -- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6aa806c240 msgr2=0x7f6aa806eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6aa806c240 0x7f6aa806eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 -- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aa80a2f90 msgr2=0x7f6aa8135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aa80a2f90 0x7f6aa8135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient: _finish_hunting 0 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 monclient: found mon.noname-c 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.43:3302/0 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 -- 172.21.15.43:0/3454864729 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa813c6f0 con 0x7f6aa80a4f70 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aadc11640 10 monclient: get_auth_request con 0x7f6aa80a2f90 auth_method 0 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aadc11640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aa80a2f90 0x7f6aa8135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-19T01:05:55.074 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6aaec13640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 0x7f6aa8135970 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6aa001c4f0 tx=0x7f6aa0019250 comp rx=0 tx=0).ready entity=mon.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6a9effd640 10 client.?.objecter ms_handle_connect 0x7f6aa80a4f70 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6a9effd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.053+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mon.4 v2:172.21.15.43:3302/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6aa00bb070 con 0x7f6aa80a4f70 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: dump: 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.075 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: _finish_auth 0 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:55.076 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mon.4 v2:172.21.15.43:3302/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa0004040 con 0x7f6aa80a4f70 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mon.4 v2:172.21.15.43:3302/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7f6aa00b0040 con 0x7f6aa80a4f70 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 5 monclient: authenticate success, global_id 184288 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 monclient: _renew_subs 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa80724f0 con 0x7f6aa80a4f70 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6aacc0f640 4 set_mon_vals no callback set 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 monclient: dump: 2024-04-19T01:05:55.077 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.078 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 client.184288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 monclient: _renew_subs 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:55.079 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa813cb00 con 0x7f6aa80a4f70 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 1 librados: init done 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mon.4 v2:172.21.15.43:3302/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7f6aa00041f0 con 0x7f6aa80a4f70 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.43:3302/0 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 --> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa806bf10 con 0x7f6aa80a4f70 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a8c038e10 0x7f6a8c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mon.4 v2:172.21.15.43:3302/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7f6aa00e2780 con 0x7f6aa80a4f70 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 client.184288.objecter ms_dispatch 0x7f6aa8000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 3 client.184288.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 3 client.184288.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 20 client.184288.objecter dump_active .. 0 homeless 2024-04-19T01:05:55.080 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6aae412640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a8c038e10 0x7f6a8c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6aae412640 10 monclient: get_auth_request con 0x7f6a8c038e10 auth_method 0 2024-04-19T01:05:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6a9effd640 10 client.184288.objecter ms_handle_connect 0x7f6a8c038e10 2024-04-19T01:05:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.057+0000 7f6aae412640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a8c038e10 0x7f6a8c03b2d0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009ac0 tx=0x7f6a98009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.061+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mon.4 v2:172.21.15.43:3302/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6aa0016130 con 0x7f6aa80a4f70 2024-04-19T01:05:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.061+0000 7f6a9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:55.081 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.061+0000 7f6a9effd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:55.225 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.225+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 --> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6aa806df70 con 0x7f6a8c038e10 2024-04-19T01:05:55.232 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6a9effd640 1 -- 172.21.15.43:0/3454864729 <== mgr.4106 v2:172.21.15.161:6824/2963482743 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+113916 (secure 0 0 0) 0x7f6aa806df70 con 0x7f6a8c038e10 2024-04-19T01:05:55.234 INFO:teuthology.orchestra.run.smithi043.stdout: 2024-04-19T01:05:55.234 INFO:teuthology.orchestra.run.smithi043.stderr:dumped all 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 10 librados: watch_flush enter 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 10 librados: watch_flush exit 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 20 client.184288.objecter shutdown clearing up homeless session... 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 10 client.184288.objecter successfully canceled tick 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a8c038e10 msgr2=0x7f6a8c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a8c038e10 0x7f6a8c03b2d0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009ac0 tx=0x7f6a98009190 comp rx=0 tx=0).stop 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 10 monclient: shutdown 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 msgr2=0x7f6aa8135970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.236 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 0x7f6aa8135970 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6aa001c4f0 tx=0x7f6aa0019250 comp rx=0 tx=0).stop 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 shutdown_connections 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] conn(0x7f6aa80a4f70 0x7f6aa8135970 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6aa80a2f90 0x7f6aa8135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7f6a8c038e10 0x7f6a8c03b2d0 unknown :-1 s=CLOSED pgs=3689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 --2- 172.21.15.43:0/3454864729 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7f6aa806c240 0x7f6aa806eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 >> 172.21.15.43:0/3454864729 conn(0x7f6aa80a62e0 msgr2=0x7f6aa8132ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 shutdown_connections 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 -- 172.21.15.43:0/3454864729 wait complete. 2024-04-19T01:05:55.237 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.233+0000 7f6ab0676640 1 librados: shutdown 2024-04-19T01:05:55.252 INFO:teuthology.orchestra.run.smithi043.stdout:{"pg_ready":true,"pg_map":{"version":1340,"stamp":"2024-04-19T01:05:53.340180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590373,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":166,"num_write_kb":2423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":448544,"kb_used_data":24112,"kb_used_omap":9,"kb_used_meta":424374,"kb_avail":565782496,"statfs":{"total":579820584960,"available":579361275904,"internally_reserved":0,"allocated":24690688,"data_stored":10993320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":434559649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"8.18","version":"0'0","reported_seq":1381,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:38.836676+0000","last_change":"2024-04-19T01:05:38.836712+0000","last_active":"2024-04-19T01:05:38.836676+0000","last_peered":"2024-04-19T01:05:38.836676+0000","last_clean":"2024-04-19T01:05:38.836676+0000","last_became_active":"2024-04-19T00:45:27.796802+0000","last_became_peered":"2024-04-19T00:45:27.796802+0000","last_unstale":"2024-04-19T01:05:38.836676+0000","last_undegraded":"2024-04-19T01:05:38.836676+0000","last_fullsized":"2024-04-19T01:05:38.836676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:38.836561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:38.836561+0000","last_clean_scrub_stamp":"2024-04-19T01:05:38.836561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T13:00:25.976419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:37.582506+0000","last_change":"2024-04-19T01:05:37.582560+0000","last_active":"2024-04-19T01:05:37.582506+0000","last_peered":"2024-04-19T01:05:37.582506+0000","last_clean":"2024-04-19T01:05:37.582506+0000","last_became_active":"2024-04-19T00:45:27.715591+0000","last_became_peered":"2024-04-19T00:45:27.715591+0000","last_unstale":"2024-04-19T01:05:37.582506+0000","last_undegraded":"2024-04-19T01:05:37.582506+0000","last_fullsized":"2024-04-19T01:05:37.582506+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:37.582336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:37.582336+0000","last_clean_scrub_stamp":"2024-04-19T01:05:37.582336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:03:49.728454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:36.581660+0000","last_change":"2024-04-19T01:05:36.581716+0000","last_active":"2024-04-19T01:05:36.581660+0000","last_peered":"2024-04-19T01:05:36.581660+0000","last_clean":"2024-04-19T01:05:36.581660+0000","last_became_active":"2024-04-19T00:45:27.744986+0000","last_became_peered":"2024-04-19T00:45:27.744986+0000","last_unstale":"2024-04-19T01:05:36.581660+0000","last_undegraded":"2024-04-19T01:05:36.581660+0000","last_fullsized":"2024-04-19T01:05:36.581660+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:36.581466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:36.581466+0000","last_clean_scrub_stamp":"2024-04-19T01:05:36.581466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:16:11.516782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1381,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:36.761274+0000","last_change":"2024-04-19T01:05:36.761336+0000","last_active":"2024-04-19T01:05:36.761274+0000","last_peered":"2024-04-19T01:05:36.761274+0000","last_clean":"2024-04-19T01:05:36.761274+0000","last_became_active":"2024-04-19T00:45:27.767040+0000","last_became_peered":"2024-04-19T00:45:27.767040+0000","last_unstale":"2024-04-19T01:05:36.761274+0000","last_undegraded":"2024-04-19T01:05:36.761274+0000","last_fullsized":"2024-04-19T01:05:36.761274+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:36.761108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:36.761108+0000","last_clean_scrub_stamp":"2024-04-19T01:05:36.761108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:16:32.522587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1389,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:38.563504+0000","last_change":"2024-04-19T01:05:38.563559+0000","last_active":"2024-04-19T01:05:38.563504+0000","last_peered":"2024-04-19T01:05:38.563504+0000","last_clean":"2024-04-19T01:05:38.563504+0000","last_became_active":"2024-04-19T00:45:27.717286+0000","last_became_peered":"2024-04-19T00:45:27.717286+0000","last_unstale":"2024-04-19T01:05:38.563504+0000","last_undegraded":"2024-04-19T01:05:38.563504+0000","last_fullsized":"2024-04-19T01:05:38.563504+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:38.563328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:38.563328+0000","last_clean_scrub_stamp":"2024-04-19T01:05:38.563328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:25:08.909294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1382,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:39.810060+0000","last_change":"2024-04-19T01:05:39.810112+0000","last_active":"2024-04-19T01:05:39.810060+0000","last_peered":"2024-04-19T01:05:39.810060+0000","last_clean":"2024-04-19T01:05:39.810060+0000","last_became_active":"2024-04-19T00:45:27.795414+0000","last_became_peered":"2024-04-19T00:45:27.795414+0000","last_unstale":"2024-04-19T01:05:39.810060+0000","last_undegraded":"2024-04-19T01:05:39.810060+0000","last_fullsized":"2024-04-19T01:05:39.810060+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:39.809882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:39.809882+0000","last_clean_scrub_stamp":"2024-04-19T01:05:39.809882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:56:21.240212+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:39.544538+0000","last_change":"2024-04-19T01:05:39.544643+0000","last_active":"2024-04-19T01:05:39.544538+0000","last_peered":"2024-04-19T01:05:39.544538+0000","last_clean":"2024-04-19T01:05:39.544538+0000","last_became_active":"2024-04-19T00:45:27.790389+0000","last_became_peered":"2024-04-19T00:45:27.790389+0000","last_unstale":"2024-04-19T01:05:39.544538+0000","last_undegraded":"2024-04-19T01:05:39.544538+0000","last_fullsized":"2024-04-19T01:05:39.544538+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:39.544318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:39.544318+0000","last_clean_scrub_stamp":"2024-04-19T01:05:39.544318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:39:38.080155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:35.695982+0000","last_change":"2024-04-19T01:05:35.696043+0000","last_active":"2024-04-19T01:05:35.695982+0000","last_peered":"2024-04-19T01:05:35.695982+0000","last_clean":"2024-04-19T01:05:35.695982+0000","last_became_active":"2024-04-19T00:45:27.735932+0000","last_became_peered":"2024-04-19T00:45:27.735932+0000","last_unstale":"2024-04-19T01:05:35.695982+0000","last_undegraded":"2024-04-19T01:05:35.695982+0000","last_fullsized":"2024-04-19T01:05:35.695982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:35.695790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:35.695790+0000","last_clean_scrub_stamp":"2024-04-19T01:05:35.695790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:49:53.187561+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1373,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:37.705300+0000","last_change":"2024-04-19T01:05:37.705356+0000","last_active":"2024-04-19T01:05:37.705300+0000","last_peered":"2024-04-19T01:05:37.705300+0000","last_clean":"2024-04-19T01:05:37.705300+0000","last_became_active":"2024-04-19T00:45:27.752978+0000","last_became_peered":"2024-04-19T00:45:27.752978+0000","last_unstale":"2024-04-19T01:05:37.705300+0000","last_undegraded":"2024-04-19T01:05:37.705300+0000","last_fullsized":"2024-04-19T01:05:37.705300+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:37.705125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:37.705125+0000","last_clean_scrub_stamp":"2024-04-19T01:05:37.705125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:47:41.519788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"707'8","reported_seq":1386,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:38.738178+0000","last_change":"2024-04-19T01:05:38.738240+0000","last_active":"2024-04-19T01:05:38.738178+0000","last_peered":"2024-04-19T01:05:38.738178+0000","last_clean":"2024-04-19T01:05:38.738178+0000","last_became_active":"2024-04-19T00:45:27.798161+0000","last_became_peered":"2024-04-19T00:45:27.798161+0000","last_unstale":"2024-04-19T01:05:38.738178+0000","last_undegraded":"2024-04-19T01:05:38.738178+0000","last_fullsized":"2024-04-19T01:05:38.738178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"707'8","last_scrub_stamp":"2024-04-19T01:05:38.737984+0000","last_deep_scrub":"707'8","last_deep_scrub_stamp":"2024-04-19T01:05:38.737984+0000","last_clean_scrub_stamp":"2024-04-19T01:05:38.737984+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:41:56.414920+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1382,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:40.830049+0000","last_change":"2024-04-19T01:05:40.830100+0000","last_active":"2024-04-19T01:05:40.830049+0000","last_peered":"2024-04-19T01:05:40.830049+0000","last_clean":"2024-04-19T01:05:40.830049+0000","last_became_active":"2024-04-19T00:45:27.797033+0000","last_became_peered":"2024-04-19T00:45:27.797033+0000","last_unstale":"2024-04-19T01:05:40.830049+0000","last_undegraded":"2024-04-19T01:05:40.830049+0000","last_fullsized":"2024-04-19T01:05:40.830049+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:40.829878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:40.829878+0000","last_clean_scrub_stamp":"2024-04-19T01:05:40.829878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:39:14.093359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:36.665159+0000","last_change":"2024-04-19T01:05:36.665213+0000","last_active":"2024-04-19T01:05:36.665159+0000","last_peered":"2024-04-19T01:05:36.665159+0000","last_clean":"2024-04-19T01:05:36.665159+0000","last_became_active":"2024-04-19T00:45:27.757597+0000","last_became_peered":"2024-04-19T00:45:27.757597+0000","last_unstale":"2024-04-19T01:05:36.665159+0000","last_undegraded":"2024-04-19T01:05:36.665159+0000","last_fullsized":"2024-04-19T01:05:36.665159+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:36.664967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:36.664967+0000","last_clean_scrub_stamp":"2024-04-19T01:05:36.664967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:15:17.409601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1396,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:36.452985+0000","last_change":"2024-04-19T01:05:36.453037+0000","last_active":"2024-04-19T01:05:36.452985+0000","last_peered":"2024-04-19T01:05:36.452985+0000","last_clean":"2024-04-19T01:05:36.452985+0000","last_became_active":"2024-04-19T00:44:34.474322+0000","last_became_peered":"2024-04-19T00:44:34.474322+0000","last_unstale":"2024-04-19T01:05:36.452985+0000","last_undegraded":"2024-04-19T01:05:36.452985+0000","last_fullsized":"2024-04-19T01:05:36.452985+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:36.452805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:36.452805+0000","last_clean_scrub_stamp":"2024-04-19T01:05:36.452805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:15:42.650710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1389,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:39.599929+0000","last_change":"2024-04-19T01:05:39.599984+0000","last_active":"2024-04-19T01:05:39.599929+0000","last_peered":"2024-04-19T01:05:39.599929+0000","last_clean":"2024-04-19T01:05:39.599929+0000","last_became_active":"2024-04-19T00:45:27.740409+0000","last_became_peered":"2024-04-19T00:45:27.740409+0000","last_unstale":"2024-04-19T01:05:39.599929+0000","last_undegraded":"2024-04-19T01:05:39.599929+0000","last_fullsized":"2024-04-19T01:05:39.599929+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:39.599766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:39.599766+0000","last_clean_scrub_stamp":"2024-04-19T01:05:39.599766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:31:24.241435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":1396,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:35.588003+0000","last_change":"2024-04-19T01:05:35.588057+0000","last_active":"2024-04-19T01:05:35.588003+0000","last_peered":"2024-04-19T01:05:35.588003+0000","last_clean":"2024-04-19T01:05:35.588003+0000","last_became_active":"2024-04-19T00:44:34.472776+0000","last_became_peered":"2024-04-19T00:44:34.472776+0000","last_unstale":"2024-04-19T01:05:35.588003+0000","last_undegraded":"2024-04-19T01:05:35.588003+0000","last_fullsized":"2024-04-19T01:05:35.588003+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:35.587824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:35.587824+0000","last_clean_scrub_stamp":"2024-04-19T01:05:35.587824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:01:18.664390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1389,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:36.563833+0000","last_change":"2024-04-19T01:05:36.563889+0000","last_active":"2024-04-19T01:05:36.563833+0000","last_peered":"2024-04-19T01:05:36.563833+0000","last_clean":"2024-04-19T01:05:36.563833+0000","last_became_active":"2024-04-19T00:45:27.677388+0000","last_became_peered":"2024-04-19T00:45:27.677388+0000","last_unstale":"2024-04-19T01:05:36.563833+0000","last_undegraded":"2024-04-19T01:05:36.563833+0000","last_fullsized":"2024-04-19T01:05:36.563833+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:36.563653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:36.563653+0000","last_clean_scrub_stamp":"2024-04-19T01:05:36.563653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:48:45.409626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1389,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:35.804462+0000","last_change":"2024-04-19T01:05:35.804521+0000","last_active":"2024-04-19T01:05:35.804462+0000","last_peered":"2024-04-19T01:05:35.804462+0000","last_clean":"2024-04-19T01:05:35.804462+0000","last_became_active":"2024-04-19T00:44:34.473253+0000","last_became_peered":"2024-04-19T00:44:34.473253+0000","last_unstale":"2024-04-19T01:05:35.804462+0000","last_undegraded":"2024-04-19T01:05:35.804462+0000","last_fullsized":"2024-04-19T01:05:35.804462+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:35.804249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:35.804249+0000","last_clean_scrub_stamp":"2024-04-19T01:05:35.804249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:28:55.142019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1376,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:38.592207+0000","last_change":"2024-04-19T01:05:38.592207+0000","last_active":"2024-04-19T01:05:38.592207+0000","last_peered":"2024-04-19T01:05:38.592207+0000","last_clean":"2024-04-19T01:05:38.592207+0000","last_became_active":"2024-04-19T00:45:27.785226+0000","last_became_peered":"2024-04-19T00:45:27.785226+0000","last_unstale":"2024-04-19T01:05:38.592207+0000","last_undegraded":"2024-04-19T01:05:38.592207+0000","last_fullsized":"2024-04-19T01:05:38.592207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:38.591963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:38.591963+0000","last_clean_scrub_stamp":"2024-04-19T01:05:38.591963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:09:11.788120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.690161+0000","last_change":"2024-04-19T01:05:32.690213+0000","last_active":"2024-04-19T01:05:32.690161+0000","last_peered":"2024-04-19T01:05:32.690161+0000","last_clean":"2024-04-19T01:05:32.690161+0000","last_became_active":"2024-04-19T00:44:34.069958+0000","last_became_peered":"2024-04-19T00:44:34.069958+0000","last_unstale":"2024-04-19T01:05:32.690161+0000","last_undegraded":"2024-04-19T01:05:32.690161+0000","last_fullsized":"2024-04-19T01:05:32.690161+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.690040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:32.690040+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.690040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:01:45.849298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"707'119","reported_seq":1548,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.445125+0000","last_change":"2024-04-19T01:05:34.445258+0000","last_active":"2024-04-19T01:05:34.445125+0000","last_peered":"2024-04-19T01:05:34.445125+0000","last_clean":"2024-04-19T01:05:34.445125+0000","last_became_active":"2024-04-19T00:44:34.072434+0000","last_became_peered":"2024-04-19T00:44:34.072434+0000","last_unstale":"2024-04-19T01:05:34.445125+0000","last_undegraded":"2024-04-19T01:05:34.445125+0000","last_fullsized":"2024-04-19T01:05:34.445125+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"707'119","last_scrub_stamp":"2024-04-19T01:05:34.444920+0000","last_deep_scrub":"707'119","last_deep_scrub_stamp":"2024-04-19T01:05:34.444920+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.444920+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:53:20.931019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:41.673503+0000","last_change":"2024-04-19T01:05:41.673609+0000","last_active":"2024-04-19T01:05:41.673503+0000","last_peered":"2024-04-19T01:05:41.673503+0000","last_clean":"2024-04-19T01:05:41.673503+0000","last_became_active":"2024-04-19T00:45:27.813538+0000","last_became_peered":"2024-04-19T00:45:27.813538+0000","last_unstale":"2024-04-19T01:05:41.673503+0000","last_undegraded":"2024-04-19T01:05:41.673503+0000","last_fullsized":"2024-04-19T01:05:41.673503+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:41.673289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:41.673289+0000","last_clean_scrub_stamp":"2024-04-19T01:05:41.673289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:36:38.024065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":1379,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.696496+0000","last_change":"2024-04-19T01:05:33.696524+0000","last_active":"2024-04-19T01:05:33.696496+0000","last_peered":"2024-04-19T01:05:33.696496+0000","last_clean":"2024-04-19T01:05:33.696496+0000","last_became_active":"2024-04-19T00:44:34.071525+0000","last_became_peered":"2024-04-19T00:44:34.071525+0000","last_unstale":"2024-04-19T01:05:33.696496+0000","last_undegraded":"2024-04-19T01:05:33.696496+0000","last_fullsized":"2024-04-19T01:05:33.696496+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:33.696400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:33.696400+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.696400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:30:37.718545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1389,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:41.608037+0000","last_change":"2024-04-19T01:05:41.608037+0000","last_active":"2024-04-19T01:05:41.608037+0000","last_peered":"2024-04-19T01:05:41.608037+0000","last_clean":"2024-04-19T01:05:41.608037+0000","last_became_active":"2024-04-19T00:45:27.763230+0000","last_became_peered":"2024-04-19T00:45:27.763230+0000","last_unstale":"2024-04-19T01:05:41.608037+0000","last_undegraded":"2024-04-19T01:05:41.608037+0000","last_fullsized":"2024-04-19T01:05:41.608037+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:41.607817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:41.607817+0000","last_clean_scrub_stamp":"2024-04-19T01:05:41.607817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:47:21.744037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":1389,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.756335+0000","last_change":"2024-04-19T01:05:34.756387+0000","last_active":"2024-04-19T01:05:34.756335+0000","last_peered":"2024-04-19T01:05:34.756335+0000","last_clean":"2024-04-19T01:05:34.756335+0000","last_became_active":"2024-04-19T00:44:34.064060+0000","last_became_peered":"2024-04-19T00:44:34.064060+0000","last_unstale":"2024-04-19T01:05:34.756335+0000","last_undegraded":"2024-04-19T01:05:34.756335+0000","last_fullsized":"2024-04-19T01:05:34.756335+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:34.756144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:34.756144+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.756144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:52:14.473833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1376,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:32.544115+0000","last_change":"2024-04-19T01:05:32.544115+0000","last_active":"2024-04-19T01:05:32.544115+0000","last_peered":"2024-04-19T01:05:32.544115+0000","last_clean":"2024-04-19T01:05:32.544115+0000","last_became_active":"2024-04-19T00:45:27.647106+0000","last_became_peered":"2024-04-19T00:45:27.647106+0000","last_unstale":"2024-04-19T01:05:32.544115+0000","last_undegraded":"2024-04-19T01:05:32.544115+0000","last_fullsized":"2024-04-19T01:05:32.544115+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:32.543872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:32.543872+0000","last_clean_scrub_stamp":"2024-04-19T01:05:32.543872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:06:37.170038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":1395,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:35.404268+0000","last_change":"2024-04-19T01:05:35.404402+0000","last_active":"2024-04-19T01:05:35.404268+0000","last_peered":"2024-04-19T01:05:35.404268+0000","last_clean":"2024-04-19T01:05:35.404268+0000","last_became_active":"2024-04-19T00:44:34.472447+0000","last_became_peered":"2024-04-19T00:44:34.472447+0000","last_unstale":"2024-04-19T01:05:35.404268+0000","last_undegraded":"2024-04-19T01:05:35.404268+0000","last_fullsized":"2024-04-19T01:05:35.404268+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:35.404041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:35.404041+0000","last_clean_scrub_stamp":"2024-04-19T01:05:35.404041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T03:19:39.210968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:33.515165+0000","last_change":"2024-04-19T01:05:33.515165+0000","last_active":"2024-04-19T01:05:33.515165+0000","last_peered":"2024-04-19T01:05:33.515165+0000","last_clean":"2024-04-19T01:05:33.515165+0000","last_became_active":"2024-04-19T00:45:27.659197+0000","last_became_peered":"2024-04-19T00:45:27.659197+0000","last_unstale":"2024-04-19T01:05:33.515165+0000","last_undegraded":"2024-04-19T01:05:33.515165+0000","last_fullsized":"2024-04-19T01:05:33.515165+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:33.515032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:33.515032+0000","last_clean_scrub_stamp":"2024-04-19T01:05:33.515032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:34:25.042255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":1395,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.589157+0000","last_change":"2024-04-19T01:05:34.589212+0000","last_active":"2024-04-19T01:05:34.589157+0000","last_peered":"2024-04-19T01:05:34.589157+0000","last_clean":"2024-04-19T01:05:34.589157+0000","last_became_active":"2024-04-19T00:44:34.065981+0000","last_became_peered":"2024-04-19T00:44:34.065981+0000","last_unstale":"2024-04-19T01:05:34.589157+0000","last_undegraded":"2024-04-19T01:05:34.589157+0000","last_fullsized":"2024-04-19T01:05:34.589157+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:34.588990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:34.588990+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.588990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:45:33.845700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.528423+0000","last_change":"2024-04-19T01:05:34.528423+0000","last_active":"2024-04-19T01:05:34.528423+0000","last_peered":"2024-04-19T01:05:34.528423+0000","last_clean":"2024-04-19T01:05:34.528423+0000","last_became_active":"2024-04-19T00:45:27.661535+0000","last_became_peered":"2024-04-19T00:45:27.661535+0000","last_unstale":"2024-04-19T01:05:34.528423+0000","last_undegraded":"2024-04-19T01:05:34.528423+0000","last_fullsized":"2024-04-19T01:05:34.528423+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:34.528282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:34.528282+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.528282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:19:56.342230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:35.570317+0000","last_change":"2024-04-19T01:05:35.570374+0000","last_active":"2024-04-19T01:05:35.570317+0000","last_peered":"2024-04-19T01:05:35.570317+0000","last_clean":"2024-04-19T01:05:35.570317+0000","last_became_active":"2024-04-19T00:45:27.719717+0000","last_became_peered":"2024-04-19T00:45:27.719717+0000","last_unstale":"2024-04-19T01:05:35.570317+0000","last_undegraded":"2024-04-19T01:05:35.570317+0000","last_fullsized":"2024-04-19T01:05:35.570317+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:35.570133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:35.570133+0000","last_clean_scrub_stamp":"2024-04-19T01:05:35.570133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T02:55:14.847989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1372,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:34.688857+0000","last_change":"2024-04-19T01:05:34.688990+0000","last_active":"2024-04-19T01:05:34.688857+0000","last_peered":"2024-04-19T01:05:34.688857+0000","last_clean":"2024-04-19T01:05:34.688857+0000","last_became_active":"2024-04-19T00:45:27.719793+0000","last_became_peered":"2024-04-19T00:45:27.719793+0000","last_unstale":"2024-04-19T01:05:34.688857+0000","last_undegraded":"2024-04-19T01:05:34.688857+0000","last_fullsized":"2024-04-19T01:05:34.688857+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:34.688651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:34.688651+0000","last_clean_scrub_stamp":"2024-04-19T01:05:34.688651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T04:49:08.355953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1375,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:37.606593+0000","last_change":"2024-04-19T01:05:37.606649+0000","last_active":"2024-04-19T01:05:37.606593+0000","last_peered":"2024-04-19T01:05:37.606593+0000","last_clean":"2024-04-19T01:05:37.606593+0000","last_became_active":"2024-04-19T00:45:27.769635+0000","last_became_peered":"2024-04-19T00:45:27.769635+0000","last_unstale":"2024-04-19T01:05:37.606593+0000","last_undegraded":"2024-04-19T01:05:37.606593+0000","last_fullsized":"2024-04-19T01:05:37.606593+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:37.606385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:37.606385+0000","last_clean_scrub_stamp":"2024-04-19T01:05:37.606385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T09:28:57.246940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:40.557676+0000","last_change":"2024-04-19T01:05:40.557778+0000","last_active":"2024-04-19T01:05:40.557676+0000","last_peered":"2024-04-19T01:05:40.557676+0000","last_clean":"2024-04-19T01:05:40.557676+0000","last_became_active":"2024-04-19T00:45:27.751895+0000","last_became_peered":"2024-04-19T00:45:27.751895+0000","last_unstale":"2024-04-19T01:05:40.557676+0000","last_undegraded":"2024-04-19T01:05:40.557676+0000","last_fullsized":"2024-04-19T01:05:40.557676+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:40.557467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:40.557467+0000","last_clean_scrub_stamp":"2024-04-19T01:05:40.557467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T08:34:51.828676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1371,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:39.733716+0000","last_change":"2024-04-19T01:05:39.733771+0000","last_active":"2024-04-19T01:05:39.733716+0000","last_peered":"2024-04-19T01:05:39.733716+0000","last_clean":"2024-04-19T01:05:39.733716+0000","last_became_active":"2024-04-19T00:45:27.812520+0000","last_became_peered":"2024-04-19T00:45:27.812520+0000","last_unstale":"2024-04-19T01:05:39.733716+0000","last_undegraded":"2024-04-19T01:05:39.733716+0000","last_fullsized":"2024-04-19T01:05:39.733716+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:39.733525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:39.733525+0000","last_clean_scrub_stamp":"2024-04-19T01:05:39.733525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T07:47:39.822482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:41.466876+0000","last_change":"2024-04-19T01:05:41.466976+0000","last_active":"2024-04-19T01:05:41.466876+0000","last_peered":"2024-04-19T01:05:41.466876+0000","last_clean":"2024-04-19T01:05:41.466876+0000","last_became_active":"2024-04-19T00:45:27.795698+0000","last_became_peered":"2024-04-19T00:45:27.795698+0000","last_unstale":"2024-04-19T01:05:41.466876+0000","last_undegraded":"2024-04-19T01:05:41.466876+0000","last_fullsized":"2024-04-19T01:05:41.466876+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:41.466670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:41.466670+0000","last_clean_scrub_stamp":"2024-04-19T01:05:41.466670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T05:22:21.585560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:38.472080+0000","last_change":"2024-04-19T01:05:38.472184+0000","last_active":"2024-04-19T01:05:38.472080+0000","last_peered":"2024-04-19T01:05:38.472080+0000","last_clean":"2024-04-19T01:05:38.472080+0000","last_became_active":"2024-04-19T00:45:27.750623+0000","last_became_peered":"2024-04-19T00:45:27.750623+0000","last_unstale":"2024-04-19T01:05:38.472080+0000","last_undegraded":"2024-04-19T01:05:38.472080+0000","last_fullsized":"2024-04-19T01:05:38.472080+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:38.471871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:38.471871+0000","last_clean_scrub_stamp":"2024-04-19T01:05:38.471871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T12:10:49.054789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:39.456075+0000","last_change":"2024-04-19T01:05:39.456133+0000","last_active":"2024-04-19T01:05:39.456075+0000","last_peered":"2024-04-19T01:05:39.456075+0000","last_clean":"2024-04-19T01:05:39.456075+0000","last_became_active":"2024-04-19T00:45:27.758559+0000","last_became_peered":"2024-04-19T00:45:27.758559+0000","last_unstale":"2024-04-19T01:05:39.456075+0000","last_undegraded":"2024-04-19T01:05:39.456075+0000","last_fullsized":"2024-04-19T01:05:39.456075+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:39.455844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:39.455844+0000","last_clean_scrub_stamp":"2024-04-19T01:05:39.455844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:35:01.658633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:37.480789+0000","last_change":"2024-04-19T01:05:37.480843+0000","last_active":"2024-04-19T01:05:37.480789+0000","last_peered":"2024-04-19T01:05:37.480789+0000","last_clean":"2024-04-19T01:05:37.480789+0000","last_became_active":"2024-04-19T00:45:27.697796+0000","last_became_peered":"2024-04-19T00:45:27.697796+0000","last_unstale":"2024-04-19T01:05:37.480789+0000","last_undegraded":"2024-04-19T01:05:37.480789+0000","last_fullsized":"2024-04-19T01:05:37.480789+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:37.480608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:37.480608+0000","last_clean_scrub_stamp":"2024-04-19T01:05:37.480608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T01:29:27.721453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1371,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:40.697054+0000","last_change":"2024-04-19T01:05:40.697108+0000","last_active":"2024-04-19T01:05:40.697054+0000","last_peered":"2024-04-19T01:05:40.697054+0000","last_clean":"2024-04-19T01:05:40.697054+0000","last_became_active":"2024-04-19T00:45:27.814761+0000","last_became_peered":"2024-04-19T00:45:27.814761+0000","last_unstale":"2024-04-19T01:05:40.697054+0000","last_undegraded":"2024-04-19T01:05:40.697054+0000","last_fullsized":"2024-04-19T01:05:40.697054+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:40.696871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:40.696871+0000","last_clean_scrub_stamp":"2024-04-19T01:05:40.696871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T10:37:32.880118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1388,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:40.441658+0000","last_change":"2024-04-19T01:05:40.441710+0000","last_active":"2024-04-19T01:05:40.441658+0000","last_peered":"2024-04-19T01:05:40.441658+0000","last_clean":"2024-04-19T01:05:40.441658+0000","last_became_active":"2024-04-19T00:45:27.779682+0000","last_became_peered":"2024-04-19T00:45:27.779682+0000","last_unstale":"2024-04-19T01:05:40.441658+0000","last_undegraded":"2024-04-19T01:05:40.441658+0000","last_fullsized":"2024-04-19T01:05:40.441658+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:40.441488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:40.441488+0000","last_clean_scrub_stamp":"2024-04-19T01:05:40.441488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T06:13:20.557159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1381,"reported_epoch":707,"state":"active+clean","last_fresh":"2024-04-19T01:05:37.809227+0000","last_change":"2024-04-19T01:05:37.809280+0000","last_active":"2024-04-19T01:05:37.809227+0000","last_peered":"2024-04-19T01:05:37.809227+0000","last_clean":"2024-04-19T01:05:37.809227+0000","last_became_active":"2024-04-19T00:45:27.788325+0000","last_became_peered":"2024-04-19T00:45:27.788325+0000","last_unstale":"2024-04-19T01:05:37.809227+0000","last_undegraded":"2024-04-19T01:05:37.809227+0000","last_fullsized":"2024-04-19T01:05:37.809227+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-19T01:05:37.809053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-19T01:05:37.809053+0000","last_clean_scrub_stamp":"2024-04-19T01:05:37.809053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-20T11:23:03.940135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607814,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75276,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296564,"statfs":{"total":96636764160,"available":96559681536,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607813,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75276,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296564,"statfs":{"total":96636764160,"available":96559681536,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73136582},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607812,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75796,"kb_used_data":4364,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94296044,"statfs":{"total":96636764160,"available":96559149056,"internally_reserved":0,"allocated":4468736,"data_stored":2184837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640517,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79436,"kb_used_data":3844,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292404,"statfs":{"total":96636764160,"available":96555421696,"internally_reserved":0,"allocated":3936256,"data_stored":1655909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640514,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71640,"kb_used_data":4368,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300200,"statfs":{"total":96636764160,"available":96563404800,"internally_reserved":0,"allocated":4472832,"data_stored":2184842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640515,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71120,"kb_used_data":3848,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300720,"statfs":{"total":96636764160,"available":96563937280,"internally_reserved":0,"allocated":3940352,"data_stored":1655914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-19T01:05:55.255 DEBUG:teuthology.orchestra.run.smithi043:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient: get_monmap_and_config 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient: build_initial_monmap 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient: monmap: 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T01:05:55.374936+0000 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T01:05:55.374936+0000 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.372 INFO:teuthology.orchestra.run.smithi043.stderr:0: v1:172.21.15.43:6789/0 mon.noname-a-legacy 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:1: v1:172.21.15.161:6789/0 mon.noname-f-legacy 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:9: v2:172.21.15.43:3300/0 mon.noname-a 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr:10: v2:172.21.15.161:3300/0 mon.noname-f 2024-04-19T01:05:55.373 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 Processor -- start 2024-04-19T01:05:55.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 -- start start 2024-04-19T01:05:55.382 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient: init 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient: _add_conns ranks=[9,8,6,1,4,7,3,5,0,10,2] 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 --2- >> v2:172.21.15.43:3300/0 conn(0x7fa914138cd0 0x7fa9141390b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): picked mon.noname-a con 0x7fa914138cd0 addr v2:172.21.15.43:3300/0 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 0x7fa9141395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): picked mon.noname-e con 0x7fa9140a5b80 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa9141358b0 0x7fa914135d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): picked mon.noname-d con 0x7fa9141358b0 addr [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 -- --> v2:172.21.15.43:3300/0 -- mon_getmap magic: 0 -- 0x7fa9140a4410 con 0x7fa914138cd0 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 -- --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_getmap magic: 0 -- 0x7fa914139b30 con 0x7fa9141358b0 2024-04-19T01:05:55.383 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa9140a3dc0 con 0x7fa9140a5b80 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa918d11640 10 monclient(hunting): authenticate will time out at 2225.502257s 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 0x7fa9141395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 --2- >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa9141358b0 0x7fa914135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 0x7fa9141395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:47734/0 (socket says 172.21.15.43:47734) 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 1 -- 172.21.15.43:0/888331354 learned_addr learned my addr 172.21.15.43:0/888331354 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): get_auth_request con 0x7fa9140a5b80 auth_method 0 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): get_auth_request con 0x7fa9141358b0 auth_method 0 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.384 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 1 --2- 172.21.15.43:0/888331354 >> v2:172.21.15.43:3300/0 conn(0x7fa914138cd0 0x7fa9141390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): get_auth_request con 0x7fa914138cd0 auth_method 0 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.385 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa91259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient(hunting): handle_auth_done global_id 184308 payload 274 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 -- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 msgr2=0x7fa9141395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 --2- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 0x7fa9141395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 -- 172.21.15.43:0/888331354 >> v2:172.21.15.43:3300/0 conn(0x7fa914138cd0 msgr2=0x7fa9141390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-19T01:05:55.386 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 --2- 172.21.15.43:0/888331354 >> v2:172.21.15.43:3300/0 conn(0x7fa914138cd0 0x7fa9141390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient: _finish_hunting 0 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 monclient: found mon.noname-d 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 10 monclient: _send_mon_message to mon.noname-d at v2:172.21.15.43:3303/0 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa912d9d640 1 -- 172.21.15.43:0/888331354 --> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9140a40a0 con 0x7fa9141358b0 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.373+0000 7fa911d9b640 1 --2- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 0x7fa9141395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa912d9d640 1 --2- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa9141358b0 0x7fa914135d40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009d90 tx=0x7fa90c01c7d0 comp rx=0 tx=0).ready entity=mon.6 client_cookie=e9b157b54d28bb2a server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 1 -- 172.21.15.43:0/888331354 <== mon.6 v2:172.21.15.43:3303/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa90c0b2070 con 0x7fa9141358b0 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: got monmap 1 from mon.noname-d (according to old e1) 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: dump: 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.387 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.388 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: _finish_auth 0 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: _check_auth_tickets 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:55.389 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 5 monclient: authenticate success, global_id 184308 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 1 -- 172.21.15.43:0/888331354 <== mon.6 v2:172.21.15.43:3303/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa90c0a5990 con 0x7fa9141358b0 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: get_monmap_and_config success 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 4 set_mon_vals no callback set 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 1 -- 172.21.15.43:0/888331354 <== mon.6 v2:172.21.15.43:3303/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa90c0a5bc0 con 0x7fa9141358b0 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.390 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: got monmap 1 from mon.d (according to old e1) 2024-04-19T01:05:55.393 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa91159a640 10 monclient: dump: 2024-04-19T01:05:55.393 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.393 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.393 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.394 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: shutdown 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa9141358b0 msgr2=0x7fa914135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa9141358b0 0x7fa914135d40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009d90 tx=0x7fa90c01c7d0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- 172.21.15.43:0/888331354 shutdown_connections 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] conn(0x7fa9141358b0 0x7fa914135d40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- 172.21.15.43:0/888331354 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9140a5b80 0x7fa9141395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- 172.21.15.43:0/888331354 >> v2:172.21.15.43:3300/0 conn(0x7fa914138cd0 0x7fa9141390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- 172.21.15.43:0/888331354 >> 172.21.15.43:0/888331354 conn(0x7fa9140ab2f0 msgr2=0x7fa9140ab700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- 172.21.15.43:0/888331354 shutdown_connections 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- 172.21.15.43:0/888331354 wait complete. 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: build_initial_monmap 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: monmap: 2024-04-19T01:05:55.395 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 0 2024-04-19T01:05:55.396 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-19T01:05:55.396 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 0.000000 2024-04-19T01:05:55.396 INFO:teuthology.orchestra.run.smithi043.stderr:created 0.000000 2024-04-19T01:05:55.396 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 0 (unknown) 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 1 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.noname-a 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.noname-f 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.noname-b 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.noname-g 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.noname-c 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.noname-h 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.noname-d 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.noname-i 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.noname-e 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 librados: starting msgr at 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 librados: starting objecter 2024-04-19T01:05:55.397 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 Processor -- start 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- start start 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 librados: setting wanted keys 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 librados: calling monclient init 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: init 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: _reopen_session rank -1 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient: _add_conns ranks=[5,8,3,6,0,1,4,7,2] 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa9140a5b80 0x7fa9141d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): picked mon.noname-h con 0x7fa9140a5b80 addr [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] 2024-04-19T01:05:55.398 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 0x7fa91413f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): picked mon.noname-e con 0x7fa9141358b0 addr [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 --2- >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa914138cd0 0x7fa91413f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): picked mon.noname-g con 0x7fa914138cd0 addr [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- --> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] -- mon_getmap magic: 0 -- 0x7fa9140a2010 con 0x7fa914138cd0 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- --> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] -- mon_getmap magic: 0 -- 0x7fa9140a1e90 con 0x7fa9140a5b80 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): start opening mon connection 2024-04-19T01:05:55.399 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 1 -- --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_getmap magic: 0 -- 0x7fa914142aa0 con 0x7fa9141358b0 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): _renew_subs 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa918d11640 10 monclient(hunting): authenticate will time out at 2225.507314s 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 0x7fa91413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 --2- >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 0x7fa91413f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.43:3304/0 says I am v2:172.21.15.43:47746/0 (socket says 172.21.15.43:47746) 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 -- 172.21.15.43:0/2956323937 learned_addr learned my addr 172.21.15.43:0/2956323937 (peer_addr_for_me v2:172.21.15.43:0/0) 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): get_auth_request con 0x7fa9141358b0 auth_method 0 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): _init_auth method 2 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient(hunting): handle_auth_done global_id 144818 payload 824 2024-04-19T01:05:55.400 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 -- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa9140a5b80 msgr2=0x7fa9141d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa9140a5b80 0x7fa9141d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 -- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa914138cd0 msgr2=0x7fa91413f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa912d9d640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa914138cd0 0x7fa91413f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa914138cd0 0x7fa91413f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient: _finish_hunting 0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 monclient: found mon.noname-e 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 10 monclient: _send_mon_message to mon.noname-e at v2:172.21.15.43:3304/0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 -- 172.21.15.43:0/2956323937 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91413a010 con 0x7fa9141358b0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa911d9b640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 0x7fa91413f100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa90400e9b0 tx=0x7fa90400ee70 comp rx=0 tx=0).ready entity=mon.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa902ffd640 10 client.?.objecter ms_handle_connect 0x7fa9141358b0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.377+0000 7fa902ffd640 10 client.?.objecter resend_mon_ops 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 1 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa904017070 con 0x7fa9141358b0 2024-04-19T01:05:55.401 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: got monmap 1 from mon.noname-e (according to old e1) 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: dump: 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.402 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: _finish_auth 0 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: _check_auth_tickets 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa904013470 con 0x7fa9141358b0 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 5 monclient: authenticate success, global_id 144818 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 monclient: _renew_subs 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9141d1460 con 0x7fa9141358b0 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: handle_config config(0 keys) 2024-04-19T01:05:55.403 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 3 ==== mon_map magic: 0 ==== 1037+0+0 (secure 0 0 0) 0x7fa904013a90 con 0x7fa9141358b0 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: got monmap 1 from mon.e (according to old e1) 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa903fff640 4 set_mon_vals no callback set 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 monclient: dump: 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:epoch 1 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:fsid 1280a06a-2032-4fc0-a20f-543d29f598e9 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:last_changed 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:created 2024-04-19T00:43:51.655686+0000 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:min_mon_release 19 (squid) 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:election_strategy: 3 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:0: [v2:172.21.15.43:3300/0,v1:172.21.15.43:6789/0] mon.a 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:1: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.f 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:2: [v2:172.21.15.43:3301/0,v1:172.21.15.43:6790/0] mon.b 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:3: [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] mon.g 2024-04-19T01:05:55.404 INFO:teuthology.orchestra.run.smithi043.stderr:4: [v2:172.21.15.43:3302/0,v1:172.21.15.43:6791/0] mon.c 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:5: [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] mon.h 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:6: [v2:172.21.15.43:3303/0,v1:172.21.15.43:6792/0] mon.d 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:7: [v2:172.21.15.161:3303/0,v1:172.21.15.161:6792/0] mon.i 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:8: [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] mon.e 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 client.144818.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 monclient: _renew_subs 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9140a3040 con 0x7fa9141358b0 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 1 librados: init done 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 4 ==== mgrmap(e 5) ==== 44324+0+0 (secure 0 0 0) 0x7fa90401a070 con 0x7fa9141358b0 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:55.405 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9140693b0 con 0x7fa9141358b0 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa8f4038e10 0x7fa8f403b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 5 ==== osd_map(707..707 src has 682..707) ==== 5497+0+0 (secure 0 0 0) 0x7fa90400f040 con 0x7fa9141358b0 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 client.144818.objecter ms_dispatch 0x7fa914000ba0 osd_map(707..707 src has 682..707) 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 3 client.144818.objecter handle_osd_map got epochs [707,707] > 0 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 3 client.144818.objecter handle_osd_map decoding full epoch 707 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 20 client.144818.objecter dump_active .. 0 homeless 2024-04-19T01:05:55.406 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa91259c640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa8f4038e10 0x7fa8f403b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-19T01:05:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa91259c640 10 monclient: get_auth_request con 0x7fa8f4038e10 auth_method 0 2024-04-19T01:05:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa91259c640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa8f4038e10 0x7fa8f403b2d0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fa9080099c0 tx=0x7fa908009480 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-19T01:05:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.381+0000 7fa902ffd640 10 client.144818.objecter ms_handle_connect 0x7fa8f4038e10 2024-04-19T01:05:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.385+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa90401a360 con 0x7fa9141358b0 2024-04-19T01:05:55.407 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.385+0000 7fa902ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-19T01:05:55.408 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.385+0000 7fa902ffd640 10 monclient: _finish_command 1 = system:0 2024-04-19T01:05:55.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.541+0000 7fa918d11640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-19T01:05:55.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.541+0000 7fa918d11640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-19T01:05:55.542 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.541+0000 7fa918d11640 10 monclient: _send_mon_message to mon.e at v2:172.21.15.43:3304/0 2024-04-19T01:05:55.543 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.541+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 --> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa91413b4c0 con 0x7fa9141358b0 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa904013c30 con 0x7fa9141358b0 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa902ffd640 10 monclient: handle_config config(1 keys) 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa903fff640 4 set_mon_vals no callback set 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa903fff640 10 set_mon_vals mon_health_to_clog = false 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa902ffd640 1 -- 172.21.15.43:0/2956323937 <== mon.8 v2:172.21.15.43:3304/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7fa904011de0 con 0x7fa9141358b0 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa902ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-19T01:05:55.553 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa902ffd640 10 monclient: _finish_command 2 = system:0 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 10 librados: watch_flush enter 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 10 librados: watch_flush exit 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 20 client.144818.objecter shutdown clearing up homeless session... 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 10 client.144818.objecter successfully canceled tick 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa8f4038e10 msgr2=0x7fa8f403b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa8f4038e10 0x7fa8f403b2d0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fa9080099c0 tx=0x7fa908009480 comp rx=0 tx=0).stop 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 10 monclient: shutdown 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-19T01:05:55.556 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 msgr2=0x7fa91413f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.553+0000 7fa918d11640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 0x7fa91413f100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa90400e9b0 tx=0x7fa90400ee70 comp rx=0 tx=0).stop 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 shutdown_connections 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3301/0,v1:172.21.15.161:6790/0] conn(0x7fa914138cd0 0x7fa91413f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:6824/2963482743,v1:172.21.15.161:6825/2963482743] conn(0x7fa8f4038e10 0x7fa8f403b2d0 unknown :-1 s=CLOSED pgs=3690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.43:3304/0,v1:172.21.15.43:6793/0] conn(0x7fa9141358b0 0x7fa91413f100 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 --2- 172.21.15.43:0/2956323937 >> [v2:172.21.15.161:3302/0,v1:172.21.15.161:6791/0] conn(0x7fa9140a5b80 0x7fa9141d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 >> 172.21.15.43:0/2956323937 conn(0x7fa9140ab2f0 msgr2=0x7fa9140a7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 shutdown_connections 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 -- 172.21.15.43:0/2956323937 wait complete. 2024-04-19T01:05:55.557 INFO:teuthology.orchestra.run.smithi043.stderr:2024-04-19T01:05:55.557+0000 7fa918d11640 1 librados: shutdown 2024-04-19T01:05:55.571 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-19T01:05:55.571 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-19T01:05:55.571 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-19T01:05:55.571 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:05:55.643 INFO:tasks.ceph.osd.0:Stopped 2024-04-19T01:05:55.643 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-19T01:05:55.643 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:05:55.731 INFO:tasks.ceph.osd.1:Stopped 2024-04-19T01:05:55.731 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-19T01:05:55.731 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:05:55.821 INFO:tasks.ceph.osd.2:Stopped 2024-04-19T01:05:55.822 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-19T01:05:55.822 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:05:55.917 INFO:tasks.ceph.osd.3:Stopped 2024-04-19T01:05:55.917 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-19T01:05:55.917 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:00.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-19T01:06:02.019 INFO:tasks.ceph.osd.4:Stopped 2024-04-19T01:06:02.020 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-19T01:06:02.020 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.095 INFO:tasks.ceph.osd.5:Stopped 2024-04-19T01:06:02.096 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-19T01:06:02.096 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-19T01:06:02.096 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.126 INFO:tasks.ceph.mgr.x:Stopped 2024-04-19T01:06:02.126 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-19T01:06:02.127 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-19T01:06:02.127 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.149 INFO:tasks.ceph.mon.a:Stopped 2024-04-19T01:06:02.150 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-19T01:06:02.150 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.219 INFO:tasks.ceph.mon.b:Stopped 2024-04-19T01:06:02.219 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-19T01:06:02.219 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.283 INFO:tasks.ceph.mon.c:Stopped 2024-04-19T01:06:02.283 DEBUG:tasks.ceph.mon.d:waiting for process to exit 2024-04-19T01:06:02.283 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.348 INFO:tasks.ceph.mon.d:Stopped 2024-04-19T01:06:02.348 DEBUG:tasks.ceph.mon.e:waiting for process to exit 2024-04-19T01:06:02.348 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.416 INFO:tasks.ceph.mon.e:Stopped 2024-04-19T01:06:02.416 DEBUG:tasks.ceph.mon.f:waiting for process to exit 2024-04-19T01:06:02.416 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.439 INFO:tasks.ceph.mon.f:Stopped 2024-04-19T01:06:02.439 DEBUG:tasks.ceph.mon.g:waiting for process to exit 2024-04-19T01:06:02.439 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.501 INFO:tasks.ceph.mon.g:Stopped 2024-04-19T01:06:02.501 DEBUG:tasks.ceph.mon.h:waiting for process to exit 2024-04-19T01:06:02.501 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.563 INFO:tasks.ceph.mon.h:Stopped 2024-04-19T01:06:02.563 DEBUG:tasks.ceph.mon.i:waiting for process to exit 2024-04-19T01:06:02.563 INFO:teuthology.orchestra.run:waiting for 300 2024-04-19T01:06:02.629 INFO:tasks.ceph.mon.i:Stopped 2024-04-19T01:06:02.630 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-19T01:06:02.630 DEBUG:teuthology.orchestra.run.smithi043:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-04-19T01:06:02.677 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T01:06:02.677 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-19T01:06:03.056 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T01:06:03.056 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-19T01:06:03.289 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T01:06:03.289 DEBUG:teuthology.orchestra.run.smithi043:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-19T01:06:03.463 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T01:06:03.463 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-19T01:06:03.740 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T01:06:03.740 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-19T01:06:03.914 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T01:06:03.915 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-19T01:06:04.056 INFO:tasks.ceph:Archiving mon data... 2024-04-19T01:06:04.057 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.a.tgz 2024-04-19T01:06:04.058 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-19T01:06:04.064 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.bLrrDsCgbr 2024-04-19T01:06:04.064 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.bLrrDsCgbr 2024-04-19T01:06:04.321 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.bLrrDsCgbr 2024-04-19T01:06:04.384 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.bLrrDsCgbr is 1MB 2024-04-19T01:06:04.458 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.bLrrDsCgbr 2024-04-19T01:06:04.464 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.b.tgz 2024-04-19T01:06:04.464 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-19T01:06:04.512 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.pvrlpqRxvC 2024-04-19T01:06:04.513 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pvrlpqRxvC 2024-04-19T01:06:04.769 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.pvrlpqRxvC 2024-04-19T01:06:04.831 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.pvrlpqRxvC is 1MB 2024-04-19T01:06:04.906 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.pvrlpqRxvC 2024-04-19T01:06:04.912 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.c.tgz 2024-04-19T01:06:04.912 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-19T01:06:04.960 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.33d8qjnUw6 2024-04-19T01:06:04.960 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.33d8qjnUw6 2024-04-19T01:06:05.222 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.33d8qjnUw6 2024-04-19T01:06:05.283 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.33d8qjnUw6 is 1MB 2024-04-19T01:06:05.358 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.33d8qjnUw6 2024-04-19T01:06:05.364 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-d to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.d.tgz 2024-04-19T01:06:05.364 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-19T01:06:05.412 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.PSPUCf6sQN 2024-04-19T01:06:05.413 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-d -- . > /tmp/tmp.PSPUCf6sQN 2024-04-19T01:06:05.673 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.PSPUCf6sQN 2024-04-19T01:06:05.735 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.PSPUCf6sQN is 1MB 2024-04-19T01:06:05.806 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.PSPUCf6sQN 2024-04-19T01:06:05.812 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/mon/ceph-e to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.e.tgz 2024-04-19T01:06:05.812 DEBUG:teuthology.orchestra.run.smithi043:> mktemp 2024-04-19T01:06:05.860 INFO:teuthology.orchestra.run.smithi043.stdout:/tmp/tmp.ozfe27FZ6R 2024-04-19T01:06:05.860 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-e -- . > /tmp/tmp.ozfe27FZ6R 2024-04-19T01:06:05.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-19T01:06:06.123 DEBUG:teuthology.orchestra.run.smithi043:> sudo chmod 0666 /tmp/tmp.ozfe27FZ6R 2024-04-19T01:06:06.188 DEBUG:teuthology.orchestra.remote:smithi043:/tmp/tmp.ozfe27FZ6R is 1MB 2024-04-19T01:06:06.263 DEBUG:teuthology.orchestra.run.smithi043:> rm -fr /tmp/tmp.ozfe27FZ6R 2024-04-19T01:06:06.269 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-f to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.f.tgz 2024-04-19T01:06:06.269 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-19T01:06:06.275 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.l4PYTxv17i 2024-04-19T01:06:06.275 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-f -- . > /tmp/tmp.l4PYTxv17i 2024-04-19T01:06:06.529 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.l4PYTxv17i 2024-04-19T01:06:06.595 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.l4PYTxv17i is 1MB 2024-04-19T01:06:06.673 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.l4PYTxv17i 2024-04-19T01:06:06.679 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-g to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.g.tgz 2024-04-19T01:06:06.679 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-19T01:06:06.727 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.azes35uDSq 2024-04-19T01:06:06.727 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-g -- . > /tmp/tmp.azes35uDSq 2024-04-19T01:06:06.981 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.azes35uDSq 2024-04-19T01:06:07.044 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.azes35uDSq is 1MB 2024-04-19T01:06:07.121 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.azes35uDSq 2024-04-19T01:06:07.127 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-h to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.h.tgz 2024-04-19T01:06:07.127 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-19T01:06:07.175 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.fwJT8EOtim 2024-04-19T01:06:07.175 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-h -- . > /tmp/tmp.fwJT8EOtim 2024-04-19T01:06:07.431 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.fwJT8EOtim 2024-04-19T01:06:07.495 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.fwJT8EOtim is 1MB 2024-04-19T01:06:07.565 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.fwJT8EOtim 2024-04-19T01:06:07.571 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-i to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/data/mon.i.tgz 2024-04-19T01:06:07.571 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-04-19T01:06:07.622 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.vyZPZKeHU1 2024-04-19T01:06:07.623 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-i -- . > /tmp/tmp.vyZPZKeHU1 2024-04-19T01:06:07.873 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.vyZPZKeHU1 2024-04-19T01:06:07.934 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.vyZPZKeHU1 is 1MB 2024-04-19T01:06:08.043 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.vyZPZKeHU1 2024-04-19T01:06:08.129 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-19T01:06:08.129 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:06:08.131 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-19T01:06:08.192 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:06:08.231 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-19T01:06:08.232 INFO:tasks.ceph:Archiving crash dumps... 2024-04-19T01:06:08.233 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/remote/smithi043/crash 2024-04-19T01:06:08.234 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:06:08.249 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/remote/smithi161/crash 2024-04-19T01:06:08.250 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-19T01:06:08.288 INFO:tasks.ceph:Compressing logs... 2024-04-19T01:06:08.288 DEBUG:teuthology.orchestra.run.smithi043:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:06:08.297 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-19T01:06:08.317 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26862.log 2024-04-19T01:06:08.318 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36433.log 2024-04-19T01:06:08.318 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34864.log 2024-04-19T01:06:08.319 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36433.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26862.log.gz 2024-04-19T01:06:08.319 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27482.log 2024-04-19T01:06:08.319 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26888.log 2024-04-19T01:06:08.319 INFO:teuthology.orchestra.run.smithi043.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36433.log.gz 2024-04-19T01:06:08.320 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27482.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34864.log.gz 2024-04-19T01:06:08.320 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35284.log 2024-04-19T01:06:08.320 INFO:teuthology.orchestra.run.smithi043.stderr:gzip 88.6% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.27482.log.gz /var/log/ceph/ceph-client.admin.34090.log 2024-04-19T01:06:08.321 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:06:08.321 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26888.log: /var/log/ceph/ceph-client.admin.35284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33255.log 2024-04-19T01:06:08.323 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34610.log 2024-04-19T01:06:08.323 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.26888.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35284.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33255.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34090.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31054.log 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36511.log 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34610.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33255.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30388.log 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34942.log 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31054.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30388.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36511.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34402.log 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.34610.log.gz 2024-04-19T01:06:08.324 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31550.log 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34942.log: /var/log/ceph/ceph-client.admin.34402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35100.log 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30388.log.gz 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29836.log 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31550.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34942.log.gz 89.5% 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34402.log.gz 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36094.log 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29836.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31550.log.gz 2024-04-19T01:06:08.330 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28922.log 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35100.log.gz 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36302.log 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29836.log.gz 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34064.log 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28922.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36094.log.gz 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29082.log 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34480.log 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36302.log.gz 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28922.log.gz 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35520.log 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34064.log.gz 2024-04-19T01:06:08.331 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30176.log 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29082.log.gz 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35520.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34480.log.gz 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30766.log 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27363.log 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35520.log.gz 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30766.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36328.log 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30176.log.gz 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.0.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27363.log.gz 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36354.log 2024-04-19T01:06:08.332 INFO:teuthology.orchestra.run.smithi043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30766.log.gz 2024-04-19T01:06:08.345 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32301.log 2024-04-19T01:06:08.345 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35468.log 2024-04-19T01:06:08.345 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36354.log: /var/log/ceph/ceph-client.admin.32301.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36328.log.gz 2024-04-19T01:06:08.345 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34690.log 2024-04-19T01:06:08.345 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29758.log 2024-04-19T01:06:08.345 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32301.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36354.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36120.log 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35468.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35598.log 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29758.log: /var/log/ceph/ceph-client.admin.36120.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34690.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32561.log 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35598.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36120.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35860.log 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29758.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32821.log 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32561.log: /var/log/ceph/ceph-client.admin.35860.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35598.log.gz 2024-04-19T01:06:08.346 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32717.log 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32561.log.gz 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32821.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35860.log.gz 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33725.log 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34768.log 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27924.log 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33725.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32821.log.gz 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34768.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32717.log.gz 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34350.log 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27924.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.34768.log.gz 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33725.log.gz 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36042.log 2024-04-19T01:06:08.347 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31236.log 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36042.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27924.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36406.log 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34350.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36042.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24052.log 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28708.log 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24052.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31236.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32200.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36406.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34968.log 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24052.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28708.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-04-19T01:06:08.348 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33333.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27742.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33333.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.28708.log.gz 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29108.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33333.log.gz 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34968.log.gz 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27362.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30686.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35442.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27742.log.gz 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27362.log: /var/log/ceph/ceph-client.admin.30686.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29108.log.gz 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-04-19T01:06:08.349 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27362.log.gz 2024-04-19T01:06:08.350 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30686.log.gz 2024-04-19T01:06:08.350 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32044.log 2024-04-19T01:06:08.350 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32327.log 2024-04-19T01:06:08.350 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32847.log 2024-04-19T01:06:08.350 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32044.log: /var/log/ceph/ceph-client.admin.32327.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35442.log.gz 2024-04-19T01:06:08.350 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29212.log 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32327.log.gz 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32847.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32044.log.gz 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34532.log 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28002.log 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.34532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26809.log 2024-04-19T01:06:08.351 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29212.log.gz 2024-04-19T01:06:08.352 INFO:teuthology.orchestra.run.smithi043.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.32847.log.gz 2024-04-19T01:06:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29453.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29650.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28741.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29040.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29650.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29453.log.gz 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30232.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.f.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30232.log: /var/log/ceph/ceph-client.admin.28733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28717.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.f.log: gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30232.log.gz 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30157.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30104.log 2024-04-19T01:06:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24198.log 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30104.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29650.log.gz 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29040.log.gz 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29666.log 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.30104.log.gz 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.24198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29697.log 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 91.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.30157.log.gz -- 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.29257.log 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29666.log: /var/log/ceph/ceph-client.admin.29697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29269.log 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24198.log.gz 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29110.log 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29269.log: 92.2% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29257.log.gz 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-04-19T01:06:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24174.log 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr: 90.1%/var/log/ceph/ceph-client.admin.29110.log: -- replaced with /var/log/ceph/ceph-client.admin.29697.log.gz 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28879.log 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.24174.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.24174.log.gz 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29622.log 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28831.log 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29622.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29269.log.gz 2024-04-19T01:06:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28939.log 2024-04-19T01:06:08.384 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28831.log: 94.1% -- replaced with /var/log/ceph/ceph-client.admin.28741.log.gz 2024-04-19T01:06:08.384 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29276.log 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28939.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28879.log.gz 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29085.log 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29276.log: 91.2% 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29666.log.gz 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29110.log.gz 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28762.log 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30053.log 2024-04-19T01:06:08.392 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28762.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29276.log.gz 2024-04-19T01:06:08.393 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-19T01:06:08.393 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30053.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28762.log.gz 2024-04-19T01:06:08.400 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30155.log 2024-04-19T01:06:08.400 INFO:teuthology.orchestra.run.smithi161.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30053.log.gz 2024-04-19T01:06:08.400 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30155.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28939.log.gz 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 92.1% -- replaced with /var/log/ceph/ceph-client.admin.29085.log.gz -5 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29196.log 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28857.log 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29196.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30155.log.gz 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24102.log 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28857.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29196.log.gz 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30156.log 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.24102.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.24102.log.gz 2024-04-19T01:06:08.401 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29352.log 2024-04-19T01:06:08.408 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30156.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30156.log.gz 2024-04-19T01:06:08.408 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- 93.0% /var/log/ceph/ceph-mon.i.log 2024-04-19T01:06:08.408 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29622.log.gz 2024-04-19T01:06:08.408 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29352.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29352.log.gz 2024-04-19T01:06:08.408 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29680.log 2024-04-19T01:06:08.408 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.i.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28900.log 2024-04-19T01:06:08.412 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29680.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28717.log.gz 2024-04-19T01:06:08.412 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30158.log 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30310.log 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34532.log.gz 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35020.log 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28002.log.gz 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33412.log 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30310.log: /var/log/ceph/ceph-client.admin.35020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35886.log 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.26809.log.gz 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31732.log 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30310.log.gz 2024-04-19T01:06:08.415 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35020.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34038.log 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35886.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33412.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26976.log 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31732.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34837.log 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35886.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36016.log 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32509.log 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34038.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.26976.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36016.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34837.log.gz 2024-04-19T01:06:08.416 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29402.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31862.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31576.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36016.log.gz 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32509.log.gz 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35494.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29402.log.gz 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31576.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31862.log.gz 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33177.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29238.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.a.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31576.log.gz 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32925.log 2024-04-19T01:06:08.417 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27508.log 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33177.log.gz/var/log/ceph/ceph-client.admin.32925.log: 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34890.log 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35494.log.gz 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29238.log.gz 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34246.log 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32925.log.gz 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34890.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.e.log 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27508.log.gz 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34116.log 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34890.log.gz 2024-04-19T01:06:08.418 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32457.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34246.log.gz 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33003.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32379.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32457.log: 90.1%/var/log/ceph/ceph-client.admin.33003.log: -- replaced with /var/log/ceph/ceph-client.admin.34116.log.gz 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31288.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32457.log.gz 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32379.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33003.log.gz 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32379.log.gz 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30256.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29706.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32977.log 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31288.log.gz 2024-04-19T01:06:08.419 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31784.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31342.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31784.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29706.log.gz 92.5% -- replaced with /var/log/ceph/ceph-client.admin.30256.log.gz 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32977.log.gz 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31784.log.gz 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32018.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36146.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36198.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31342.log.gz 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28948.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33672.log 2024-04-19T01:06:08.420 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36198.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36146.log.gz 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28948.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.32018.log.gz 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36198.log.gz 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28948.log.gz 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35154.log 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34994.log 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30740.log 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33672.log.gz 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30870.log 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35154.log.gz/var/log/ceph/ceph-client.admin.30740.log: 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32122.log 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34994.log.gz 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27185.log 2024-04-19T01:06:08.421 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30870.log: /var/log/ceph/ceph-client.admin.32122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27560.log 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30740.log.gz 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27185.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32122.log.gz 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30924.log 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27185.log.gz 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32613.log 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28760.log 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30924.log: /var/log/ceph/ceph-client.admin.32613.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27560.log.gz 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36250.log 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28760.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30924.log.gz 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr:gzip 90.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.32613.log.gz -- 2024-04-19T01:06:08.422 INFO:teuthology.orchestra.run.smithi043.stderr: /var/log/ceph/ceph-client.admin.31888.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36250.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28760.log.gz 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27586.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31132.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31888.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36250.log.gz/var/log/ceph/ceph-client.admin.27586.log: 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34428.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31888.log.gz 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31132.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27586.log.gz 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27386.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31132.log.gz 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32665.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28109.log 2024-04-19T01:06:08.423 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5%/var/log/ceph/ceph-client.admin.32665.log: -- replaced with /var/log/ceph/ceph-client.admin.34428.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27846.log 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27386.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33829.log 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28109.log: /var/log/ceph/ceph-client.admin.27846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32743.log 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.32665.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33829.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.27846.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28109.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33829.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29160.log 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:gzip/var/log/ceph/ceph-client.admin.32743.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27820.log 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33029.log 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27820.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32743.log.gz 2024-04-19T01:06:08.424 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33281.log 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29160.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33594.log 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33029.log: /var/log/ceph/ceph-client.admin.33281.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.27820.log.gz 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5% 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33029.log.gz -- replaced with /var/log/ceph/ceph-client.admin.33281.log.gz 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26932.log 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27716.log 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34194.log 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5%/var/log/ceph/ceph-client.admin.27716.log: -- replaced with /var/log/ceph/ceph-client.admin.33594.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35990.log 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.26932.log.gz 2024-04-19T01:06:08.425 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34194.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27716.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29914.log 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34194.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29134.log 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35990.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31628.log 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29914.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28031.log 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29134.log: /var/log/ceph/ceph-client.admin.31628.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29134.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31628.log.gz 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32148.log 2024-04-19T01:06:08.426 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33203.log 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.33203.log: /var/log/ceph/ceph-client.admin.32096.log 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28031.log.gz 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34376.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32148.log.gz 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33203.log.gz 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27334.log 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34716.log 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36224.log 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32096.log.gz 2024-04-19T01:06:08.427 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34506.log 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27334.log.gz 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36276.log 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34716.log.gz 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35364.log 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34506.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36224.log.gz 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36276.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34506.log.gz 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36276.log.gz 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29942.log 2024-04-19T01:06:08.428 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35364.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35310.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35364.log.gz 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29030.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29942.log.gz 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32691.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32535.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29030.log: /var/log/ceph/ceph-client.admin.32691.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35310.log.gz 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29030.log.gz 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:gzip 90.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.32691.log.gz --verbose 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr: -- /var/log/ceph/ceph-client.admin.27768.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23820.log 2024-04-19T01:06:08.429 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29732.log 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32535.log.gz 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.23820.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27768.log.gz 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.23820.log.gz 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35938.log 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29732.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.d.log 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29456.log 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29732.log.gz 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.d.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35938.log.gz 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33359.log 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34220.log 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33359.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29456.log.gz 2024-04-19T01:06:08.430 INFO:teuthology.orchestra.run.smithi043.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33359.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35912.log 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33490.log 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35912.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34220.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35912.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36563.log 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27183.log 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36563.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33490.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36563.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35206.log 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33055.log 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35206.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27183.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35206.log.gz 2024-04-19T01:06:08.431 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27534.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28187.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33055.log.gz 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28734.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28083.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28734.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27534.log.gz 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28734.log.gz 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31836.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27664.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31836.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28083.log.gz 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31836.log.gz 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33385.log 2024-04-19T01:06:08.432 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35964.log 2024-04-19T01:06:08.433 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33385.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27664.log.gz 2024-04-19T01:06:08.433 INFO:teuthology.orchestra.run.smithi043.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33385.log.gz 2024-04-19T01:06:08.433 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:06:08.433 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33986.log 2024-04-19T01:06:08.433 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35964.log.gz 2024-04-19T01:06:08.437 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28900.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29680.log.gz 2024-04-19T01:06:08.438 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.g.log 2024-04-19T01:06:08.438 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30158.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28900.log.gz 2024-04-19T01:06:08.438 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-04-19T01:06:08.438 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.g.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.30158.log.gz 2024-04-19T01:06:08.438 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-19T01:06:08.438 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29643.log: 92.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28974.log 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27872.log 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33986.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.33986.log.gz 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28682.log 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27872.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.27872.log.gz 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30844.log 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28682.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28682.log.gz 2024-04-19T01:06:08.446 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34272.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30844.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30844.log.gz 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31106.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34272.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34272.log.gz 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27189.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31106.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31106.log.gz 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27326.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27189.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27189.log.gz 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32639.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35180.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29428.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31472.log 2024-04-19T01:06:08.447 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32639.log: /var/log/ceph/ceph-client.admin.35180.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27326.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35180.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32639.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29428.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29428.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27898.log 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31472.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31472.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27976.log 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27898.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.27898.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28870.log 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27976.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27976.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28057.log 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28870.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28870.log.gz 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30072.log 2024-04-19T01:06:08.448 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28057.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28057.log.gz 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33803.log 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30072.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30072.log.gz 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35074.log 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33803.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33803.log.gz 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32431.log 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35074.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35074.log.gz 2024-04-19T01:06:08.449 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31706.log 2024-04-19T01:06:08.450 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-04-19T01:06:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32431.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32431.log.gz 2024-04-19T01:06:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-04-19T01:06:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31706.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31706.log.gz 2024-04-19T01:06:08.452 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34916.log 2024-04-19T01:06:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35572.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-04-19T01:06:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34168.log 2024-04-19T01:06:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34916.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34916.log.gz 2024-04-19T01:06:08.453 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27794.log 2024-04-19T01:06:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34168.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34168.log.gz 2024-04-19T01:06:08.454 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30818.log 2024-04-19T01:06:08.455 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-19T01:06:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27794.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.27794.log.gz 2024-04-19T01:06:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35782.log 2024-04-19T01:06:08.456 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30818.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30818.log.gz 2024-04-19T01:06:08.457 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-19T01:06:08.458 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35782.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35782.log.gz 2024-04-19T01:06:08.459 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31184.log 2024-04-19T01:06:08.459 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28974.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.28974.log.gz 2024-04-19T01:06:08.466 INFO:teuthology.orchestra.run.smithi161.stderr: 95.3% -- replaced with /var/log/ceph/ceph-client.admin.28831.log.gz 2024-04-19T01:06:08.467 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30098.log 2024-04-19T01:06:08.469 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31184.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31184.log.gz 2024-04-19T01:06:08.471 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28725.log 2024-04-19T01:06:08.479 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33751.log 2024-04-19T01:06:08.481 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30098.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30098.log.gz 2024-04-19T01:06:08.481 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33438.log 2024-04-19T01:06:08.482 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33751.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.33751.log.gz 2024-04-19T01:06:08.486 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29557.log 2024-04-19T01:06:08.491 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32070.log 2024-04-19T01:06:08.493 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33438.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33438.log.gz 2024-04-19T01:06:08.493 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28213.log 2024-04-19T01:06:08.495 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30106.log 2024-04-19T01:06:08.497 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29557.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29557.log.gz 2024-04-19T01:06:08.503 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32070.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.32070.log.gz 2024-04-19T01:06:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30420.log 2024-04-19T01:06:08.510 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30106.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30106.log.gz 2024-04-19T01:06:08.511 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30124.log 2024-04-19T01:06:08.511 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28213.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28213.log.gz 2024-04-19T01:06:08.515 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30054.log 2024-04-19T01:06:08.526 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34638.log 2024-04-19T01:06:08.527 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30124.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30124.log.gz 2024-04-19T01:06:08.527 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30420.log: 91.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30105.log 2024-04-19T01:06:08.531 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30054.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.30054.log.gz 2024-04-19T01:06:08.531 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30420.log.gz 2024-04-19T01:06:08.531 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31940.log 2024-04-19T01:06:08.532 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34638.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.34638.log.gz 2024-04-19T01:06:08.535 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.h.log 2024-04-19T01:06:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29264.log 2024-04-19T01:06:08.540 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31940.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.31940.log.gz 2024-04-19T01:06:08.548 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34298.log 2024-04-19T01:06:08.549 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29264.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.29264.log.gz 2024-04-19T01:06:08.550 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30105.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.30105.log.gz 2024-04-19T01:06:08.555 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35048.log 2024-04-19T01:06:08.556 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34298.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34298.log.gz 2024-04-19T01:06:08.558 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30233.log 2024-04-19T01:06:08.563 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28135.log 2024-04-19T01:06:08.564 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35048.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35048.log.gz 2024-04-19T01:06:08.571 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.h.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24222.log 2024-04-19T01:06:08.571 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28239.log 2024-04-19T01:06:08.572 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28135.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.28135.log.gz 2024-04-19T01:06:08.574 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30233.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.30233.log.gz 2024-04-19T01:06:08.579 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30231.log 2024-04-19T01:06:08.579 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30714.log 2024-04-19T01:06:08.580 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.24222.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24222.log.gz 2024-04-19T01:06:08.580 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28239.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28239.log.gz 2024-04-19T01:06:08.587 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35834.log 2024-04-19T01:06:08.588 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30714.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30714.log.gz 2024-04-19T01:06:08.595 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29225.log 2024-04-19T01:06:08.596 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30228.log 2024-04-19T01:06:08.597 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35834.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35834.log.gz 2024-04-19T01:06:08.598 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30231.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.30231.log.gz 2024-04-19T01:06:08.602 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28725.log.gz 2024-04-19T01:06:08.603 INFO:teuthology.orchestra.run.smithi043.stderr: 90.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:06:08.604 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34558.log 2024-04-19T01:06:08.605 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30228.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30228.log.gz 2024-04-19T01:06:08.605 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29678.log 2024-04-19T01:06:08.606 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34558.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34558.log.gz 2024-04-19T01:06:08.606 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27181.log 2024-04-19T01:06:08.608 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29678.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29678.log.gz 2024-04-19T01:06:08.608 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32899.log 2024-04-19T01:06:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27181.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27181.log.gz 2024-04-19T01:06:08.609 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30634.log 2024-04-19T01:06:08.610 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32899.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32899.log.gz 2024-04-19T01:06:08.610 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35258.log 2024-04-19T01:06:08.611 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30634.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30634.log.gz 2024-04-19T01:06:08.612 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28842.log 2024-04-19T01:06:08.613 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35258.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35258.log.gz 2024-04-19T01:06:08.613 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31654.log 2024-04-19T01:06:08.614 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28842.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28842.log.gz 2024-04-19T01:06:08.615 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30229.log 2024-04-19T01:06:08.615 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:06:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31654.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31654.log.gz 2024-04-19T01:06:08.616 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27026.log 2024-04-19T01:06:08.631 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29716.log 2024-04-19T01:06:08.633 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30229.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.30229.log.gz 2024-04-19T01:06:08.635 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29994.log 2024-04-19T01:06:08.636 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27026.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27026.log.gz 2024-04-19T01:06:08.639 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29364.log 2024-04-19T01:06:08.647 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29716.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-19T01:06:08.648 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29364.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29364.log.gz 2024-04-19T01:06:08.651 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34454.log 2024-04-19T01:06:08.652 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29994.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29994.log.gz 2024-04-19T01:06:08.653 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29888.log 2024-04-19T01:06:08.653 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34454.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34454.log.gz 2024-04-19T01:06:08.658 INFO:teuthology.orchestra.run.smithi043.stderr: 96.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:06:08.659 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31002.log 2024-04-19T01:06:08.660 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29888.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29888.log.gz 2024-04-19T01:06:08.661 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31602.log 2024-04-19T01:06:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31002.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31002.log.gz 2024-04-19T01:06:08.662 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30202.log 2024-04-19T01:06:08.663 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31602.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.31602.log.gz 2024-04-19T01:06:08.664 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31992.log 2024-04-19T01:06:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30202.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30202.log.gz 2024-04-19T01:06:08.665 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35624.log 2024-04-19T01:06:08.666 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31992.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31992.log.gz 2024-04-19T01:06:08.667 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32873.log 2024-04-19T01:06:08.668 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35624.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35624.log.gz 2024-04-19T01:06:08.668 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27032.log 2024-04-19T01:06:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32873.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32873.log.gz 2024-04-19T01:06:08.669 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35128.log 2024-04-19T01:06:08.670 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27032.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27032.log.gz 2024-04-19T01:06:08.671 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30230.log 2024-04-19T01:06:08.671 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28816.log 2024-04-19T01:06:08.672 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35128.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35128.log.gz 2024-04-19T01:06:08.672 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33151.log 2024-04-19T01:06:08.673 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28816.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28816.log.gz 2024-04-19T01:06:08.674 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30896.log 2024-04-19T01:06:08.675 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33151.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33151.log.gz 2024-04-19T01:06:08.675 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32174.log 2024-04-19T01:06:08.677 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30896.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30896.log.gz 2024-04-19T01:06:08.677 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34142.log 2024-04-19T01:06:08.677 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32174.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32174.log.gz 2024-04-19T01:06:08.678 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30976.log 2024-04-19T01:06:08.679 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34142.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34142.log.gz 2024-04-19T01:06:08.680 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27950.log 2024-04-19T01:06:08.680 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30976.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30976.log.gz 2024-04-19T01:06:08.681 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35808.log 2024-04-19T01:06:08.682 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27950.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.27950.log.gz 2024-04-19T01:06:08.682 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36380.log 2024-04-19T01:06:08.683 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29028.log 2024-04-19T01:06:08.683 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35808.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35808.log.gz 2024-04-19T01:06:08.684 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36539.log 2024-04-19T01:06:08.684 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30230.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30230.log.gz 2024-04-19T01:06:08.685 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36380.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36380.log.gz 2024-04-19T01:06:08.685 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32795.log 2024-04-19T01:06:08.686 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36539.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36539.log.gz 2024-04-19T01:06:08.687 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36068.log 2024-04-19T01:06:08.688 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32795.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32795.log.gz 2024-04-19T01:06:08.688 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29810.log 2024-04-19T01:06:08.689 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36068.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36068.log.gz 2024-04-19T01:06:08.690 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31314.log 2024-04-19T01:06:08.692 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30159.log 2024-04-19T01:06:08.692 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29810.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29810.log.gz 2024-04-19T01:06:08.692 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31080.log 2024-04-19T01:06:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31314.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31314.log.gz 2024-04-19T01:06:08.693 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31524.log 2024-04-19T01:06:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31080.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31080.log.gz 2024-04-19T01:06:08.694 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29350.log 2024-04-19T01:06:08.696 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31524.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31524.log.gz 2024-04-19T01:06:08.696 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34584.log 2024-04-19T01:06:08.697 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29350.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29350.log.gz 2024-04-19T01:06:08.698 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31810.log 2024-04-19T01:06:08.699 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29028.log: 90.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29055.log 2024-04-19T01:06:08.699 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29028.log.gz 2024-04-19T01:06:08.700 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34584.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34584.log.gz 2024-04-19T01:06:08.700 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27690.log 2024-04-19T01:06:08.700 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31810.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31810.log.gz 2024-04-19T01:06:08.701 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30362.log 2024-04-19T01:06:08.705 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27690.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27690.log.gz 2024-04-19T01:06:08.705 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-04-19T01:06:08.708 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30362.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30362.log.gz 2024-04-19T01:06:08.708 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33568.log 2024-04-19T01:06:08.708 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28161.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-04-19T01:06:08.708 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32587.log 2024-04-19T01:06:08.708 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33568.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33568.log.gz 2024-04-19T01:06:08.708 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33516.log 2024-04-19T01:06:08.709 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30159.log: 91.4% 92.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24150.log 2024-04-19T01:06:08.709 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32587.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32587.log.gz 2024-04-19T01:06:08.709 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34742.log 2024-04-19T01:06:08.711 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33516.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.33516.log.gz 2024-04-19T01:06:08.711 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30950.log 2024-04-19T01:06:08.712 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34742.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.34742.log.gz 2024-04-19T01:06:08.713 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30660.log 2024-04-19T01:06:08.713 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30950.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30950.log.gz 2024-04-19T01:06:08.714 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-04-19T01:06:08.715 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29055.log: 92.1% 93.0% -- replaced with /var/log/ceph/ceph-client.admin.30159.log.gz 2024-04-19T01:06:08.715 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29628.log 2024-04-19T01:06:08.716 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30660.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30660.log.gz 2024-04-19T01:06:08.716 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-04-19T01:06:08.717 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.24150.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.24150.log.gz 2024-04-19T01:06:08.717 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29056.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-04-19T01:06:08.717 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36172.log 2024-04-19T01:06:08.719 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29225.log.gz 2024-04-19T01:06:08.719 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35390.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-04-19T01:06:08.719 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36459.log 2024-04-19T01:06:08.720 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36172.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36172.log.gz 2024-04-19T01:06:08.720 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27612.log 2024-04-19T01:06:08.721 INFO:teuthology.orchestra.run.smithi161.stderr: 96.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-19T01:06:08.722 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36459.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36459.log.gz 2024-04-19T01:06:08.722 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33542.log 2024-04-19T01:06:08.726 INFO:teuthology.orchestra.run.smithi161.stderr: 90.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-19T01:06:08.732 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29055.log.gz -- replaced with /var/log/ceph/ceph-client.admin.29716.log.gz 2024-04-19T01:06:08.732 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27612.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27612.log.gz 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35416.log 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33542.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33542.log.gz 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30580.log 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35416.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35416.log.gz 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29784.log 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30580.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30580.log.gz 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29004.log 2024-04-19T01:06:08.734 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29784.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29784.log.gz 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32951.log 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29004.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29004.log.gz 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27029.log 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32951.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32951.log.gz 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33646.log 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27029.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27029.log.gz 2024-04-19T01:06:08.735 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31966.log 2024-04-19T01:06:08.736 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29350.log 2024-04-19T01:06:08.736 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33646.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33646.log.gz 2024-04-19T01:06:08.736 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33777.log 2024-04-19T01:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31966.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31966.log.gz 2024-04-19T01:06:08.737 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30792.log 2024-04-19T01:06:08.738 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33777.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33777.log.gz 2024-04-19T01:06:08.740 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35546.log 2024-04-19T01:06:08.741 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30792.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30792.log.gz 2024-04-19T01:06:08.741 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29324.log 2024-04-19T01:06:08.742 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35546.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.35546.log.gz 2024-04-19T01:06:08.742 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33307.log 2024-04-19T01:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29324.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29324.log.gz 2024-04-19T01:06:08.743 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-19T01:06:08.745 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33307.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33307.log.gz 2024-04-19T01:06:08.745 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33699.log 2024-04-19T01:06:08.751 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29277.log 2024-04-19T01:06:08.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29350.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29350.log.gz 2024-04-19T01:06:08.763 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30421.log 2024-04-19T01:06:08.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29277.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29277.log.gz 2024-04-19T01:06:08.767 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34324.log 2024-04-19T01:06:08.769 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33699.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.33699.log.gz 2024-04-19T01:06:08.771 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-04-19T01:06:08.773 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30421.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30421.log.gz 2024-04-19T01:06:08.779 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29467.log 2024-04-19T01:06:08.781 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30055.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-04-19T01:06:08.787 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28785.log 2024-04-19T01:06:08.787 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29968.log 2024-04-19T01:06:08.788 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34324.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.34324.log.gz 2024-04-19T01:06:08.790 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29467.log: -- replaced with /var/log/ceph/ceph-client.admin.29628.log.gz 2024-04-19T01:06:08.796 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33620.log 2024-04-19T01:06:08.796 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29968.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.29968.log.gz 2024-04-19T01:06:08.798 INFO:teuthology.orchestra.run.smithi161.stderr: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-19T01:06:08.806 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28785.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29467.log.gz 2024-04-19T01:06:08.807 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28786.log 2024-04-19T01:06:08.808 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33620.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.33620.log.gz 2024-04-19T01:06:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36485.log 2024-04-19T01:06:08.816 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28786.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28786.log.gz 2024-04-19T01:06:08.818 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28785.log.gz 2024-04-19T01:06:08.823 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33229.log 2024-04-19T01:06:08.824 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.36485.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36485.log.gz 2024-04-19T01:06:08.827 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29252.log 2024-04-19T01:06:08.831 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27330.log 2024-04-19T01:06:08.832 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33229.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.33229.log.gz 2024-04-19T01:06:08.839 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29241.log 2024-04-19T01:06:08.839 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31914.log 2024-04-19T01:06:08.840 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29252.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29252.log.gz 2024-04-19T01:06:08.840 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27330.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27330.log.gz 2024-04-19T01:06:08.847 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28757.log 2024-04-19T01:06:08.847 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34664.log 2024-04-19T01:06:08.848 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31914.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31914.log.gz 2024-04-19T01:06:08.855 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31210.log 2024-04-19T01:06:08.856 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34664.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34664.log.gz 2024-04-19T01:06:08.859 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29241.log: 94.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29484.log 2024-04-19T01:06:08.863 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30284.log 2024-04-19T01:06:08.864 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31210.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.31210.log.gz 2024-04-19T01:06:08.866 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28757.log: -- replaced with /var/log/ceph/ceph-client.admin.29241.log.gz 2024-04-19T01:06:08.871 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31498.log 2024-04-19T01:06:08.872 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30284.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30284.log.gz 2024-04-19T01:06:08.874 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30419.log 2024-04-19T01:06:08.891 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29484.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28749.log 2024-04-19T01:06:08.891 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29484.log.gz 2024-04-19T01:06:08.891 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29376.log 2024-04-19T01:06:08.892 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31498.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31498.log.gz 2024-04-19T01:06:08.894 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.30419.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30419.log.gz 2024-04-19T01:06:08.911 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28801.log 2024-04-19T01:06:08.911 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28749.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.28749.log.gz 2024-04-19T01:06:08.911 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35336.log 2024-04-19T01:06:08.912 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29376.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29376.log.gz 2024-04-19T01:06:08.914 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24126.log 2024-04-19T01:06:08.920 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30150.log 2024-04-19T01:06:08.920 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35336.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.35336.log.gz 2024-04-19T01:06:08.933 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.28801.log: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28757.log.gz 2024-04-19T01:06:08.935 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29186.log 2024-04-19T01:06:08.936 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30150.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30150.log.gz 2024-04-19T01:06:08.938 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29093.log 2024-04-19T01:06:08.940 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.24126.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.24126.log.gz 2024-04-19T01:06:08.943 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32483.log 2024-04-19T01:06:08.944 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29186.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29186.log.gz 2024-04-19T01:06:08.947 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29270.log 2024-04-19T01:06:08.947 INFO:teuthology.orchestra.run.smithi161.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28857.log.gz 2024-04-19T01:06:08.951 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30020.log 2024-04-19T01:06:08.952 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32483.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32483.log.gz 2024-04-19T01:06:08.959 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29652.log 2024-04-19T01:06:08.960 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30020.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30020.log.gz 2024-04-19T01:06:08.965 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.29093.log: 92.3%/var/log/ceph/ceph-client.admin.29270.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.29270.log.gz 2024-04-19T01:06:08.967 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29093.log.gz 2024-04-19T01:06:08.967 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27638.log 2024-04-19T01:06:08.968 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29652.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29652.log.gz 2024-04-19T01:06:08.976 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29298.log 2024-04-19T01:06:08.976 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27638.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27638.log.gz 2024-04-19T01:06:08.983 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28896.log 2024-04-19T01:06:08.984 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29298.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29298.log.gz 2024-04-19T01:06:08.991 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31028.log 2024-04-19T01:06:08.992 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28896.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28896.log.gz 2024-04-19T01:06:08.999 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31262.log 2024-04-19T01:06:09.000 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31028.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31028.log.gz 2024-04-19T01:06:09.010 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34012.log 2024-04-19T01:06:09.010 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31262.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31262.log.gz 2024-04-19T01:06:09.015 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27034.log 2024-04-19T01:06:09.016 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.34012.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34012.log.gz 2024-04-19T01:06:09.023 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33125.log 2024-04-19T01:06:09.024 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27034.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.27034.log.gz 2024-04-19T01:06:09.031 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27028.log 2024-04-19T01:06:09.032 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33125.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33125.log.gz 2024-04-19T01:06:09.039 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32353.log 2024-04-19T01:06:09.040 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27028.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27028.log.gz 2024-04-19T01:06:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32405.log 2024-04-19T01:06:09.048 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32353.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32353.log.gz 2024-04-19T01:06:09.049 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-19T01:06:09.055 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27030.log 2024-04-19T01:06:09.056 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32405.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32405.log.gz 2024-04-19T01:06:09.063 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31680.log 2024-04-19T01:06:09.064 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27030.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27030.log.gz 2024-04-19T01:06:09.071 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31758.log 2024-04-19T01:06:09.072 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31680.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.31680.log.gz 2024-04-19T01:06:09.080 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35232.log 2024-04-19T01:06:09.080 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31758.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31758.log.gz 2024-04-19T01:06:09.087 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31158.log 2024-04-19T01:06:09.088 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.35232.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35232.log.gz 2024-04-19T01:06:09.099 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29862.log 2024-04-19T01:06:09.099 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.31158.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31158.log.gz 2024-04-19T01:06:09.103 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20137.log 2024-04-19T01:06:09.104 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29862.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29862.log.gz 2024-04-19T01:06:09.115 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26834.log 2024-04-19T01:06:09.115 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph.tmp-client.admin.20137.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20137.log.gz 2024-04-19T01:06:09.119 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29482.log 2024-04-19T01:06:09.120 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.26834.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.26834.log.gz 2024-04-19T01:06:09.127 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32275.log 2024-04-19T01:06:09.128 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.29482.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29482.log.gz 2024-04-19T01:06:09.135 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33464.log 2024-04-19T01:06:09.136 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32275.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32275.log.gz 2024-04-19T01:06:09.144 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32769.log 2024-04-19T01:06:09.144 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.33464.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33464.log.gz 2024-04-19T01:06:09.153 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28978.log 2024-04-19T01:06:09.153 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.32769.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.32769.log.gz 2024-04-19T01:06:09.159 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24126.log 2024-04-19T01:06:09.160 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.28978.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28978.log.gz 2024-04-19T01:06:09.167 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27170.log 2024-04-19T01:06:09.168 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.24126.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.24126.log.gz 2024-04-19T01:06:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30046.log 2024-04-19T01:06:09.176 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27170.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27170.log.gz 2024-04-19T01:06:09.184 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27190.log 2024-04-19T01:06:09.184 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30046.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30046.log.gz 2024-04-19T01:06:09.193 INFO:teuthology.orchestra.run.smithi043.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30336.log 2024-04-19T01:06:09.194 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.27190.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.27190.log.gz 2024-04-19T01:06:09.202 INFO:teuthology.orchestra.run.smithi043.stderr:/var/log/ceph/ceph-client.admin.30336.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30336.log.gz 2024-04-19T01:06:09.318 INFO:teuthology.orchestra.run.smithi161.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.h.log.gz 2024-04-19T01:06:09.377 INFO:teuthology.orchestra.run.smithi043.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.d.log.gz 2024-04-19T01:06:09.449 INFO:teuthology.orchestra.run.smithi161.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.f.log.gz 2024-04-19T01:06:09.573 INFO:teuthology.orchestra.run.smithi161.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.i.log.gz 2024-04-19T01:06:09.577 INFO:teuthology.orchestra.run.smithi043.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-19T01:06:09.642 INFO:teuthology.orchestra.run.smithi043.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-19T01:06:10.127 INFO:teuthology.orchestra.run.smithi043.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.e.log.gz 2024-04-19T01:06:10.159 INFO:teuthology.orchestra.run.smithi161.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mon.g.log.gz 2024-04-19T01:06:10.352 INFO:teuthology.orchestra.run.smithi043.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-19T01:06:10.493 INFO:teuthology.orchestra.run.smithi161.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28801.log.gz 2024-04-19T01:06:15.546 INFO:teuthology.orchestra.run.smithi043.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-19T01:06:16.050 INFO:teuthology.orchestra.run.smithi043.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-19T01:06:16.079 INFO:teuthology.orchestra.run.smithi161.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-19T01:06:16.170 INFO:teuthology.orchestra.run.smithi043.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-19T01:06:16.171 INFO:teuthology.orchestra.run.smithi043.stderr: 2024-04-19T01:06:16.171 INFO:teuthology.orchestra.run.smithi043.stderr:real 0m7.873s 2024-04-19T01:06:16.171 INFO:teuthology.orchestra.run.smithi043.stderr:user 0m29.125s 2024-04-19T01:06:16.171 INFO:teuthology.orchestra.run.smithi043.stderr:sys 0m1.002s 2024-04-19T01:06:16.187 INFO:teuthology.orchestra.run.smithi161.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-19T01:06:16.502 INFO:teuthology.orchestra.run.smithi161.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-19T01:06:16.504 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-04-19T01:06:16.504 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m8.167s 2024-04-19T01:06:16.504 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m29.763s 2024-04-19T01:06:16.504 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m1.065s 2024-04-19T01:06:16.504 INFO:tasks.ceph:Archiving logs... 2024-04-19T01:06:16.505 DEBUG:teuthology.misc:Transferring archived files from smithi043:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/remote/smithi043/log 2024-04-19T01:06:16.506 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:06:20.414 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/remote/smithi161/log 2024-04-19T01:06:20.415 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-19T01:06:24.285 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-19T01:06:24.295 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-19T01:06:24.295 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:06:24.301 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-19T01:06:24.373 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-19T01:06:24.383 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-19T01:06:24.383 DEBUG:teuthology.orchestra.run.smithi043:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:06:24.386 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-19T01:06:24.396 INFO:teuthology.orchestra.run.smithi043.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:06:24.396 INFO:teuthology.orchestra.run.smithi043.stdout:============================================================================== 2024-04-19T01:06:24.396 INFO:teuthology.orchestra.run.smithi043.stdout:*hv01.front.sepi 67.205.162.81 3 u 29 128 377 0.086 -2.715 0.246 2024-04-19T01:06:24.397 INFO:teuthology.orchestra.run.smithi043.stdout:+hv02.front.sepi 63.231.80.2 3 u 24 128 377 0.082 -0.379 0.233 2024-04-19T01:06:24.397 INFO:teuthology.orchestra.run.smithi043.stdout:+hv03.front.sepi 74.6.168.72 3 u 25 128 377 0.063 -1.942 0.351 2024-04-19T01:06:24.397 INFO:teuthology.orchestra.run.smithi043.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-04-19T01:06:24.398 INFO:teuthology.orchestra.run.smithi161.stdout: remote refid st t when poll reach delay offset jitter 2024-04-19T01:06:24.398 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================== 2024-04-19T01:06:24.398 INFO:teuthology.orchestra.run.smithi161.stdout:*hv01.front.sepi 67.205.162.81 3 u 21 64 377 0.088 -4.789 0.407 2024-04-19T01:06:24.398 INFO:teuthology.orchestra.run.smithi161.stdout:+hv02.front.sepi 63.231.80.2 3 u 20 64 377 0.087 -3.079 0.521 2024-04-19T01:06:24.399 INFO:teuthology.orchestra.run.smithi161.stdout:+hv03.front.sepi 74.6.168.72 3 u 22 64 377 0.100 -4.382 0.478 2024-04-19T01:06:24.399 INFO:teuthology.orchestra.run.smithi161.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-04-19T01:06:24.399 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-19T01:06:24.409 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-19T01:06:24.410 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-19T01:06:24.445 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-19T01:06:24.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-19T01:06:24.511 INFO:teuthology.task.internal:Duration was 1901.940672 seconds 2024-04-19T01:06:24.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-19T01:06:24.544 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-19T01:06:24.545 DEBUG:teuthology.orchestra.run.smithi043:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:06:24.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-19T01:06:24.586 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-19T01:06:24.586 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi043.front.sepia.ceph.com 2024-04-19T01:06:24.586 DEBUG:teuthology.orchestra.run.smithi043:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:06:24.636 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-04-19T01:06:24.637 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-19T01:06:24.648 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-19T01:06:24.648 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:06:24.681 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-19T01:06:24.722 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-19T01:06:24.723 DEBUG:teuthology.orchestra.run.smithi043:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:06:24.749 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-19T01:06:24.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-19T01:06:24.840 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-19T01:06:24.841 DEBUG:teuthology.orchestra.run.smithi043:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:06:24.854 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-19T01:06:24.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-19T01:06:24.878 DEBUG:teuthology.orchestra.run.smithi043:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:06:24.897 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:06:24.905 INFO:teuthology.orchestra.run.smithi043.stdout:kernel.core_pattern = core 2024-04-19T01:06:24.920 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-04-19T01:06:24.933 DEBUG:teuthology.orchestra.run.smithi043:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:06:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:06:24.972 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-19T01:06:24.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-19T01:06:24.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-19T01:06:24.989 INFO:teuthology.task.internal:Transferring archived files... 2024-04-19T01:06:24.990 DEBUG:teuthology.misc:Transferring archived files from smithi043:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/remote/smithi043 2024-04-19T01:06:24.990 DEBUG:teuthology.orchestra.run.smithi043:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:06:25.036 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-18_22:49:05-rados-wip-yuri5-testing-2024-04-17-1400-distro-default-smithi/7663199/remote/smithi161 2024-04-19T01:06:25.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-19T01:06:25.080 INFO:teuthology.task.internal:Removing archive directory... 2024-04-19T01:06:25.080 DEBUG:teuthology.orchestra.run.smithi043:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:06:25.083 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-19T01:06:25.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-19T01:06:25.115 INFO:teuthology.task.internal:Not uploading archives. 2024-04-19T01:06:25.115 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-19T01:06:25.138 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-19T01:06:25.139 DEBUG:teuthology.orchestra.run.smithi043:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:06:25.141 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-19T01:06:25.145 INFO:teuthology.orchestra.run.smithi043.stdout: 658754 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:06 /home/ubuntu/cephtest 2024-04-19T01:06:25.147 INFO:teuthology.orchestra.run.smithi161.stdout: 658460 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 19 01:06 /home/ubuntu/cephtest 2024-04-19T01:06:25.148 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-19T01:06:25.171 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-19T01:06:25.241 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1901.9406719207764 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-19T01:06:25.241 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-19T01:06:25.320 INFO:teuthology.run:pass